COSIMA 2019 Report

This report summarises the fourth meeting of the Consortium for Ocean Sea Ice Modelling in Australia (COSIMA), held in Canberra on 3-4 September 2019. Shweta Sharma has provided a more informal (and entertaining) report here.

Aims & Goals

The annual COSIMA workshop aims to:

  • Maintain and grow the established community around ocean-sea ice modelling in Australia;
  • Discuss recent scientific advances in ocean and sea ice research in a forum that is inclusive and model-agnostic, particularly including observational programs;
  • Agree on immediate next steps in the COSIMA model development plan; and
  • Develop a long-term vision for ocean-sea ice model development to support Australian researchers.

Participants


Attendees included Alberto Alberello (U Adelaide), Christopher Bladwell (UNSW), Fabio Boeira Dias (UTAS/CSIRO), Gary Brassington (BOM), Matt Chamberlain (CSIRO), Navid Constantinou (ANU), Prasanth Divakaran (BOM), Kelsey Druken (NCI), Matthew England (UNSW), Ben Evans (NCI), Hakase Hayashida (IMAS, UTAS), Petra Heil (AAD & AAPP), Andy Hogg (ANU), Ryan Holmes (UNSW), Maurice Huguenin (UNSW), Yi Jin (CSIRO), Andrew Kiss (ANU), Andreas Klocker (UTAS), Qian Li (UNSW), Kewei Lyu (CSIRO), Simon Marsland (CSIRO), Josue Martinez Moreno (ANU), Richard Matear (CSIRO), Ruth Moorman (ANU), Adele Morrison (ANU), Eric Mortenson (CSIRO), Jemima Rama (ANU), Paul Sandery (CSIRO), Abhishek Savita (UTAS/IMAS/CSIRO), Callum Shakespeare (ANU), Shweta Sharma (UNSW), Callum Shaw (ANU), Taimoor Sohail (ANU), Paul Spence (UNSW), Kial Stewart (ANU), Veronica Tamsitt (UNSW), Mirko Velic (BOM), Nick Velzeboer (ANU), Jingbo Wang (NCI), Xuebin Zhang (CSIRO), Xihan Zhang (ANU), Aihong Zhong (BOM), plus those who attended via video conference.

Program

Tuesday 3rd September

Session 1 (Chair – Navid Constantinou)

Andrew Kiss (ANU): ACCESS-OM2 update
Simon Marsland (CSIRO): ACCESS and CMIP6
Hakase Hayashida (IMAS, UTAS): Preliminary results of biogeochemistry simulation with ACCESS-OM2 and plans for OMIP-BGC and IAMIP
Ben Evans (NCI): Addressing the next HPC challenges for Climate and Weather

Session 2 (Chair – Andreas Klocker)

Veronica Tamsitt (UNSW): Lagrangian pathways and residence time of warm Circumpolar Deep Water on the Antarctic continental shelf
Ruth Moorman (ANU): Response of Antarctic ocean circulation to increased glacial meltwater
Kewei Lyu (CSIRO): Southern Ocean heat uptake and redistribution in theoretical framework and model perturbation experiments
Fabio Boeira Dias (UTAS/CSIRO): High-latitude Southern Ocean response to changes in surface momentum, heat and freshwater fluxes under 2xCO2 concentration

Session 3 (Chair – Simon Marsland)

Xuebin Zhang (CSIRO): Dynamical downscaling of climate changes with OFAM3
Matt Chamberlain (CSIRO): Multiscale data assimilation in Bluelink Reanalysis
Paul Sandery (CSIRO): A data assimilation framework for ocean-sea-ice prediction
Prasanth Divakaran (Bureau of Meteorology): OceanMAPS 3.3 Developments

Wednesday 4th September

Session 4 (Chair – Veronica Tamsitt)

Ryan Holmes (UNSW):  Atlantic ocean heat transport enabled by Indo-Pacific heat uptake and mixing
Eric Mortenson (CSIRO): Decoupling of carbon and heat uptake rates of the global ocean over the 21st century
Christopher Bladwell (UNSW): Diahaline transport in global ocean models
Abhishek Savita (UTAS, IMAS, CSIRO): Uncertainty in the estimation of global and regional ocean heat content since 1970
Gary Brassington (BOM): Comparison of ACCESS-OM2-01 to other models and observations

Session 5 (Chair – Qian Li)

Xihan Zhang (ANU): Gulf Stream separation in ACCESS-OM2
Alberto Alberello (U Adelaide): Impacts of winter cyclones on sea ice dynamics
Petra Heil (AAD): Sea ice in the ACCESS-OM2-01: Exploring near-coastal processes

COSIMA Discussion (Chair – Paul Spence)

Open discussion highlighted a number of potential avenues for work in the near-term, as well as some suggestions for directions that could be included in a future COSIMA funding bid.

Near-term Priorities

  • Running the COSIMA cookbook on the VDI is becoming untenable, and recent improvements in the cookbook have not been widely adopted. This should be a priority, possibly with a tutorial session at the CLEx Annual Workshop?
  • Start investigating coupled data assimilation for parameter estimation, especially for sea ice.
  • Start serious perturbation experiments with ACCESS-OM2-01, potentially including:
    • SAMx (RYF forcing with SAM Extreme Years).
    • Adding katabatic winds?
    • Tropical mixing and the AMOC.
    • Influence of the Amundsen Sea Low on the Southern Ocean.
    • Turbulent Kinetic Energy and winds in the Southern Ocean.
  • OMIP2 contribution for CMIP6.
  • Improve communication of COSIMA achievements.
  • Begin work on nesting regional MOM6 models.

Longer term suggestions

  • Better connection with Paleo community.
  • Improve links with the wave community.
  • Start running ensemble simulations?
  • Do we need to move to CICE6?
  • Capacity to run future scenarios based on coupled model output.

It was agreed that COSIMA V will be held in 2020, hosted by Xuebin Zhang in Hobart.

Additional discussion points are given here.

Awards

The COSIMA Most Selfless Contributor Awards for 2017, 2018 and 2019 were presented in absentia to

  • 2017 James Munroe
  • 2018 Marshall Ward
  • 2019 Russ Fiedler (pictured)

in appreciation of their tireless efforts which have greatly improved the software used by the COSIMA community.

Past, Present and Future of the COSIMA Community

Being the newcomer in the field of climate modelling, I got a chance to attend the fourth annual COSIMA workshop, which was held in ANU, Canberra from 3-4 September. As it was my first visit to ANU and Canberra, I was so excited beforehand. For me, the spotlight of enthusiasm was to meet all people in person whom I was just seen through multiple small screens during all MOM meetings. Parallel to my thrill, I was worried about the cold weather of Canberra, which I thought going to be harsh but in contradictory, it was so pleasant with great sunshine which makes my three days journey in Canberra the best so far.

COSIMA is a Consortium for Ocean-Sea Ice Modelling in Australia, which is supported by Australian researchers from number of different university nodes like ANU, UNSW, UTAS and by public-funded research centres like CSIRO, Bureau of Meteorology, Australian Antarctic Division with strong assistance from NCI (National Computational Infrastructure) and ARC (Australian Research Council). The main aims for heading the workshop was:

  • To maintain and grow the established community around ocean-sea ice modelling in Australia.
  • To discuss recent scientific advances in the ocean and sea-ice research.
  • To plan how can we best utilise COSIMA models to address the biggest scientific questions.
  • To have suggestions and agreement on immediate future steps in the COSIMA model development plans.

In total, there were 42 participants and 20 presentations followed by deep question-answering sessions. Presentation slides are available from the workshop report.

 

The first session was started by Andrew, which was mainly focusing on the updates of COSIMA and ACCESS-OM2-01 model, 10th-degree resolution for next-generation climate models, new 10th-degree run with 1990-91 RYF (Repeat year forcing) for 50 years, the issues dealing with the crash of the model and on technical progress with COSIMA. In general, his talk was about where we stand at this stage as a COSIMA community, which was very useful to know for the beginners like me. The session was continued by Simon, which he concluded by making a critical remark “new climate models forecast a warming surge”. Further, Hakase explained his results on biogeochemistry simulation with ACCESS-OM2 and plans for OMIP-BGC (Ocean model inter-comparison project) and IAMIP. Then Ben from NCI addressed the next HP challenges for Climate and weather, how flexible are the models ready for HPC system? Ben’s talks point out challenges like how to deal with data? why it’s important to regular updates on the data? He mentioned that the requirement trend of NCI is going up with more data sets which in Ben’s view, making NCI’s life crazy, totally relatable Ben.

The second session of the first day, was mostly focusing on recent scientific advances in the community, which included talks like Response of Antarctic ocean circulation to increased glacial meltwater (in ACCESS-OM2-01), Understanding the Southern Ocean heat uptake and redistribution in theoretical framework and model perturbation experiments, Lagrangian pathways and residence time of Circumpolar Deep Water on the Antarctic continental shelf in a 1/10 degree ocean-sea ice model, High-latitude Southern Ocean response to surface forcing changes under 2xCO2. The first-day session ends with talks on OFAM3, need for the new experiments, its developments and multi-scale data assimilation in Bluelink Reanalysis.

After great talks, the day ended with a pleasant dinner, funny discussions, good bondings with new people. For me the best part of that evening rather than a splendid mulled wine was, I got a chance to know some researchers who are working in the same field, on a professional and friendly level.

The second day of the workshop started with sessions which are targeting on research done on Ocean heat contents, its projection, its variability and uncertainty in the estimation of global and regional ocean heat content. The second half session was intensive on sea ice dynamics in theACCESS-OM2 suite, Impacts of winter cyclones on sea ice dynamics and Gulf Stream separation in ACCESS-OM2.

The spotlight of the second day was a session held by Paul, that is the award ceremony and closing out the workshop. I was totally unaware of a pretty cool ceremony. From 2017, since COSIMA community started their annual workshop, they are presenting this award to recognise the hardcore personalities who worked passionately for the development of COSIMA  community, cookbook and have active participation in resolving issues which researchers getting while handling the models. In 2017, the award was given to James Munroe who is contributing to Cosima-cookbook and Cosima-recipe. In 2018, it was passed to Marshall Ward who is now in GFDL, Princeton but still gives his precious contributions to the Community. This year COSIMA most selfless contributor award was honoured to Russell Fiedler from CSIRO for his unconditional support to the COSIMA-Community, the best part of the afternoon was his speech, which tells about his sense of humour. I have never met Russell or other awardees in person but I am looking forward to meeting them in future. It was remarked in the ceremony that if you will get the award next year, it is mandatory not to be present there as, since 2017, all awards presented virtually to the nominees [NB: this was a joke!]. I hope, next year I can see the trophy with the award holder. All in my hope. Anyways, I like the idea of felicitating the contributors in the community so that it motivates new researchers to work in that direction, not in a way to get that trophy but in a way to make their path through some contribution in the community.

One of the focal points of the closing session was how to improve Cosima-cookbook and recipe,  and how to increase the users for the cookbook. As a new user to the python language,  the Cosima-cookbook and recipe examples were very helpful for me. With the help, I can able to make basic scripts in the jupyter notebook. The cookbook examples were onto my expectations and make my life easy in analysing and visualising the basic plots. I would like to recommend this to other researchers in the field who want to write their scripts in the jupyter notebook and want to do the same kind of analysis.

The workshop was closed by discussing and summarizing some important aspects of COSIMA:

1. Science Part:

  • Future scientific/technical ambitions for COSIMA?
  • Role of waves in the air-sea interactions and upper-ocean dynamics in the presence of ice.
  • Coupled data assimilation for parameter estimation?
  • ACCESS-OM-01 IAF run does not obtain the Hogg et at the increase in Pacific and Indian Ocean EKE.  Why is this so?
  • Better connections between COSIMA and paleo community?

2. Model Developments?

  • Future model configurations/model components.
  • Sea ice modelling- new model development [CICE 2, sea ice BGC, SIS2].
  • MITgcm (model set up and limitations? ) 
  • Should we migrate to CICE6?
  • Are their code improvements or experiment methods we could share better?
  • Resolving tides in the model.

3. Technical Part: 

  • Where are the current datasets and how to better organise it to assist the work of the ocean models?
  • Computational resources at NCI as a server limit on future model development.
  • Who loves cookbook? How to make cookbook easier? May be organised cookbook workshop (I am totally into it). 

In the end, I have enjoyed all the sessions, discussions with people in the field. I have met collaborators, we went through group talk about some new ideas about the project, that motivates me a lot. Especially, I am delighted becoming the new member of the COSIMA community. I have amazed by seeing how the collaborations, deep suggestions work,  to make a better research community. I am enthusiastic to attend more such events in future and looking forward to paving my way in the community.

        

Technical Working Group Meeting, August 2019

Minutes

Date: 14th August, 2019
Attendees:

  • Aidan Heerdegen (AH) CLEX ANU, Angus Gibson (AG) RSES ANU, Andrew Kiss (AK)  COSIMA ANU
  • Russ Fiedler (RF), Matt Chamberlain (MC) CSIRO Hobart
  • Rui Yang (RY) NCI
  • Marshall Ward (MW) GFDL
  • Nich Hannah (NH), Double Precision
  • James Munroe (JM), COSIMA

PIO work with CICE

NH: PIO code in CICE not as complete or thorough as netCDF code. Nothing to suggest it won’t work. Relies on NCAR PIO library, and a CESM utility library. Dependencies which are not part of CICE. Built PIO dependency on raijin, ran into CESM dependency. Can either remove dependency or remove code.
NH: Initially thought to use the MOM approach. Tile and collate. Russ’ comments encouraged to try PIO. Will be supported in future and will be supported in CICE6. Nothing working, but will soon test with 1 degree.
RF: Real bottleneck with high freq output. Worth a go. Attempt to put this into FMS by Hartnett. AH: Different to parallel netCDF? NH: PIO is wrapper around parallel netcdf. Written by NCAR to simplify parallel netcdf. Another layer. On GitHub, continuing to be maintained. RY: Wrapper that does work to match computing to IO domain. Not so useful for MOM5 as it has io_layout already.
MW: Harntnett motivated by FE3 (forecast model) rather than ocean. Not sure what project even involved in.
NH: Big test is handling interesting CICE layout, difference between cartesian grid and PE layout. MW: PIO will support explicit decomposition and other approaches.
NH: Parallel netCDF version on raijin only links with OpenMPI3.0. RY: New machine launched soon. OpenMPI 1.* will be dropped. No new software depending on 1. MW: OpenMPI 2 is not good. Should use 3.
NH: Probably have to test this with OpenMPI 3.0 RY: 3.1.3. Switch everything to that. Good test for new machine. AH: Working now? RY: My fault. Used unmatched openMPI library. Everything looks fine. OpenMPI 2/3/4 with Intel 19. All working. 1 deg & 0.25 deg working. Tenth not working. MW: I was able to run tenth with 3.1.2/3.1.3.
MW: One of the intel compilers broke MOM. A compiler bug with types in types.
AH: Should  start an issue for testing RY: Will email MW directly. RY: Not a MOM bug.
MW: Tried MOM-SIS tenth? Good test. RY: From earlier this year do have this working. This is testing for new machine, so ACCESS-OM2.

OMIP date restart protocol

RF: Talked to Griffies. GFDL take ensemble approach. Run for N years using true dates. At finish reset back to start date with correct calendar. Storing new stuff in different directory. End up with 5 sequences of 55 years. All dates are correct. No issues with leap years going wrong. Think this is the best way to go.
AK: Came to conclusion that this was right way to go, mostly due to leap year issue. Problem is, can we get the model to do that, but Maurice and Ryan had issues. Issue with CICE getting the correct date. CICE has a flag “use_restart_dates”. Suggested set this to false, and set the dates in access_restart.nml, but CICE is not picking up dates. Looks like libaccessom2 is not passing them on to CICE. Some confusion about exactly what they have done. Some instructions on Wiki for restarting, from restarting IAF from RYF at tenth, but doesn’t work for other people. NH: I’ll look at it. AK: Will send issue. NH: Didn’t realise it was happening. CICE date handling is not great.
AH: Downside with ensemble, difficult to get metrics across the whole time series. RF: Need extra meta-data added in. Maybe which cycle you’re in. An extra variable which gives the actual number of days since the start of the run. Down with post-processing. Might be able to concatenate files using extra meta-data. AH: Always have issues with missing leap years if it spans a century. But only daily is an issue. AK: Cookbook do something. MC: Pretend it is no leap? JM: Data looking at as time series? AH: Extra metadata, say offset day is a good idea. RF: Add buffer in netCDF file so don’t need copies. mppnccombine can add padding. usually done with nccreate, make sure the header has some space. hbuf?

Strategy for CICE updates for flexibly adding fields

RF: Way CICE drivers work, variables you want are either hard coded, or muck around with pre-processing to compile them in and out. Wondering if anyone looked at doing it on the fly. Using error codes coming back when setting up variables, so have flexible number of variables passed in and out. Would like this to pass total wind speed, to harmonise code. Also Hakase wants it for some BGC stuff. Phytoplankton through to the ice. So specify the variables, work out if they’re there or not.
NH: Would want the exe to handle configuration with different sets of coupling fields. Sometimes include total wind speed, sometimes not. RF: would know complete set, if not there skip it. Currently have to be hard wired in, or make another driver. NH: Way to do it, start with superset in namcouple, and code would exclude certain variables. RF: Maybe if variable not in namcouple, return an error code, but ignore error. NH: Shouldn’t be too hard to do. NH: OASIS does return error codes that could be used. Either abort or return error code. If aborting could change that. AH: Restart fields? NH: Should do behind the scenes.

Paths for JRA55-do forcing files. Some changes to support v1.4

AH: JRA55-do not part of Input4MIPs, part of CMIP6. Have to use the copy that is CMIP6. Encodes all the metadata in filename, consequently doesn’t currently work with YATM. Circumvented by creating symbolic links that worked with YATM. When I did this couldn’t reproduce. Not sure if this is actually an issue with the fields being different or not.
AH: Tried to use testing framework NH developed for this using jenkins. The historical test that tests against known checksums doesn’t seem to actually compare them. Not sure if that is intentional. Would like to use framework, as NH has done a great job with it.
MH: MOM6 has diag_mediator, supports CMOR name alongside internal model name. Porting to MOM5 is a big task, but idea is good and saved them a lot of work. Could create a thin wrapper to translate to CMOR name if that helps. AK: How integrate with YATM? MW: Don’t know. At FMS level, so only help with 1 model (MOM). AK: YATM access the JRA files. So libaccessom2 change. AH: Looked at YATM code. Generates filename form date. Input4MIPS has current year and next year, so would require code changes. Might just be easier to create a file with date->filename mapping? AH: Possible to do. Would need to add a token for year+1. Possible to do. Probably best to do it that way.
AK: Also need code changes with v1.4. Solid and liquid runoff are separate. What to do with solid runoff? Griffies either use iceberg model, or melt them and add them to runoff. Take account latent heat of fusion? Assuming solid runoff is at zero, which could be a problem. Put in a request to download v1.4. Scripts they have should automatically download it, but not. MW: Think GFDL only has v1.3.
MW: Fields go to end of 2017, is 2018 downloaded? Looking in wrong place? Looking in ua8. AK: Should look in qv56. AK: qv56 up to feb 2018. AH: If not automatically downloading, we should ask. What does the OMIP protocol say about end date? AK: JRA55 can find out about 2018. RF: It is specified, but would like latest for ongoing runs.

Testing FMS merge

AH: Putting FMS in as a sub-repo. Just needs testing. If it reproduces checksums for a month we’re sure it is ok? Is that sufficient?
NH: When Marshall upgraded FMS, went through every MOM test. Including 0.25. Can’t recall how strict we were. AH: Testing framework still there? NH: It is there. Because it never gets used, might be rotted a bit. Can give Jenkins URL of PR and it would do it. We should work together to get that working.

New NCI HPC hardware announcement

RY: System by end of the year. 2 phases, install new machine with Cascade Lake nodes. Short period gabi and raijin run simultaneously. After that skylake and broadwell will be merged with new machine and SandyBridge nodes removed. 100 GPU installed. 16 skylake k-80 nodes. PBS pro again. Storage and network infiniband. 200GB/s transfer speed. OS is CentOS 8. AH: Trying to figure out total core count for new machine. Do you know what core count will be? RY: Not clear on exact number. Can check with system guys if they know the exact number. If 32 cores/node, 150+K processors. AH: Will runtimes be extended for new machine. Find 5 hours too low for high core count jobs. Reduces flexibility. RY: Queue time limits are per project. Quite flexible. Contact NCI help. AH: Have asked for time limit changes in past, but usually time limited. RY: Have been asked by other users, not sure about the policy. Good time to ask and get a better policy for the new machine.