COSIMA 2018 Report

Aims & Goals

The third meeting of the Consortium for Ocean Sea Ice Modelling in Australia (COSIMA) was held in Canberra on 7-8 May 2018. This annual COSIMA workshop aims to:

  • Establish a community around ocean-sea ice modelling in Australia;
  • Discuss recent scientific advances in ocean and sea ice research in a forum that is inclusive and model-agnostic, particularly including observational programs;
  • Agree on immediate next steps in the COSIMA model development plan; and
  • Develop a long-term vision for Australian scientific advances in this area.

Participants

The 2018 workshop is our largest workshop yet, with 30 talks and 49 participants.

Attendees included:

Gary Brassington (Bureau of Meteorology), Matt Chamberlain (CSIRO), Chris Chapman (CSIRO), Fabio Dias (UTAS/CSIRO), Prasanth Divakaran (Bureau of Meteorology), Peter Dobrohotoff (CSIRO), Catia Domingues (UTAS), Matthew England (UNSW), Russ Fiedler (CSIRO), Annie Foppert (CSIRO), Leela Frankcombe (UNSW), Bishakhdatta Gayen (ANU), Angus Gibson (ANU), Stephen Griffies (NOAA/GFDL), Nicholas Hannah (COSIMA), Aidan Heerdegen (ANU/CLEX), Petra Heil (AAD & ACE CRC), Andy Hogg (ANU), Ryan Holmes (UNSW), Shane Keating (UNSW), Andrew Kiss (ANU), Vassili Kitsios (CSIRO), Veronique Lago (UNSW), Clothilde Langlais (CSIRO), Andrew Lenton (CSIRO), Kewei Lyu (CSIRO), Jie Ma (CSIRO), Simon Marsland (CSIRO), Paige Martin (University of Michigan), Josue Martinez Moreno (ANU), Richard Matear (CSIRO), Laurie Menviel (UNSW), Mainak Mondal (ANU), Ruth Moorman (ANU), Adele Morrison (ANU), Terry O’Kane (CSIRO), Peter Oke (CSIRO), Ramkrushnbhai Patel (UTAS), Paul Sandery (CSIRO), Abhishek Savita (UTAS-CSIRO), Kate Snow (NCI), Paul Spence (UNSW), Kial Stewart (ANU/UNSW), Veronica Tamsitt (UNSW/CSIRO), Mirko Velic (Bureau of Meteorology), Marshall Ward (NCI), Luwei Yang (IMAS, UTAS), Rui Yang (NCI), Jan Zika (UNSW)

Status

The workshop was structured to focus on scientific questions on Day 1, particularly in the first two sessions. In these sessions, topics ranged from from Antarctic shelf processes to oceanic convection, from reversibility of the Earth system to frictional drag. The final session on day 1 focussed more on technical issues, including assessment of the optimisation status of existing models. On Day 2, talks focussed more on strategic issues, including an outline of Bluelink, ACCESS, CAFE and coastal programs. These strategic talks transitioned to small-group discussions (see synthesis below). The workshop finished with a tutorial on the COSIMA Cookbook framework for model analysis.

The Australian landscape in ocean-sea ice research involves a number of interleaving programs, each of which was represented at this workshop.  The figure below outlines the linkages between these programs:

By way of explanation:

ACCESS-CM2/-ESM1.5 will be Australia’s input to CMIP6, and use MOM5 and CICE at 1°.

CAFE is the decadal prediction system in development, which uses MOM5.

ARCCSS/CLEX, ARC CoE programs, use high-resolution ocean-sea ice models for process studies.

Bluelink/OFAM is the ocean forecasting and reanalysis system which will adopt ACCESS-OM2-01 in future versions.

CSHOR is the Centre for Southern Hemisphere Oceanographic Research; it focuses on observational studies but we hope to establish two-way interactions with this program.

Coastal Modelling includes the Australian coastal oceanography community, as well as Antarctic nearshore programs within AAD and ACE-CRC.

A major theme of the workshop was to review the status of the ACCESS-OM2 model which is the focus of COSIMA. In short, we have had success with model releases at 1° and 0.25° resolution – these models are now actively being used for scientific runs, and are available for download and use by the community. They include a recent upgrade to the file-based atmosphere (YATM) and new JRA55-do forcing datasets. The 0.1° version of the model has progressed significantly in the last year; there are outstanding tasks to evaluate model output and further optimise the model configuration.

The COSIMA Cookbook tutorial was attended by about a third of participants, and some progress was made. The aim of this tutorial was to entrain more active users to the system and encourage input from those users. The Cookbook is similar in style to the analysis system being developed for CAFE and it may be possible to merge elements of each framework at some stage in the future.

Program

Where available, talk files are linked from the presenter’s name.

Monday 7 May
10:00 Arrival & Morning tea
10:30 Session 1 (Chair – Andy Hogg)
Stephen M Griffies (NOAA/GFDL): Understanding and projecting global and regional sea level: More reasons to include refined ocean resolution in global climate models
Andrew Kiss (ANU): Overview of the ACCESS-OM2 model suite
Andrew Lenton (CSIRO): Ocean Reversibility in ACCESS-ESM
Catia Domingues (UTAS): Global and spatial temporal changes in upper-ocean thermometric sea level
Fabio Dias (UTAS/CSIRO): Mean and seasonal states of the ocean heat and salt budgets in ACCESS-OM2
Adele Morrison (ANU): Circumpolar Deep Water transport towards Antarctica driven by dense water export
Jan Zika (UNSW): Getting an ocean model to obey: Prescribing and perturbing exact fluxes of heat and fresh water
12:30 Lunch
13:30 Session 2 (Chair – Clothilde Langlais)
Petra Heil (AAD & ACE CRC): ACCESS-OM2-01 sea ice
Paul Sandery (CSIRO): Sea-ice data assimilation and forecasting using an Ensemble Transform Kalman Filter
Paul Spence (UNSW): Does the Southern Ocean have sleep apnea?
Veronique Lago (UNSW): Impact of projected amplification of Antarctic meltwater on Antarctic Bottom Water formation
Ryan Holmes (UNSW): Numerical Mixing in the COSIMA Models
Luwei Yang (IMAS, UTAS): The impacts of bottom frictional drag on the sensitivity of the Southern Ocean circulation to changing wind
Vassili Kitsios (CSIRO): Stochastic subgrid turbulence parameterisation of eddy-eddy, eddy-topographic, eddy-meanfield and meanfield-meanfield interactions
Matt Chamberlain (CSIRO): Using transport matrices to probe circulation in ocean models
15:30 Afternoon tea
16:00 Session 3 (Chair – Petra Heil)
Nicholas Hannah (COSIMA): ACCESS-OM2 Software Development
Marshall Ward (NCI): ACCESS-OM2 performance analysis
Rui Yang (NCI): Parallel IO in MOM5
Angus Gibson (ANU): Towards an adaptive vertical coordinate in MOM6
Jie Ma (CSIRO ): Investigating interannual-decadal variability of Indian Ocean temperature transport in an eddy-resolving model
Paige Martin (University of Michigan): Frequency-domain analysis of energy transfer in an idealized ocean-atmosphere model
17:30 Close
19:00 Workshop dinner (Debacle24 Lonsdale St Braddon)
Tuesday 8 May
9:00 Session 4 (Chair – Andrew Kiss)
Andy Hogg (ANU): Are we Redi for 0.25° ocean-climate models?
Kial Stewart (ANU): The Repeat Year Forcing for JRA55-do
Terry O’Kane (CSIRO): Coupled data assimilation and ensemble initialization with application to multi-year ENSO prediction
Gary Brassington (Bureau of Meteorology): Ocean forecasting status and outlook
Peter Oke (CSIRO): Bluelink activities and plans
Matthew England (UNSW): A proposal for future projection simulations using COSIMA ocean-ice models
Richard Matear (CSIRO): CSIRO Decadal Climate Forecasting, update of the project’s progress
Simon Marsland (CSIRO): Preparing ACCESS for CMIP6
Clothilde Langlais (CSIRO): Downscaling towards the coast – a perspective on where the coastal modelling group would like to go
11:00 Morning tea
11:30 Discussion: COSIMA planning and strategy
13:00 Lunch
14:00 Strategy and planning summary
14:30 COSIMA Cookbook tutorial
16:00 Close

Synthesis of Discussion

Tuesday afternoon included discussions of present and future needs and directions of the COSIMA community, via breakout sessions on the topics Sea Ice, Coastal / Forecasting, Coupled Modelling, Process Modelling, Biogeochemistry, and Technical. The overall threads of  these discussions are summarised here.

Open and accessible code, configurations, output and analysis

Transparency, accessibility and reproducibility of model code development, run configurations and output data were named as priorities by many groups. Nic Hannah’s proposed REDB (Reproducible Experiment Database, http://redb.io) was widely supported as a means to tie together and curate the source code, configurations, output and analysis of model experiments. Using consistent shared codebases was also a priority. Containerisation was suggested as a method to make experiments self-contained. Extension of the database to include idealised experiments was also suggested.

Model evaluation

There is a need for more model evaluation against observations. Several groups highlighted the importance of better integration of observations for model validation and a desire for this functionality to be better supported in the COSIMA Cookbook. Comparison of CICE to SIS-1 at 1 and 1/4 deg was also suggested.

Technical validation is also needed – e.g. BGC, bit reproducibility, broadened test suite, regression testing. Model performance and stability priorities include: resolve crashes, balance load, MPI benchmarks and stress testing.

Usability

Suggestions included a glossary for beginners, an online portal for control runs, and to minimise difficulty of running new model configurations. Standardised output files and naming conventions would facilitate analysis. Improved functionality and versatility of the COSIMA Cookbook was also suggested.

Documentation was a priority for many, in particular an ACCESS-OM2 documentation paper, but also open/evolving documentation as the models develop.

Parameter selection was also a concern for many – how to choose appropriate parameters (e.g. for ice or BGC), how to assess model sensitivity to parameters, how to document why parameters were chosen or altered. Data assimilation was suggested a way to improve ice parameter selection, including assimilation of under-ice observations (e.g. temperature). BGC was suggested as a way to constrain the dynamics.

It was pointed out that the payu run management software underpins model runs, yet formal funding for its continued development is presently lacking.

Model enhancements

Suggestions for enhanced modelling capability included: interannual forcing, WOMBAT BGC, coupling to an atmosphere model, 1-way nesting, coupling to wavewatch, explicit tides, wet/dry cells.

Community coordination, synergies and strategy

Suggestions included a streamlined process for providing community feedback and deciding on priorities, and for community involvement in developing the BGC component. It was also suggested to foster engagement with atmosphere and sea ice specialists, and have a more formalized ice group. The technical team is also seeking more input from scientists, especially regarding sea ice.

Regarding modelling strategy, it was suggested to have intelligent model diversity (not too many versions), a consensus on standard perturbation experiments, and to decide on resources to commit to MOM5 vs. MOM6.

Summary of Priority Tasks

The following list of tasks was identified as a priority for the near term. Volunteers to lead or assist with tasks much appreciated.

  1. IAF Runs: With the addition of YATM, we now have the facility to run Interannual Forcing (IAF) runs from the JRA55-do forcing dataset in ACCESS-OM2. Once YATM has been tested, we will conduct IAF runs at all resolutions, starting with 1°.
  2. Model Documentation: Production of a model documentation paper is a high priority for the coming months. This will be achieved by:
    1. Writing a larger technical documentation report (https://github.com/OceansAus/ACCESS-OM2-1-025-010deg-report) that will be stripped down to feed into a paper; and
    2. Inviting community evaluation of existing model output.
  3. Model evaluation and analysis: We propose the COSIMA Cookbook as a framework for users to contribute model analyses. In particular, we encourage observational comparisons with existing model output, and also encourage users to submit bug reports and feature requests via https://github.com/OceansAus/cosima-cookbook/issues
  4. WOMBAT: In the coming months we will look to implement the WOMBAT biogeochemistry model (already running in MOM5) into the ACCESS-OM2 framework.
  5. Capability gaps: The COSIMA community has been able to leverage expertise from a number of different programs. However, our community as a whole remains subcritical in several areas, including sea ice modelling and atmospheric dynamics.
  6. REDB: Nic Hannah proposed a new system for tracking simulations and the output data. This system was identified by many discussion groups as a potential solution to some of our collaboration roadblocks. We will investigate the viability of such a system.
  7. MOM6: Plan is to begin transition to MOM6, building up experience in the latter half of 2018.

Recommendations for COSIMA 2019 workshop

  • Institute a James Munroe award for contributions to COSIMA
  • Extend to a 2.5-day workshop to allow more time for discussion (not extra talks)

Technical Working Group Meeting, June 2018

Minutes

Date: 12th June 2018
Attendees:

  • Marshall Ward (MW) (Chair), NCI
  • Aidan Heerdegen (AH) and Andrew Kiss (AK), CLEX ANU
  • Russ Fiedler (RF), Matt Chamberlain (MC), CSIRO Hobart
  • Peter Dobrohotoff (PD), CSIRO Aspendale
  • Justin Freeman (JF) BoM Melbourne
  • Nic Hannah (NH) Double Precision

TWG Meeting

JF:  Would be able to attend more regularly if there was a calendar invite which would enable him to schedule the meeting. How do we integrate calendars for Justin

COSIMA Models

AK: Bathymetry error in tenth model in Cumberland Sound, Baffin Island. Causes model blow ups.
RF: Yes blast it out. Russ will do it today. AH: Do we need any changes to restart/input files? Russ: if below zero  for eta_t, might have to set to zero. Otherwise will complain about penetrating rock.
AK: tenth very unstable over the weekend.
MW: longjmp error means the backtrace is failing. Memory go so severely corrupted that can’t properly debug.
“nearest_index array must be monotonically increasing error”
AK: Sweep and resubmit and works.
AK: More errors since turned on diagnostics for Adele. RF: are these globals? MW: could be FMS bugs because MPI is being strained and things are out of order.
AK: daily outputs in regional area: temp, salt, uhrho_et, vhrho_nt, rho_dzt. RF: spewing output from a lot of processors as regional outputs do not use io_layout, so every affected processor outputting data. AK: only doing for 2 years and then turn it off. It has slowed it down. Become erratic in timing. RF: Some processors not outputting the field, not sure why it should make it unstable.
AK: Put up as an issue.
AH: What is the current model config for tenth, and performance? AK: 4.5K on MOM, 2K on cice. Runs with 450s timestep, 1.5 hr/mo. Now running at 400s. Crash in Baffin Island goes away with shorter timestep.
AH: Try and get tenth running faster. Ice no longer holding back timstep. AK: Was running 540s before Baffin island issues.
MW: netCDF4 v4.4 has FPE turned on. Built by a different person. Historically always had FPE disabled. AK: 4.2.1.1 in MOM. 4.3.2 in CICE. 4.4.1 in matm. OASIS has default. AK: waiting for yatm build to be signed off. Ben M suggested we should be using openmpi/1.10.7 (optionally with debug). Number of bugs fixed between 1.10.2 and 1.10.7.
AK: Want to try out orange layout with CICE. Currently 2000 cores with no landmasking and 1 block / processor. Could be run a lot cheaper. Currently MOM bound. Should be to run well below 2000 cores. Waiting for yatm to be sorted out. Trying some frankenstein builds and back porting to matm.
AK: Timing is very inconsistent. RF: Ocean eta and plot diagnose has a collective. Does a sum. Somewhere it has hung. All depend on this function. MW: Could be load imbalance in CICE.
MW: MPI_Comm_split hangs or fails intermittently.
AK: No stock runs since looking at runtime. MW: thinks his profiling was wrong because of lack of ice. AK: looking at the load imbalance there is ice. MW: ran from rest for 10 days. CICE would normally do work that wasn’t captured. MW: tried to redo profiles and all runs stopped working. Shocking.
MW: moved on to yatm. Putting scorep into yatm had issues, so not redone the profiles with realistic ice.
AK: will spin off run with no diagnostics as point of comparison. MW: at dt=300s, 100s/day seemed reproducible. Andrew’s 50% slower. Maybe more stuff happening. One of two issues that need to be resolved. CICE bound results different, second is MOM slowdown. Matching MOM-SIS important goal.
MC: how much longer running spinup? When switch to IAF? AK: will switch to IAF ASAP. Andy is running RYF @ quarter. Then Paul Spence will run IAF quarter. Currently 34 years of spin-up with 84/85 repeat year. MC: Will start from year zero? AK: there are biases in RYF, so not sure if we should spin off from this run. Might depend on how many years we have to get done.
MC: will there be multiple cycles of IAF? AK: depends. MC: start at WOA or from RYF spin up.
AH: For the model documentation paper there will be the standard 5 x IAF (JRA55) protocol for 1 degree and 0.25 degree. The MOM meeting discussed strategy for 0.1 degree. Andy Hogg thought the tenth was just too expensive to run this protocol and might have to run only one cycle of IAF, or maybe spin up with RYF and then run IAF from 85 onwards. Whatever was done would be repeated in a second quarter degree run to provide a point of comparison between the different resolutions.
RF: interested from 93 once the satellites go up.
JF: wanted to get up to speed. Looks over minutes when they come out, very useful. Mirko has been doing some runs. Will try and join in regularly. BoM will take up ACCESS-OM2 when up to speed. Will be OceanMaps version, used for forecasting.
AH: Andy running KDS50 for 0.25 deg for RYF spin up. Found KDS75 too unstable.
JF: Mirko is testing COSIMA models in back end. Mirko getting up to speed what we’ve done. Need the 75 level (COSIMA) grid. Will do some hindcast runs and compare with OceanMaps. Don’t have experience with sea ice model . Don’t know how it will affect forecasting. Need to look at the ice parameterisation. Also need to look at data assimilation. Will talk to Russ and Matt. At some point will be able to contribute back, will work from GitHub repo, using same codebase.
AK: run parameters and namelists on git repo are a long way out of date. JF: can we make sure these are updated. AK: Still in a state of flux. Still bedding down YATM configurations. Will do best.

ACCESS OM2/CM2 Code Harmonisation

AH: What is the other significant code difference in CM2 that Russ wanted to reimplement? RF: wave mixing scheme. Gets added into KPP. Comes via CVMix package. Two ways to implement. 1. 10m winds to come in via sbc. 2. Can empirically calculate them in MOM6. Russ has implemented this scheme under CM2.5 framework. Run for a while. Had to put in a limiter because it caused too much mixing. Dave reckoned it didn’t make difference. Haven’t looked at the most recent results. Running with CM2.5 coupled model.
RF: Also another scheme Russ wants to implement. Slightly different to ACCESS-CM2. Both schemes already in MOM6. One of them is in CVMix. That is what Dave has implemented in MOM5. Taken routine out of CVMix and plopped it into KPP module to give enhanced mixing. Also need 10m wind information to come in. Need changes in surface flux code. Russ has done this. Russ has implemented same thing, just change in the way winds get through. Not sure why ACCESS-CM2 didn’t see difference.
RF: Occasionally get massive mixing coefficients in KPP so put in a limiter.
RF: will put code changes into master branch. AH: when you have done this I can pull into CM2 and can test. RF: Griffies wants it in MOM5.
PD: followed along in slack channel. Not sure about all technical details. Big difference after 10 days between harmonised code and CM2 codebase. Has this been solved? How far along are we with this? Spinups will not have harmonised code if we don’t have a frozen version soon. ESM and CM2 groups want to know how close we are. We haven’t helped much to this point. How can I contribute.
PD: copied suite. Ran it. Thought was tracking down bug. PD: couldn’t find preprocessed source files. MW: do we run cpp? I get the right source code lines and don’t see .f90 files. No we don’t … which is why Peter couldn’t find them.
RF: why was red sea fix timing different? CM code has a fix? AH: might be because my fix uses relative time, not absolute model time. RF: timing fix should have absolute origin. AH: I’ll check.
AH: I don’t think there is that much more to go for the harmonisation
PD: when can I run harmonised MOM?
RF: when I can find some time to put in there. Now we have a way forward. Hopefully in a week or two.
PD: will put runs on ASAP. If harmonised code not ready, won’t be in spinups.
AH: will lease with Peter and tell him as soon as something is ready.
MW: if there are differences what do  they use? AH: they will use the MOM5 repo as far as I know.

Actions

New:

  • Edit tenth bathymetry to remove Cumberland Sound (RF)
  • Create calendar invites to TWG Meeting (AH)
  • Update model name list and other configurations on OceansAus repo (AK)
  • Check red sea fix timing is absolute, not relative (AH)

Existing:

  • Shared google doc on reproducibility strategy (AH)
  • Follow up with Andy Hogg regarding shared codebase (MW)
  • MW liase with AK about tenth model hangs (AK, MW)
  • Pull request for WOMBAT changes into MOM5 repo (MC, MW)
  • Compare out OASIS/CICE coupling code in ACCESS-CM2 and ACCESS-OM2 (RF)
  • After FMS moved to submodule, incorporate MPI-IO changes into FMS (MW)
  • Incorporate WOMBAT into CM2.5 decadal prediction codebase and publish to Github (RF)
  • Profile ACCESS-OM2-01 (MW)
  • Move FMS to submodule of MOM5 github repo (MW)
  • Make a proper plan for model release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model (MW and TWG)
  • Blog post around issues with high core count jobs and mxm mtl (NH)
  • Look into OpenDAP/THREDDS for use with MOM on raijin (AH, NH)
  • Add RF ocean bathymetry code to OceansAus repo (RF)
  • Add MPI barrier before ice halo updates timer to check if slow timing issues are just ice load imbalances that appear as longer times due to synchronisation (NH).
  • Nudging code test case (RF)
  • Redo SSS restoring with patch smoothing (AH)
  • Get Ben/Andy to endorse provision of MAS to CoE (no-one assigned)
  • CICE and MATM need to output namelists for metadata crawling (AK)