COSIMA 2 Workshop Report, May 2017.

The second meeting of the Consortium for Ocean Sea Ice Modelling in Australia (COSIMA) was held in Sydney on the 25-26 May 2017. There were 36 attendees (2 via video, 26 gave presentations) representing 9 different institutions. A full participant list and links to presentations is included at the end of this report. The workshop goals and agenda is here.

The consortium has 3 years remaining of 4 years funding from an ARC Linkage Project to build a model configuration which underpins a variety of applications. The consortium also aims to provide a lasting platform that engages and benefits all ocean-sea ice scientists and technicians in Australia. This report begins with an assessment of milestones achieved since the May 2016 meeting, followed by a list of the 2017 workshop highlights and discussion/action items to be undertaken in 2017/2018.

Assessment of  COSIMA Community Action Items from 2016 Workshop

  1. Need to create a website to outline COSIMA activities and developments. [We are in the process of acquiring the domain name cosima.org.au, and ANU will fund hosting and a web development team to put together a skeleton site.]
    • DONE (cosima.org.au).
  2. We will formalise a code of ethics for COSIMA users to abide by, based on the DRAKKAR agreement. [Spence]
    • DONE (http://cosima.org.au/index.php/about-2/ethics/) 
  3. We aim to register members on the website, and create a mailing list.
  4. We will need to formalise the use of technologies to share code configurations, analysis tools and data. It may be possible to have a data project code on NCI to help with this.
    • Substantial Progress: i) hh5 storage for COSIMA model data at nci; ii) cosima cook book github repository in use for data processing scripts; iii) git hub model code repositories in use for some configurations.
  5. We will release flagship configurations that are broadly supported by the community. The goal is to make the naming conventions consistent with the ACCESS community where possible, and to overlap with ACCESS developments where possible.  
    • Substantial Progress. New flagship configurations are under active developement: e.g. JRA-55 forcing of MOM-SIS and MOM-CICE configurations at multiple resolutions, including enhanced vertical resolution. Information regarding flagship simulations will be posted on website this quarter [Spence]. 
  6. COSIMA will hold an annual meeting in the last week of May, for two days. Venue will rotate around the partner institutions. The focus of the meeting will be on science applications of ocean models, but will also include a technical component.
    • DONE. Next meeting in first week of June, 2018, Canberra.
  7. We will aim to have more regular communications, including newsletters and video meetings.
    • Substantial Progress. Open COSIMA video conference meetings were held monthly. Linkage project CI meetings occurred quarterly. Regular Technical Working Group meeting were held. Meeting notes are often posted at cosima.org.au.
  8. We will establish working groups within the community, along with a working group chair. Proposed groups include
    • Sea Ice Modelling [Heil]
      • Limited activity from this group in 2016
    • Technical [Ward]
      • Best communicating group that holds regular meetings, posts notes, has active community and achieves results.
    • Linkage Project [Hogg]
      • Quarterly meetings are held to gage model development progress and funding usage.
  9. The major gap in the community was identified to be sea ice modelling and forecasting. We will all look for opportunities to attract visitors and expertise in this space.
    • Needs more work. 

Assessment of  ARC Linkage Project Action Items from 2016 Workshop

  1. There is a strong need to consider vertical resolution in future high resolution model developments (Stewart).
    • DONE. http://www.sciencedirect.com/science/article/pii/S1463500317300434
  2. Our MOM-CICE implementations using OASIS3-MCT coupling, but it is not clear that this solution will scale to 0.1° and beyond (Hannah)
    • Substantial Progress. Parallel OASIS-CICE now runs as fast as serial FMS-SIS with MOM5. Uncertainties remain regarding MOM6 coupling.
  3. Forced ocean-sea ice models should be transferring to the JRA-55 forcing set when possible (Marsland)
    • DONE. JRA-55 forced configurations are underway in multiple groups. Fabio Dias is running coarse resolution MOM5-OASIS-CICE simulations. The CoE Oceans Group is running MOM5-FMS-SIS and MOM5-OAIS-CICE simulations at multiple resolutions. A relative year forcing data set has been developed from JRA-55 (contact Kyle Stewart)
  4. There was general agreement that the Linkage Project should fund both the development and evaluation of new model configurations. This point implies that we should equally fund the technical and postdoctoral position, despite the partial funding of the program.
    • DONE. Hannah continues to excel at developing the model Linkage project model configurations. A scientist has been hired (starting Aug., 2017) to help develop and evaluate the model simulations.  
  5. In the first year we will look to upgrade current MOM5 implementations, focussing on the vertical grid and the incorporation of CICE.
    • Substantial Progress. A 1/10 degree 75 level ACCESS-OM2 configuration consisting of MOM5 and CICE5 is being developed. Several scalability and performance problems have been overcome and we are now completing a spin-up run. The work has also resulted in improvements to the lower resolution 1 and 1/4 degree configurations and ACCESS coupled models.
  6. In subsequent years we will look to adopt a MOM6 configuration.
    • Progress. Testing with MOM6 adaptive vertical coordinates is underway (Angus Gibson). Given that SIS2 is adopting much of the CICE5 code, uncertainties remain regarding the sea ice and coupling model to be used with MOM6.
  7. As configurations develop and have been properly evaluated, they will be distributed to the community.
    • Substantial Progress. The distribution platform is under development with substantial progress on the back-end. A COSIMA data repository on NCI exists, and currently houses several flagship simulations. A repository for python analysis code that accesses the data repository is in use (https://github.com/OceansAus/cosima-cookbook). Substantial progress has been made in adopting effective big data analysis techniques (James Munroe).

2017 Workshop_Highlights

  • Substantial progress in the development and analysis of coupled WOMBAT biogeochemistry simulations.
  • Exciting opportunities within the newly funded CSIRO Decadal Prediction team.
  • Our technical teams continue to excel at developing and optimising model configurations with effective collaborations. Hannah has made excellent progress in MOM-OASIS-CICE coupling. Ward provides a key conduit to understanding NCI optimisations. Heerdegen is supporting a multitude of operations. Fiedler continues to develop the new bathymetry.
  • The enhanced capabilities of CICE over SIS remain. CICE6 is being developed under a community framework. SIS2 has adopted some of the CICE code.
  • ACCESS is working towards the CMIP6 deadlines.
  • The blue link forecast models are now perceived to be limited more by the ocean model configuration than the data assimilation.
  • New model configurations (MOM5-OASIS-CICE, MOM5-FMS-SIS) using enhanced vertical resolution, JRA forcing (including an RYF forcing), and updated WOA initial conditions are underway.

COSIMA Community Discussion/Action Items from 2017 Workshop

  1. Develop website into more of a community portal to flagship simulations. Provide model configuration and analysis with low barrier to entry for users. (Spence)
  2. Develop the COSIMA Cookbook portal  with a gallery of options (Munroe). Improve the community usage and contribution to the cookbook. Provide a visual gallery of code examples (ALL).
  3. Provide online chat capability for COSIMA model configurations. Look into using Slack teams. (Spence)
  4. Provide list of ongoing experiments, links to data available and diagnostics wanted, contacts for expts. (Spence)
  5. Consider separating models configs into Tier 1 (control flagships), 2 (perturbations from control), 3 (parameter sensitivities) to aid communication. (Spence)
  6. Provide discussion platform for new experiments. Wider discussion of methodologies and diagnostics.
  7. Provide platform for efficient explorations of parameter spaces (sea ice, neutral physics, bgc params, SSS restoring). How to share configurations? Tracking configs with git needs to be embraced by all! (Aidan, Hogg)
  8. Explore CICE efficiency, coupling a wave model to CICE, CICE6-MOM6 capabilities. Be aware of similar progress being made at NCAR. (Heil).
  9. Consider hiring admin support for the COSIMA website (England).

ARC Linkage Project Discussion/Action Items from 2017 Workshop

  1. A new 1/10 degree model bathymetry data set continues to be underdevelopment with substantial progress towards a final product expected with a few months (Russ Fiedler)
  2. Hannah continues to develop the MOM5-OASIS-CICE5 configurations. Expects to provide spinups at 3 model resolutions to the communitu (1, 1/4, 1/10). Discussion regarding the CICE settings (Heil), vertical resolution (Stewart), JRA forcing (Stewart) are needed. Does the model currently have a closed heat budget (Dias)?
  3. Excited to welcome Andrew Kiss as the scientist funded from this ARC Linkage grant. Hoping for a model evaluation paper before the next meeting.

 

Attendees

(Where available, talk files are linked from the presenter’s name.)

Andy Hogg (ANU)
Gary Brassington (BoM)
Nic Hannah (Breakaway Labs)
Dave Bi (CSIRO)
Matthew England (UNSW)
Adele Morrison (ANU)
Ryan Holmes (UNSW)
Jan Zika (UNSW)
Angus Gibson (ANU)
Andy Kiss (ANU)
Veronique Lago (UNSW)
Laurie Meviel(UNSW)
Kaitlin Alexander (UNSW)
David Webb (UNSW)
Peter Dobrotoff (CSIRO)
Leela Frankcombe (UNSW)
James Munroe (Memorial, Canada)
Joan Llort (Utas)
Arnold Sullivan (CSIRO)
Xuebin Zhang (CSIRO)
Terry Okane (CSIRO, Remote)
Peter Oke (CSIRO)
Paul Spence (UNSW)
Richard Matear (CSIRO)
Clothilde Langlais (CSIRO)
Stephen Griffies (GFDL, Remote)
Russel Fiedler (CSIRO)
Marshall Ward (NCI)
Matthew Chamberlain (CSIRO)
Fabio Dias (UTas)
Aidan Heerdegen (ANU)
Petra Heil (AAD & ACE CRC)
Siobhan O’Farrell (Remote, CSIRO)
Xiaobing Zhou (BoM)

Technical Working Group Meeting, May 2017

Minutes

Date: 16th May 2017
Attendees:

  • Marshall Ward (NCI, Chair)
  • Aidan Heerdegen and Andy Hogg (ARCCSS ANU)
  • Scott Wales (ARCCSS Melbourne Uni)
  • Nicholas Hannah (ARCCSS/Double Precision)
  • Russ Fiedler and Matt Chamberlain (CSIRO Hobart)
  • Justin Freeman (BoM)
  • Peter Dobrohotoff and Roger Bodman (CSIRO Aspendale)

Updates

  • Roger is having issues with N95 atmosphere. Marshall says it doesn’t scale past 256 cores. Roger would like to get RTM profiling working. Martin and Peter have got multiple threads working in AMIP. Has spoken to Scott Wales. Something odd happens in decomposition. Marshall will ask Dale Roberts about effects of OpenMP. Has chemistry has been enabled for OpenMP?

Liaison with COSIMA Management

  • Andy is here to get some feedback about our activities. Andy thinks TWG is doing a good job of communicating, a goal is to get more communication amongst COSIMA in general. Management team meets quarterly. No science talk or detail. Have to be better at merging/converging disparate code, TWG is crucial for this.
  • Andy wants a better framework for analysing and post-processing runs, and access others outputs. And we are currently doing some work in this space. The effort it is scattered, no one lead. Nic has done some work in past. Now using ipython notebooks to share analysis. James Munroe working on dashboards. Justin asked if COSIMA will deliver this? Andy: not explicitly funded but will benefit uptake. Need better ways to serve data.
  • Andy would like more engagement around COSIMA website. Blog our progress? Marshall: do you have sample topics? Is it legitimate to put updates on issues with library versions say? Andy: borderline, maybe just TWG for that example. Currently reply on members of TWG to propagate information back to users. Maybe more relevant would be update on scaling of code for example. Don’t want to limit people now. Encourage as much as possible and filter if required. Peter agreed: get useful information up there. Haven’t had anything over the last year. Suggestion to make minutes more report like? Nic thought blog posts are a nice idea, but need deeper insight to be useful and interesting.
  • Andy wants science side to publish results of runs, and point to data.
  • Andy also keen for COSIMA to have information about model versions. ACCESS doesn’t have a way of releasing versions and hosting code. ACCESS is somewhat hobbled with partner disputes. Would like ACCESS-OM releases on COSIMA. Marshall pointed out the TWG was set up to address this. Models were not ready at that stage. Marshall suggested we make a proper plan for a release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model.
  • What are the expectations of TWG for the COSIMA Meeting? Andy: some already giving talks. Interact and discuss with others. Get to know each other and ambitions and look for synergies.
  • Andy will contact Paul to make sure TWG will have a slot to fill in others on progress.
  • Andy would like an email list for COSIMA announcements.

COSIMA Models

  • Nic: Was in Canberra week ago. Had tenth timestepping on <3000 cores. More than 3000 didn’t initialise. MCT couldn’t set up routing tables with more than 3000 cores. Would just hang. After discussions figured out some MPI switches and flags to get it working: mxm mtl makes it work better. Justin suggested this would make a good blog post. Marshall found MOM6 was failing at 3000 cores too. Went away with mxm mtl.
  • Andy: MOM-SIS tenth was also failing. About 30% fails. Russ has had similar. Nic now running on 6K cores in ocean.
  • Nic had discussions in Canberra around CICE halo updates. Made 12+ changes to CICE and MATM code. Made big improvements. For all three model resolutions (1,0.25,0.1) overhead of coupling is 1-2% compared to MOM solo. That is a tiny serial bit of interpolating forcing fields on to ocean. Like 20s/month for quarter degree.
  • Nic: quarter degree 1800s time step should be less than 75min/year. Andy: UNREAL!. 1 deg is also running super fast. 50 years/day. Did a new compile on MCT library to squeeze as much performance as possible.
  • Now have 3 new configs. 1 deg and 0.25 deg could be used. Focussing on tenth at the moment.
  • Andy: 70min MOM-SIS-025  + CORE, JRA55 adds 30%.
  • In old config, all models block waiting for MATM to read files. Now MATM has sent everything. Reduces difference between CORE and JRA55. Nic has not done longer runs as yet. No longer buffering multiple years of MATM output.
  • Agreed Andy should get these configs and do some longer runs.
  • Andy talking to NCAR about JRA55 forcing. CORE used NYF. JRA doesn’t have that. Others have used a single year. Our strategy is May-May forcing with a shock at the end of May. Candidate years are 84/85, 91/92, 03/04. Want to test this at 0.25 deg and 1deg. Should adopt just what Nic has done.
  • These are MOM-SIS as Andy wants a baseline. Doing MOM-SIS from CORE with WOA13. Repeat with JRA55 RYFs. Want to compare to ACCESS-OM config of Nic.
  • Nic and Andy to talk offline and try out a repeat year.
  • Although it is fast, the tenth is inefficient, as there is no ocean masking currently. This is the next priority. Probably beyond Nic’s current contract.
  • Nic can I use unmasked restarts? Russ: yes. Just need to combine them.

COSIMA Workshop

  •  Marshall: can we agree to transfer to CM2? To get on common version of CICE.
  • Andy: will the code we release in OM2 be different in ocean and/or ice? Can we manage it in one codebase? Marshall: should be possible. Set aside time to discuss this at meeting.
  • Discuss moving to common CICE repo for all.
  • Marshall: Justin need some info from us on OM config? Nic and Justin will liase.
  • Justin won’t be at the COSIMA Meeting, does he want us to cover anything? Justin: staying up to date with what we’re doing, will be engaging much more in future.

Updates on previous actions

  • Nic: has updated OceansAus repo to Peter’s CICE. Can Peter look at the code and check it.
  • Marshall: Justin need some info from us on OM config? Nic and Justin will liase.
  • Russ has been doing a lot of clicking for bathymetry. Aus and PNG done. Need help.

Actions

New:

  • Ask Dale Roberts about effects of OpenMP for Roger (Marshall)
  • Make a proper plan for model release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model (Marshall and TWG)
  • Contact Paul Spence about TWG speaking slot at meeting (Andy)
  • Prepare slides for TWG presentation at COSIMA meeting, and present (Aidan and Marshall)
  • Email list for COSIMA announcements (Aidan)
  • Blog post around issues with high core count jobs and mxm mtl (Nic)
  • Do longer runs with Nic’s 1 deg and 0.25 deg ACCESS-OM2-JRA55 configs (Andy and Aidan)
  • Try repeat year forcing with Nic’s configurations (Nic and Andy)

Existing:

  • Nic to present MATM code re-write proposal to TWG for feedback before sign-off. Will then be presented to Andy Hogg for approval.
  • Nic create a discussion document (on COSIMA?) to document current approaches and strategies for future
  • Move FMS to submodule of MOM5 github repo (Marshall). Liase with Nic on implementation?
  • Test Nic’s access-om model config on OceansAus (All)
  • Work up test cases to cover the nudging code (Justin, Mirko) and supply them to Nic.
  • Add new test cases to Jenkins test suite (Nic).
  • Start a new google doc about coupler issues and MATM (Marshall)