Technical Working Group Meeting, April 2018

Minutes

Date: 10th April 2018
Attendees:

  • Marshall Ward (MW) (Chair), NCI
  • Aidan Heerdegen (AH) and Andrew Kiss (AK), CLEX ANU
  • Russ Fiedler (RF), Matt Chamberlain (MC), CSIRO Hobart
  • Peter Dobrohotoff (PD), CSIRO Aspendale

COSIMA Meeting

MW: Will we present something again? Same as last time, a list of achievements? Consensus was yes we should.
MW: Want to make time during the meeting to work together on collective problem? NH: Yes, definitely. Even for just a short time.

Common codebase

AH: Have rebased PDs MOM code on to latest MOM5 source. Wrapped all PDs changes that are incompatible in ifdef ACCESS_CM statements. This is available as branch cm2 on MOM5 GitHub repo. Created a pull request to allow code review (https://github.com/mom-ocean/MOM5/pull/214).

AH: PD has provided a rose suite (u-aw048) for testing. AH successfully copied (u-aw405) and ran this suite. Created another suite (u-aw445) to test that this reproduces first copy with no changes. It does. Created a third suite (u-aw497), changed git URL to point at cm2 branch, but initial compile failed to find the source. Eventually got it to recognise the updated source, now compile fails due to an absence of a main routine. Needs some modification.

PD: Best to make small incremental changes to a suite. In this case just change the fortran files and see if it works. Avoid changing how the compile is done, definitely avoid changing rose app conf. Was just trying to determine the compile flags use in UM fcm build from Met Office was not a trivial task.

MW: cylc is good, small and configurable. rose is difficult and opaque.

AH: Will get some help from others via the cm2-om2-harmonisation slack channel

AH: Next step is to do the same for CICE5 as has been done for MOM5.

NH: CSIRO is getting CICE from the UKMO. There are code changes under src, not just in the drivers.

AH: Does UKMO version of CICE5 have a special licence? Will we be able to host UKMO modifications to CICE5 on OceansAus repo? NH: CICE has a CICE licence

 MW: UKMO runs CICE as a NEMO/CICE5 executable, not linked through OASIS like us.

Models

PD: ACCESS-CM2 doesn’t reproduce over restarts. Would like to run CICE stand alone. Does CICE reproduce over restarts?

NH: cleanest way to test is in the coupling code, before any model sends anything to another model, checksum fields. That is the point you can compare the output of a model. In the case of restarts, can include in checksum what current model run time is.

MW: there are 2 restarts, individual models and oasis restarts. Have to make sure trigger the same number of time steps in models.

NH: OASIS restarts are not a problem. OASIS get tells you if it read out of a restart. Not reading out magically, still working using PUT and GET. GET is from a file instead of from a model.
After each GET and before each PUT, print out time, processor, checksum can see when the checksums diverge. Will be different before the PUT, can then identify the model. If you determine CICE was the culprit can then look at CICE only run.
PD: How to do checksum? NH: Just sum whole array. MW: use MPP_checksum? NH: not in CICE. Our CICE code will output checksums if required. NH has already done this for CICE.
MW: does OM2 reproduce? NH: I spent some time on this. Repro for a couple of coupling time steps, then diverges.
MW: PD has 1×2 days not same as 2×1. Have we done equivalent tests with OM2? NH: yes, not passing.
NH: MOM by itself might not do this anymore? MC: restarts ok. RF: if you had the redsea bug it wouldn’t reproduce. Old repro results null and void. Models also have coupling code that might cause repo issues.
MW: UM passes, would think GC3 reproduces.
MW: interesting OM2 does not reproduce. Easier platform to test.  ACCESS-om2 needs to reproduce. NH: looking at this. With MATM changes, needs to make sure it works to get others to use it.
MW: Does someone want to check MOM?  Restarts, processor layouts. AK: don’t change layouts often so wouldn’t know if it does currently reproduce with layout changes. NH: non repo with layoutt changes indicates bug. MW: maybe not bug, but definitely volatile behaviour, maybe in a collective.
MW: Did find a repo problem with MPP_sum. Ran MPP_sum and MPP_reprosum, and got difference in one bit. Even something that simple can cause issues. GFDL always matches with same test. Maybe something we can control with compiler flags.
PD: As voltages go down can get make random errors occur. MW: Bob found bug with tridiagonal solver due to voltage issue in Intel chip. Maybe something going on with flags?
RF: GFDL definitely use precise option. Atmospheric model crashes otherwise. MW: MetOffice also uses precise.
MW: Maybe all could look more carefully?
AH: Do we have a reproducibility checklist? Some strategy. Shared google doc?
NH: starting work on tenth degree performance. Anyone interested in doing some profiling? MW: Any Hogg pressuring to do this. Will do it this week, and send to NH. Hope to have a bunch of profiles for the meeting.
NH: MATM is now clean, 100 lines of code, uses CMake. Hoping to start using it. All goes through CICE. Nothing about coupling has changed.
NH: Want to use newest version of OASIS-mct (v3 not v2). Improvement in performance, can collect together MPI comms.

Actions

New:

  • Poll TWG on list of achievements for Meeting presentation (MW)
  • Shared google doc on reproducibility strategy (AH)

Existing:

  • Follow up with Andy Hogg regarding shared codebase (MW)
  • MW liase with AK about tenth model hangs (AK, MW)
  • Pull request for WOMBAT changes into MOM5 repo (MC, MW)
  • Compare out OASIS/CICE coupling code in ACCESS-CM2 and ACCESS-OM2 (RF)
  • After FMS moved to submodule, incorporate MPI-IO changes into FMS (MW)
  • Incorporate WOMBAT into CM2.5 decadal prediction codebase and publish to Github (RF)
  • Profile ACCESS-OM2-01 (MW)
  • Move FMS to submodule of MOM5 github repo (MW)
  • Make a proper plan for model release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model (MW and TWG)
  • Blog post around issues with high core count jobs and mxm mtl (NH)
  • Look into OpenDAP/THREDDS for use with MOM on raijin (AH, NH)
  • Add RF ocean bathymetry code to OceansAus repo (RF)
  • Add MPI barrier before ice halo updates timer to check if slow timing issues are just ice load imbalances that appear as longer times due to synchronisation (NH).
  • Nudging code test case (RF)
  • Redo SSS restoring with patch smoothing (AH)
  • Get Ben/Andy to endorse provision of MAS to CoE (no-one assigned)
  • CICE and MATM need to output namelists for metadata crawling (AK)

2018 COSIMA Workshop

The 2018 COSIMA workshop will be held at ANU on 7-8 May.

As has been traditional with this workshop, Day 1 (Monday 7th May) will focus more on scientific aspects of our modelling, starting at about 10am and running through to 5pm.

Day 2 (Tuesday 8th May) will be more technically oriented, including a workshop on the use of the COSIMA cookbook. We will finish by 4pm to allow for those visitors who need to fly home.

Any questions, please contact Andy Hogg. A full program will be posted here shortly.

Sea-ice working group: Notes 20180228

MEETING NOTES COSIMA SEA-ICE WORKING GROUP MEETING:

28 Feb 2018, 10:00 - 11:30am
Meeting held on Zoom.

Present: Andrew Kiss, Adele Morrison, (anyone else at ANU?), Andrew Hogg, Siobhan O'Farrell, Simon Marland,
         Fabio Diaz, xxx (1 other PhD student at CSIRO Aspendale), Will Hobbs, Petra Heil and from 11am Paul Spence.
Apologies: Nicolas Hannah, Matt E.

=======================================================
1) Recap of the AMOS presentation on COSIMA results so far.
ACCESS-OM2 suite (Andrew Kiss)
Based on MOM5, CICE5, OASIS3, Repeat year atm forcing.
Tripolar grid 1/10^o, 75 vertical resolution
Slow to run: 17hours/yr; 5559 PEs; dt = 450s   --> eddy resolving
Tripolar grid 1^o, 75 vertical resolution
              6mins/yr;  252 PEs; dt = 5400s   
------
Previous issue: 
Negative solution: Russ Fiedler fixed
         If salinity locally is less than 5 PSU, then salinity comes from the 
         surrounding.
------
Sea-ice thickness (1month average): Shows neat DKPs in Arctic.
=======================================================
2) Simon M: 
Fabio is working: Restoring 1^oC: NCEP/NCAR reanal rather than core runoff.
=======================================================
3) ACCESS-OM2 CICE config

Current CICE config in the coupled ocean/sea-ice mode:
4 ice layers plus snow
5 thickness categories
mushy ice TD
classic EVP Dynamics
melt poinds 
JRA55 - do V1.3 1984/85 repeat years forcing, 0.5625^o, 3 hourly
SSS restoring to WOA 2013 V2
1200 PUs for CICE + 4358PUS for MOM + 1 for Matm
--> No polynyas in 1/10^o even though JRA has good katabatics (Adele Land)
     --> look at LHF + net ice production
---
Ice volume in 1/10^o in Arctcic is too high: in access of 30Mio km^3
  Piomass comparison
--- 
Issues: 
* spin up
* compatible physical parameters in Mom & CICE   <-- not working
- TC using mushy ice TD but Bitz & Lipscome is xxx (? cannot read my notes)
- EVP or EAP? --> EVP seems to do ok producing DKPs  --> Use revised EVP.
=======================================================
4) What to use for validation? 
Simon M: Tamura's polynya ice prodution.
=======================================================
5) IcePack has been released (Petra)
Can it be ported to NCI?
=======================================================
6) Timing of COSIMA configs? 
Nick H. is still using/working on MOM5.
Hence MOM6 work is delayed. 
 But Los Alamos are doing MOM (?? which version) with CICE6.
=======================================================
7) Next steps: 
Start MOM6 with SIS2 for ease!!!
 1) Timing of advance and retreat   --> Siobhan OF: That is controlled by JRA forcing.
 2) Ice motion?  AVHRR data to compare  <-- Follow this one up.
=======================================================
8) Various:
* John Spence: 
Compare namelists with those used by others.
PH: Check with what the Arctic high-res folks use. 

* SOF: European updata: LIM2/3, Gelato (CERFACS and UKMet).
  --> Access: Where to couple ?  <-- new UKMet model: 2019 to do runs.
                                     --> C-grid (NEMO?)
=======================================================
=======================================================
Next meeting: 
At COSIMA workshop, Canberra, 08/09 May 2018
=======================================================

ACCESS-OM2 Update

Over the last few months, COSIMA folks have been working hard on releasing our ACCESS-OM2 suite of models. The current status is that we have now completed a 500 year spinup for 3 different cases using the JRA55-do (Tsujino et al., 2017, personal communication) forcing dataset. Some preliminary results can be seen in the figures below. We are also spinning up a CORE-NYF comparison case. For a more complete analysis have a look in the COSIMA Cookbook.

Plans in the coming weeks are to finalise spinups of our 0.25° case (ACCESS-OM2-025), and to begin running our flagship 0.1° simulation, ACCESS-OM2-01.

 

Technical Working Group Meeting, November 2017

Minutes

Date: 14th November 2017
Attendees:

  • Aidan Heerdegen (Chair), Andrew Kiss (ARCCSS ANU)
  • Fanghua Wu (National Climate Center, China Meteorological Administration, Visitor ANU)
  • James Munroe (Memorial University of Newfoundland, Visitor ANU)
  • Nicholas Hannah, Anthony (Double Precision)
  • Russ Fiedler and Matt Chamberlain (CSIRO Hobart)

COSIMA Models

  • Discussion around publicising 1/10th model spin up, in case interested parties would like diagnostics saved.
  • Bluelink are interested in full JRA55-do IAF style spin up, and would want 15-20 years of daily full 3D U,V,T,S and eta fields from that. What is required to construct ensembles/climatologies.
  • Nic looking into ACCESS-OM2-01 performance issues. Lots of time in ice coupling field halo updates. In serial so holding up ocean when it does this. Definite target for optimisation. Should use OASIS to fill the halos when it does the coupling step? Russ disagrees. OASIS shouldn’t know anything about what goes on in models. Gridding using block trains, a 1:1 mapping between grids. If you do this have a 1:many mapping. No longer have identical grids when put in halo information, might break optimisations. When Russ looked at 1/4 deg, hold up was due to synchro just before that. Not sure about 1/10th. Want a barrier just before calling clock before halo update. See if synchronisation issue, or actual time take with halo distribution. 5 halo distributions being done. Heaps more done in CICE itself. Nic: land imbalance between ice processors? Russ: yes my hunch. Load imbalances change a lot with resolution and processor layout. Nic: a problem doing halo updates without considering where field is used. Russ: agree. Velocities need updating, not sure about tracers.
  • Fanghua has been running the new tenth bathymetry with the MOM-SIS-KDS75 config. With JRA55 RYF forcing time step now 450s (from 150s initially). Runoff data now a problem with very low salinities in the arctic at about 7m depth, even with 150s timestep. Created new runoff data, spread more into the ocean but still have issues. Russ saw very high salinities in the Arctic (Laptev Sea). Might be brine rejection from forming sea ice from ice free start. Suggests decreasing salinity restoring timescale from current 60 days to 10 days or even 1 day, to get the model over the initialisation. Andrew suggested issue could be resolved with initial sea ice climatology. There were issues with these files and not been used for a long time. Recent poster to mom users google group has identified some of the problems.
  • Nic’s online runoff redistribution may help, as it is possible to specify maximum runoff per cell, which can help in these areas with very large runoff. Would require ACCESS-OM2-01 config.
  • Nic currently working on getting ACCESS-OM2-01 working with Russ’ new bathymetry. Had a couple of attempts. Getting close, various technical glitches with masks and so on.
  • Andy Hogg has MATM issue when running ACCESS-OM2-1deg for more than 4 years at a time. There is an error on netCDF open call, which comes from HDF layer. Nic ran valgrind, found a bunch of errors, and so recommends everyone update their MATM, but this did not fix the 5 year issue. Determined this was not a memory errors, but an HDF library error. Russ suggested using some HDF library calls to try and determine why the crash occurred. Also try different versions of the netcdf library.
  • Nic suggested we could change MATM to make few file open calls. Aidan has a new payu feature that allows multiple runs per PBS submission, so decided not a priority as MATM needs complete rewrite.
  • Regridding. Nic: need to choose which interpolation schemes to use for which fields. 2nd order cons for everything? Russ: Velocity should not be conservative. Momentum is not conserved. Patch for velocities, T and S. Will give smooth flux fields. Nic: 2nd order cons will be very smooth. Russ: do whatever is cheaper for T, S. U,V should be as smooth as possible. Patch should be 1st order cons, possible 1nd order.  AK: 1st order cons is piecewise constant (bad for wind stress curl). 2nd order is piecewise linear? So similar to bilinear. Need to go to patch for smoother. Russ: tried 2nd order cons, see problems at corners, nodes and edges with wind stress curl. Coarse to fine get artefacts. Patch should work. AK: half of the fields are fluxes. Those should be conservative (2nd order ideally). The remaining are not fluxes, don’t see strong argument for conservative. Is there an issue with different interpolation schemes from different fields? Will bulk formula at fine scale be an issue? Russ: will get jumps in some of the calculated fields. Quantities like T, S should be done with patch, end up with smooth fluxes. AK: Surface stress bulk formula, does it take atmosphere stability into account? Any drag coefficient? Russ: it does. Looks at a profile, figure out a profile. AK: Use SST and 10m T to determine stability? MC: Yes. Say warm atmosphere sitting over cold surface, that’s stable so air would slide over. Daytime, warm surface, near neutral stability so not so sensitive. Possible for temperature and humidity to have small effect on drag coefficient. AK: If we use different interpolation method for 10m winds/T, will it cause issues? Russ: Small jump in sensible heat maybe? Just go with patch or bilinear for all scalar quantities. Velocity go for patch. How will it take into account rotation in tripolar? Presume it is handled  well? AK: only an issue with velocity. Checked with current forcing fields and was ok. Will check new fields the same way.
  • AK: Final decision:
    • patch (the smoothest available) for u_10, v_10
    • 2nd order conservative for fluxes (rain, rdls, rsds, runoff_all and snow)
    • patch or bilinear for non-flux scalars (q_10, slp, t_10) suggest trying patch and only using bilinear if performance with patch is bad
  • Nic: what does MOM-SIS do? Aidan: Thought Steve said bicubic, used to use bilinear but wasn’t smooth enough. Smoother the better.
  • AK: Should WOA salinity restoring fields be smooth in the same way? Nic: What do we currently do? Nic: bilinear? Aidan did it. Russ: not a big issue if salinity restoring not too strong.

Task follow ups

  • Should be using GFDL FMS code directly. Would work better to collaborate with GFDL. Use same code, submit bug reports easily.
  • Once we have FMS as submodule, use all pre/post processing code from GFDL. Make MOM5 leaner, easier to keep updated. Russ: what is the latest FMS version? Aidan: don’t know, and it is hard to tell. Russ: noticed there are new features, like new diagnostic output options, e.g. RMS on the fly, statistics. So things like diag_manager has been updated. Could be some other powerful tools.
  • Aidan: Currently huge step to upgrade. Small step, but could be really good. Not sure how Marshall did it, but not simple.
  • Nic has updated the access-om2 repo structure. Every single test case/experiment is in it’s own repo. Makes it easier for users to grab config without worrying about other configs and source code. OceansAus now has more experiment repos. Aidan: Andy has an issue with git clashes with multiple runs in a single repo. This will fix this.
  • Blog posts?

Actions

New:

  • Will have a December meeting. Tue 12th.
  • Determine if COSIMA intend to do IAF JRA55 spinup of tenth model (Aidan)
  • Send link to spinup diagnostics spreadsheet to Russ (Andrew Kiss)
  • Nic add MPI barrier before ice halo updates timer to check if slow timing issues are just ice load imbalances that appear as longer times due to synchronisation.
  • Test Andy’s 5 year config with different netcdf library versions to check MATM error is not a just a library issue (Aidan)
  • Check current sea surface salinity restoring smoothing (Aidan)

Existing:

  • Russ to add all his ocean bathymetry code to OceansAus repo.
  • Nic to help Peter get his MOM repo up to date with MOM5 master branch, and then merge changes
  • Look into OpenDAP/THREDDS for use with MOM on raijin (Aidan, Nic, Marshall)
  • Nic to present MATM code re-write proposal to TWG for feedback before sign-off. Will then be presented to Andy Hogg for approval.
  • Nic create a discussion document (on COSIMA?) to document current approaches and strategies for future
  • Move FMS to submodule of MOM5 github repo (Marshall). Liase with Nic on implementation?
  • Work up test cases to cover the nudging code (Justin, Mirko) and supply them to Nic.
  • Add new test cases to Jenkins test suite (Nic).
  • Start a new google doc about coupler issues and MATM (Marshall)
  • Ask Dale Roberts about effects of OpenMP for Roger (Marshall)
  • Make a proper plan for model release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model (Marshall and TWG)
  • Blog post around issues with high core count jobs and mxm mtl (Nic)
  • Create document outlining options for configuration sharing (?)

COSIMA Cookbook

One of the goals of the COSIMA community is to create a common framework for evaluating and diagnosing model output. Our initial attempts to do this are encapsulated in the COSIMA Cookbook. The Cookbook aims to:

  • Create standard Jupyter Notebooks for diagnosing model simulations, or groups of simulations;
  • Documenting methods to efficiently interrogate large model datasets;
  • Document model sensitivity studies; and
  • Provide a gallery highlighting commonly used diagnostics.

The COSIMA Cookbook is publicly available on the OceansAus Github repository. We are also in the process of building documentation and the COSIMA gallery.

COSIMA 2 Workshop Report, May 2017.

The second meeting of the Consortium for Ocean Sea Ice Modelling in Australia (COSIMA) was held in Sydney on the 25-26 May 2017. There were 36 attendees (2 via video, 26 gave presentations) representing 9 different institutions. A full participant list and links to presentations is included at the end of this report. The workshop goals and agenda is here.

The consortium has 3 years remaining of 4 years funding from an ARC Linkage Project to build a model configuration which underpins a variety of applications. The consortium also aims to provide a lasting platform that engages and benefits all ocean-sea ice scientists and technicians in Australia. This report begins with an assessment of milestones achieved since the May 2016 meeting, followed by a list of the 2017 workshop highlights and discussion/action items to be undertaken in 2017/2018.

Assessment of  COSIMA Community Action Items from 2016 Workshop

  1. Need to create a website to outline COSIMA activities and developments. [We are in the process of acquiring the domain name cosima.org.au, and ANU will fund hosting and a web development team to put together a skeleton site.]
    • DONE (cosima.org.au).
  2. We will formalise a code of ethics for COSIMA users to abide by, based on the DRAKKAR agreement. [Spence]
    • DONE (http://cosima.org.au/index.php/about-2/ethics/) 
  3. We aim to register members on the website, and create a mailing list.
  4. We will need to formalise the use of technologies to share code configurations, analysis tools and data. It may be possible to have a data project code on NCI to help with this.
    • Substantial Progress: i) hh5 storage for COSIMA model data at nci; ii) cosima cook book github repository in use for data processing scripts; iii) git hub model code repositories in use for some configurations.
  5. We will release flagship configurations that are broadly supported by the community. The goal is to make the naming conventions consistent with the ACCESS community where possible, and to overlap with ACCESS developments where possible.  
    • Substantial Progress. New flagship configurations are under active developement: e.g. JRA-55 forcing of MOM-SIS and MOM-CICE configurations at multiple resolutions, including enhanced vertical resolution. Information regarding flagship simulations will be posted on website this quarter [Spence]. 
  6. COSIMA will hold an annual meeting in the last week of May, for two days. Venue will rotate around the partner institutions. The focus of the meeting will be on science applications of ocean models, but will also include a technical component.
    • DONE. Next meeting in first week of June, 2018, Canberra.
  7. We will aim to have more regular communications, including newsletters and video meetings.
    • Substantial Progress. Open COSIMA video conference meetings were held monthly. Linkage project CI meetings occurred quarterly. Regular Technical Working Group meeting were held. Meeting notes are often posted at cosima.org.au.
  8. We will establish working groups within the community, along with a working group chair. Proposed groups include
    • Sea Ice Modelling [Heil]
      • Limited activity from this group in 2016
    • Technical [Ward]
      • Best communicating group that holds regular meetings, posts notes, has active community and achieves results.
    • Linkage Project [Hogg]
      • Quarterly meetings are held to gage model development progress and funding usage.
  9. The major gap in the community was identified to be sea ice modelling and forecasting. We will all look for opportunities to attract visitors and expertise in this space.
    • Needs more work. 

Assessment of  ARC Linkage Project Action Items from 2016 Workshop

  1. There is a strong need to consider vertical resolution in future high resolution model developments (Stewart).
    • DONE. http://www.sciencedirect.com/science/article/pii/S1463500317300434
  2. Our MOM-CICE implementations using OASIS3-MCT coupling, but it is not clear that this solution will scale to 0.1° and beyond (Hannah)
    • Substantial Progress. Parallel OASIS-CICE now runs as fast as serial FMS-SIS with MOM5. Uncertainties remain regarding MOM6 coupling.
  3. Forced ocean-sea ice models should be transferring to the JRA-55 forcing set when possible (Marsland)
    • DONE. JRA-55 forced configurations are underway in multiple groups. Fabio Dias is running coarse resolution MOM5-OASIS-CICE simulations. The CoE Oceans Group is running MOM5-FMS-SIS and MOM5-OAIS-CICE simulations at multiple resolutions. A relative year forcing data set has been developed from JRA-55 (contact Kyle Stewart)
  4. There was general agreement that the Linkage Project should fund both the development and evaluation of new model configurations. This point implies that we should equally fund the technical and postdoctoral position, despite the partial funding of the program.
    • DONE. Hannah continues to excel at developing the model Linkage project model configurations. A scientist has been hired (starting Aug., 2017) to help develop and evaluate the model simulations.  
  5. In the first year we will look to upgrade current MOM5 implementations, focussing on the vertical grid and the incorporation of CICE.
    • Substantial Progress. A 1/10 degree 75 level ACCESS-OM2 configuration consisting of MOM5 and CICE5 is being developed. Several scalability and performance problems have been overcome and we are now completing a spin-up run. The work has also resulted in improvements to the lower resolution 1 and 1/4 degree configurations and ACCESS coupled models.
  6. In subsequent years we will look to adopt a MOM6 configuration.
    • Progress. Testing with MOM6 adaptive vertical coordinates is underway (Angus Gibson). Given that SIS2 is adopting much of the CICE5 code, uncertainties remain regarding the sea ice and coupling model to be used with MOM6.
  7. As configurations develop and have been properly evaluated, they will be distributed to the community.
    • Substantial Progress. The distribution platform is under development with substantial progress on the back-end. A COSIMA data repository on NCI exists, and currently houses several flagship simulations. A repository for python analysis code that accesses the data repository is in use (https://github.com/OceansAus/cosima-cookbook). Substantial progress has been made in adopting effective big data analysis techniques (James Munroe).

2017 Workshop_Highlights

  • Substantial progress in the development and analysis of coupled WOMBAT biogeochemistry simulations.
  • Exciting opportunities within the newly funded CSIRO Decadal Prediction team.
  • Our technical teams continue to excel at developing and optimising model configurations with effective collaborations. Hannah has made excellent progress in MOM-OASIS-CICE coupling. Ward provides a key conduit to understanding NCI optimisations. Heerdegen is supporting a multitude of operations. Fiedler continues to develop the new bathymetry.
  • The enhanced capabilities of CICE over SIS remain. CICE6 is being developed under a community framework. SIS2 has adopted some of the CICE code.
  • ACCESS is working towards the CMIP6 deadlines.
  • The blue link forecast models are now perceived to be limited more by the ocean model configuration than the data assimilation.
  • New model configurations (MOM5-OASIS-CICE, MOM5-FMS-SIS) using enhanced vertical resolution, JRA forcing (including an RYF forcing), and updated WOA initial conditions are underway.

COSIMA Community Discussion/Action Items from 2017 Workshop

  1. Develop website into more of a community portal to flagship simulations. Provide model configuration and analysis with low barrier to entry for users. (Spence)
  2. Develop the COSIMA Cookbook portal  with a gallery of options (Munroe). Improve the community usage and contribution to the cookbook. Provide a visual gallery of code examples (ALL).
  3. Provide online chat capability for COSIMA model configurations. Look into using Slack teams. (Spence)
  4. Provide list of ongoing experiments, links to data available and diagnostics wanted, contacts for expts. (Spence)
  5. Consider separating models configs into Tier 1 (control flagships), 2 (perturbations from control), 3 (parameter sensitivities) to aid communication. (Spence)
  6. Provide discussion platform for new experiments. Wider discussion of methodologies and diagnostics.
  7. Provide platform for efficient explorations of parameter spaces (sea ice, neutral physics, bgc params, SSS restoring). How to share configurations? Tracking configs with git needs to be embraced by all! (Aidan, Hogg)
  8. Explore CICE efficiency, coupling a wave model to CICE, CICE6-MOM6 capabilities. Be aware of similar progress being made at NCAR. (Heil).
  9. Consider hiring admin support for the COSIMA website (England).

ARC Linkage Project Discussion/Action Items from 2017 Workshop

  1. A new 1/10 degree model bathymetry data set continues to be underdevelopment with substantial progress towards a final product expected with a few months (Russ Fiedler)
  2. Hannah continues to develop the MOM5-OASIS-CICE5 configurations. Expects to provide spinups at 3 model resolutions to the communitu (1, 1/4, 1/10). Discussion regarding the CICE settings (Heil), vertical resolution (Stewart), JRA forcing (Stewart) are needed. Does the model currently have a closed heat budget (Dias)?
  3. Excited to welcome Andrew Kiss as the scientist funded from this ARC Linkage grant. Hoping for a model evaluation paper before the next meeting.

 

Attendees

(Where available, talk files are linked from the presenter’s name.)

Andy Hogg (ANU)
Gary Brassington (BoM)
Nic Hannah (Breakaway Labs)
Dave Bi (CSIRO)
Matthew England (UNSW)
Adele Morrison (ANU)
Ryan Holmes (UNSW)
Jan Zika (UNSW)
Angus Gibson (ANU)
Andy Kiss (ANU)
Veronique Lago (UNSW)
Laurie Meviel(UNSW)
Kaitlin Alexander (UNSW)
David Webb (UNSW)
Peter Dobrotoff (CSIRO)
Leela Frankcombe (UNSW)
James Munroe (Memorial, Canada)
Joan Llort (Utas)
Arnold Sullivan (CSIRO)
Xuebin Zhang (CSIRO)
Terry Okane (CSIRO, Remote)
Peter Oke (CSIRO)
Paul Spence (UNSW)
Richard Matear (CSIRO)
Clothilde Langlais (CSIRO)
Stephen Griffies (GFDL, Remote)
Russel Fiedler (CSIRO)
Marshall Ward (NCI)
Matthew Chamberlain (CSIRO)
Fabio Dias (UTas)
Aidan Heerdegen (ANU)
Petra Heil (AAD & ACE CRC)
Siobhan O’Farrell (Remote, CSIRO)
Xiaobing Zhou (BoM)

Technical Working Group Meeting, May 2017

Minutes

Date: 16th May 2017
Attendees:

  • Marshall Ward (NCI, Chair)
  • Aidan Heerdegen and Andy Hogg (ARCCSS ANU)
  • Scott Wales (ARCCSS Melbourne Uni)
  • Nicholas Hannah (ARCCSS/Double Precision)
  • Russ Fiedler and Matt Chamberlain (CSIRO Hobart)
  • Justin Freeman (BoM)
  • Peter Dobrohotoff and Roger Bodman (CSIRO Aspendale)

Updates

  • Roger is having issues with N95 atmosphere. Marshall says it doesn’t scale past 256 cores. Roger would like to get RTM profiling working. Martin and Peter have got multiple threads working in AMIP. Has spoken to Scott Wales. Something odd happens in decomposition. Marshall will ask Dale Roberts about effects of OpenMP. Has chemistry has been enabled for OpenMP?

Liaison with COSIMA Management

  • Andy is here to get some feedback about our activities. Andy thinks TWG is doing a good job of communicating, a goal is to get more communication amongst COSIMA in general. Management team meets quarterly. No science talk or detail. Have to be better at merging/converging disparate code, TWG is crucial for this.
  • Andy wants a better framework for analysing and post-processing runs, and access others outputs. And we are currently doing some work in this space. The effort it is scattered, no one lead. Nic has done some work in past. Now using ipython notebooks to share analysis. James Munroe working on dashboards. Justin asked if COSIMA will deliver this? Andy: not explicitly funded but will benefit uptake. Need better ways to serve data.
  • Andy would like more engagement around COSIMA website. Blog our progress? Marshall: do you have sample topics? Is it legitimate to put updates on issues with library versions say? Andy: borderline, maybe just TWG for that example. Currently reply on members of TWG to propagate information back to users. Maybe more relevant would be update on scaling of code for example. Don’t want to limit people now. Encourage as much as possible and filter if required. Peter agreed: get useful information up there. Haven’t had anything over the last year. Suggestion to make minutes more report like? Nic thought blog posts are a nice idea, but need deeper insight to be useful and interesting.
  • Andy wants science side to publish results of runs, and point to data.
  • Andy also keen for COSIMA to have information about model versions. ACCESS doesn’t have a way of releasing versions and hosting code. ACCESS is somewhat hobbled with partner disputes. Would like ACCESS-OM releases on COSIMA. Marshall pointed out the TWG was set up to address this. Models were not ready at that stage. Marshall suggested we make a proper plan for a release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model.
  • What are the expectations of TWG for the COSIMA Meeting? Andy: some already giving talks. Interact and discuss with others. Get to know each other and ambitions and look for synergies.
  • Andy will contact Paul to make sure TWG will have a slot to fill in others on progress.
  • Andy would like an email list for COSIMA announcements.

COSIMA Models

  • Nic: Was in Canberra week ago. Had tenth timestepping on <3000 cores. More than 3000 didn’t initialise. MCT couldn’t set up routing tables with more than 3000 cores. Would just hang. After discussions figured out some MPI switches and flags to get it working: mxm mtl makes it work better. Justin suggested this would make a good blog post. Marshall found MOM6 was failing at 3000 cores too. Went away with mxm mtl.
  • Andy: MOM-SIS tenth was also failing. About 30% fails. Russ has had similar. Nic now running on 6K cores in ocean.
  • Nic had discussions in Canberra around CICE halo updates. Made 12+ changes to CICE and MATM code. Made big improvements. For all three model resolutions (1,0.25,0.1) overhead of coupling is 1-2% compared to MOM solo. That is a tiny serial bit of interpolating forcing fields on to ocean. Like 20s/month for quarter degree.
  • Nic: quarter degree 1800s time step should be less than 75min/year. Andy: UNREAL!. 1 deg is also running super fast. 50 years/day. Did a new compile on MCT library to squeeze as much performance as possible.
  • Now have 3 new configs. 1 deg and 0.25 deg could be used. Focussing on tenth at the moment.
  • Andy: 70min MOM-SIS-025  + CORE, JRA55 adds 30%.
  • In old config, all models block waiting for MATM to read files. Now MATM has sent everything. Reduces difference between CORE and JRA55. Nic has not done longer runs as yet. No longer buffering multiple years of MATM output.
  • Agreed Andy should get these configs and do some longer runs.
  • Andy talking to NCAR about JRA55 forcing. CORE used NYF. JRA doesn’t have that. Others have used a single year. Our strategy is May-May forcing with a shock at the end of May. Candidate years are 84/85, 91/92, 03/04. Want to test this at 0.25 deg and 1deg. Should adopt just what Nic has done.
  • These are MOM-SIS as Andy wants a baseline. Doing MOM-SIS from CORE with WOA13. Repeat with JRA55 RYFs. Want to compare to ACCESS-OM config of Nic.
  • Nic and Andy to talk offline and try out a repeat year.
  • Although it is fast, the tenth is inefficient, as there is no ocean masking currently. This is the next priority. Probably beyond Nic’s current contract.
  • Nic can I use unmasked restarts? Russ: yes. Just need to combine them.

COSIMA Workshop

  •  Marshall: can we agree to transfer to CM2? To get on common version of CICE.
  • Andy: will the code we release in OM2 be different in ocean and/or ice? Can we manage it in one codebase? Marshall: should be possible. Set aside time to discuss this at meeting.
  • Discuss moving to common CICE repo for all.
  • Marshall: Justin need some info from us on OM config? Nic and Justin will liase.
  • Justin won’t be at the COSIMA Meeting, does he want us to cover anything? Justin: staying up to date with what we’re doing, will be engaging much more in future.

Updates on previous actions

  • Nic: has updated OceansAus repo to Peter’s CICE. Can Peter look at the code and check it.
  • Marshall: Justin need some info from us on OM config? Nic and Justin will liase.
  • Russ has been doing a lot of clicking for bathymetry. Aus and PNG done. Need help.

Actions

New:

  • Ask Dale Roberts about effects of OpenMP for Roger (Marshall)
  • Make a proper plan for model release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model (Marshall and TWG)
  • Contact Paul Spence about TWG speaking slot at meeting (Andy)
  • Prepare slides for TWG presentation at COSIMA meeting, and present (Aidan and Marshall)
  • Email list for COSIMA announcements (Aidan)
  • Blog post around issues with high core count jobs and mxm mtl (Nic)
  • Do longer runs with Nic’s 1 deg and 0.25 deg ACCESS-OM2-JRA55 configs (Andy and Aidan)
  • Try repeat year forcing with Nic’s configurations (Nic and Andy)

Existing:

  • Nic to present MATM code re-write proposal to TWG for feedback before sign-off. Will then be presented to Andy Hogg for approval.
  • Nic create a discussion document (on COSIMA?) to document current approaches and strategies for future
  • Move FMS to submodule of MOM5 github repo (Marshall). Liase with Nic on implementation?
  • Test Nic’s access-om model config on OceansAus (All)
  • Work up test cases to cover the nudging code (Justin, Mirko) and supply them to Nic.
  • Add new test cases to Jenkins test suite (Nic).
  • Start a new google doc about coupler issues and MATM (Marshall)

COSIMA Linkage Project

The COSIMA consortium is partly supported by an ARC Linkage Project. This project is now active, following sign-off from all partners in December 2016. The management committee for the project formally met for the first time on Friday 9th December. Following are abbreviated minutes from the meeting which outline our plans for 2017:

  1. To advertise for a Postdoctoral/Research Fellow to coordinate COSIMA projects.
  2. To enhance the vertical resolution of existing MOM5-SIS 0.1° model, with no change in horizontal resolution;
  3. To enhance model bathymetry in coastal regions to take advantage of reduced minimum depth stemming from vertical resolution changes;
  4. To couple MOM5 with CICE at 0.1° using OASIS3-MCT (we have a contract with Nic Hannah from Double Precision to perform this work);
  5. To evaluate, refine, optimise and document the new model configuration.

Furthermore, we have confirmed that the 2017 COSIMA meeting will be in Sydney on May 25-26.

Technical Working Group Meeting, October 2016

Minutes

Date: 11th October 2016
Attendees:

  • Marshall Ward (NCI, Chair)
  • Aidan Heerdegen (ARCCSS)
  • Nick Hannah (ARCCSS/Breakaway Labs)
  • Justin Freeman and Mirko Velic (BoM)
  • Russ Fiedler and Matt Chamberlain (CSIRO Hobart)
  • Peter Dobrohotoff (CSIRO Aspendale)

Code submissions

  • Mirko submitted major refactoring update to the nudging code for MOM. Three different options depending on namelist. One just sponge, one does nudging, another does adaptive nudging. Added instantaneous update from datafile. Wanted to reproduce the MOM4 behaviour. Tested, and now works. Was broken previously.
  • Can merge, but we need some testing on other coverage. Currently have a dozen test cases. Not sure any touch this, but will run them anyway. Justin suggests they provide a test case which covers some of these sections.
  • Nic asked that if possible functional and formatting changes be separate commits, as it makes approving pull requests much easier
  • Maybe not merge yet, but get testing working to cover this. Justin will look at adapting an existing MOM test case for this purpose.

Exchange grids and smoothing

  • Justin was talking to Paul Sandery about exchange grids. An issue with tiling as a result of remapping. Was asking about how Russ implemented smoothing.
  • If you took interim fields end up with horrible pattern with convergence of winds with 1st or 2nd order remapping due to discontinuity. Russ wrote some code that does 2D smoothing within the surface boundary condition. Bypasses the exchange grid and used the flux exchange to native grids options(?). GFDL apply an interpolation when they read in via data override. So can use the data override to interpolate to the finer grid and can control this.
  • This is only a problem with conservative remapping with exchange grids.
  • Nic didn’t think this was a problem with standard MOM-SIS runs, but Russ said it should still be visible in the fluxes with coarse (1deg) forcing fields.
  • With ACCESS high res ocean the fields and fluxes are extremely blocky, so Nic smooths on the ice grid, before it comes into the ocean grid, on a tile by tile basis.
  • If you want local conservation, cannot get around this. In ACCESS can use linear interpolation and then post-process to get global conservation. Doesn’t work with local conservation.
  • Marshall suggested we have some test cases that don’t run the model but test coupling and fields
  • These effects most often seen when there is a big difference in resolution between model fields and input fields. Look at wind stress fields. Maybe some of the barotropic fields, height and definitely convergence in barotropic restart file.
  • Paul’s runs do not use conservative remapping. Don’t see the horrible features with some of the other schemes.
  • Nic: do we need a central document discussing this?

OceanMaps 4

  • Justin is trying to prototype OceanMaps 4. Picking up on Paul Sandery’s work. He has been using MOM5-SIS and using bulk fluxes to link the models. Would like to standardise, or make these things available. Not sure how it connects to linkage project.
  • Nic felt it was good to know what Paul does. So far no code changes?.

FMS

  • Aidan got a query from Dave Hutchinson, asking if latest version of FMS was included in the code on MOM5 repo. Marshall has updated FMS in the master branch to Ulm, but not to Verona, the latest version.
  • Move FMS to a submodule of MOM5 rather than manually included inline
  • Goal is for Rui Yang (NCI) to work on parallel netCDF in MOM5

Model release naming and definition

  • Still an issue
  • Nic has put an access-om model on OceansAus. Has version controlled input files and code. Can be downloaded, compiled and run.

CICE

  • ACCESS-OM models are using CICE4.?, but Peter is using 5.1.
  • There are many bug fixes and performance improvements in the version of CICE Nic has been working on that would be beneficial to Peter.
  • Peter is working on a refactoring of CICE5.1
  • We should align our work to the same version of CICE.
  • First step is for Peter’s version of CICE5.1 to be hosted on OceansAus and development work to be based from that so we can work together. Some discussion about the best way to do this.

Actions

  • Just and Mirko work up test cases to cover the nudging code and give them to Nic.
  • Nic to add new test cases to Jenkins test suite.
  • Aidan to add mom-ocean.org and mom-ocean.org.au to uptime monitoring service (Uptime Robot).
  • Add Peter’s CICE5.1 config to OceansAus github repo
  • Nic create a discussion document (on COSIMA?) to document current approaches and strategies for future
  • Marshall to move FMS to submodule of MOM5 github repo. Liase with Nic on implementation?
  • Others test Nic’s access-om model config on OceansAus