ACCESS-OM2 Update

Over the last few months, COSIMA folks have been working hard on releasing our ACCESS-OM2 suite of models. The current status is that we have now completed a 500 year spinup for 3 different cases using the JRA55-do (Tsujino et al., 2017, personal communication) forcing dataset. Some preliminary results can be seen in the figures below. We are also spinning up a CORE-NYF comparison case. For a more complete analysis have a look in the COSIMA Cookbook.

Plans in the coming weeks are to finalise spinups of our 0.25° case (ACCESS-OM2-025), and to begin running our flagship 0.1° simulation, ACCESS-OM2-01.

 

Technical Working Group Meeting, November 2017

Minutes

Date: 14th November 2017
Attendees:

  • Aidan Heerdegen (Chair), Andrew Kiss (ARCCSS ANU)
  • Fanghua Wu (National Climate Center, China Meteorological Administration, Visitor ANU)
  • James Munroe (Memorial University of Newfoundland, Visitor ANU)
  • Nicholas Hannah, Anthony (Double Precision)
  • Russ Fiedler and Matt Chamberlain (CSIRO Hobart)

COSIMA Models

  • Discussion around publicising 1/10th model spin up, in case interested parties would like diagnostics saved.
  • Bluelink are interested in full JRA55-do IAF style spin up, and would want 15-20 years of daily full 3D U,V,T,S and eta fields from that. What is required to construct ensembles/climatologies.
  • Nic looking into ACCESS-OM2-01 performance issues. Lots of time in ice coupling field halo updates. In serial so holding up ocean when it does this. Definite target for optimisation. Should use OASIS to fill the halos when it does the coupling step? Russ disagrees. OASIS shouldn’t know anything about what goes on in models. Gridding using block trains, a 1:1 mapping between grids. If you do this have a 1:many mapping. No longer have identical grids when put in halo information, might break optimisations. When Russ looked at 1/4 deg, hold up was due to synchro just before that. Not sure about 1/10th. Want a barrier just before calling clock before halo update. See if synchronisation issue, or actual time take with halo distribution. 5 halo distributions being done. Heaps more done in CICE itself. Nic: land imbalance between ice processors? Russ: yes my hunch. Load imbalances change a lot with resolution and processor layout. Nic: a problem doing halo updates without considering where field is used. Russ: agree. Velocities need updating, not sure about tracers.
  • Fanghua has been running the new tenth bathymetry with the MOM-SIS-KDS75 config. With JRA55 RYF forcing time step now 450s (from 150s initially). Runoff data now a problem with very low salinities in the arctic at about 7m depth, even with 150s timestep. Created new runoff data, spread more into the ocean but still have issues. Russ saw very high salinities in the Arctic (Laptev Sea). Might be brine rejection from forming sea ice from ice free start. Suggests decreasing salinity restoring timescale from current 60 days to 10 days or even 1 day, to get the model over the initialisation. Andrew suggested issue could be resolved with initial sea ice climatology. There were issues with these files and not been used for a long time. Recent poster to mom users google group has identified some of the problems.
  • Nic’s online runoff redistribution may help, as it is possible to specify maximum runoff per cell, which can help in these areas with very large runoff. Would require ACCESS-OM2-01 config.
  • Nic currently working on getting ACCESS-OM2-01 working with Russ’ new bathymetry. Had a couple of attempts. Getting close, various technical glitches with masks and so on.
  • Andy Hogg has MATM issue when running ACCESS-OM2-1deg for more than 4 years at a time. There is an error on netCDF open call, which comes from HDF layer. Nic ran valgrind, found a bunch of errors, and so recommends everyone update their MATM, but this did not fix the 5 year issue. Determined this was not a memory errors, but an HDF library error. Russ suggested using some HDF library calls to try and determine why the crash occurred. Also try different versions of the netcdf library.
  • Nic suggested we could change MATM to make few file open calls. Aidan has a new payu feature that allows multiple runs per PBS submission, so decided not a priority as MATM needs complete rewrite.
  • Regridding. Nic: need to choose which interpolation schemes to use for which fields. 2nd order cons for everything? Russ: Velocity should not be conservative. Momentum is not conserved. Patch for velocities, T and S. Will give smooth flux fields. Nic: 2nd order cons will be very smooth. Russ: do whatever is cheaper for T, S. U,V should be as smooth as possible. Patch should be 1st order cons, possible 1nd order.  AK: 1st order cons is piecewise constant (bad for wind stress curl). 2nd order is piecewise linear? So similar to bilinear. Need to go to patch for smoother. Russ: tried 2nd order cons, see problems at corners, nodes and edges with wind stress curl. Coarse to fine get artefacts. Patch should work. AK: half of the fields are fluxes. Those should be conservative (2nd order ideally). The remaining are not fluxes, don’t see strong argument for conservative. Is there an issue with different interpolation schemes from different fields? Will bulk formula at fine scale be an issue? Russ: will get jumps in some of the calculated fields. Quantities like T, S should be done with patch, end up with smooth fluxes. AK: Surface stress bulk formula, does it take atmosphere stability into account? Any drag coefficient? Russ: it does. Looks at a profile, figure out a profile. AK: Use SST and 10m T to determine stability? MC: Yes. Say warm atmosphere sitting over cold surface, that’s stable so air would slide over. Daytime, warm surface, near neutral stability so not so sensitive. Possible for temperature and humidity to have small effect on drag coefficient. AK: If we use different interpolation method for 10m winds/T, will it cause issues? Russ: Small jump in sensible heat maybe? Just go with patch or bilinear for all scalar quantities. Velocity go for patch. How will it take into account rotation in tripolar? Presume it is handled  well? AK: only an issue with velocity. Checked with current forcing fields and was ok. Will check new fields the same way.
  • AK: Final decision:
    • patch (the smoothest available) for u_10, v_10
    • 2nd order conservative for fluxes (rain, rdls, rsds, runoff_all and snow)
    • patch or bilinear for non-flux scalars (q_10, slp, t_10) suggest trying patch and only using bilinear if performance with patch is bad
  • Nic: what does MOM-SIS do? Aidan: Thought Steve said bicubic, used to use bilinear but wasn’t smooth enough. Smoother the better.
  • AK: Should WOA salinity restoring fields be smooth in the same way? Nic: What do we currently do? Nic: bilinear? Aidan did it. Russ: not a big issue if salinity restoring not too strong.

Task follow ups

  • Should be using GFDL FMS code directly. Would work better to collaborate with GFDL. Use same code, submit bug reports easily.
  • Once we have FMS as submodule, use all pre/post processing code from GFDL. Make MOM5 leaner, easier to keep updated. Russ: what is the latest FMS version? Aidan: don’t know, and it is hard to tell. Russ: noticed there are new features, like new diagnostic output options, e.g. RMS on the fly, statistics. So things like diag_manager has been updated. Could be some other powerful tools.
  • Aidan: Currently huge step to upgrade. Small step, but could be really good. Not sure how Marshall did it, but not simple.
  • Nic has updated the access-om2 repo structure. Every single test case/experiment is in it’s own repo. Makes it easier for users to grab config without worrying about other configs and source code. OceansAus now has more experiment repos. Aidan: Andy has an issue with git clashes with multiple runs in a single repo. This will fix this.
  • Blog posts?

Actions

New:

  • Will have a December meeting. Tue 12th.
  • Determine if COSIMA intend to do IAF JRA55 spinup of tenth model (Aidan)
  • Send link to spinup diagnostics spreadsheet to Russ (Andrew Kiss)
  • Nic add MPI barrier before ice halo updates timer to check if slow timing issues are just ice load imbalances that appear as longer times due to synchronisation.
  • Test Andy’s 5 year config with different netcdf library versions to check MATM error is not a just a library issue (Aidan)
  • Check current sea surface salinity restoring smoothing (Aidan)

Existing:

  • Russ to add all his ocean bathymetry code to OceansAus repo.
  • Nic to help Peter get his MOM repo up to date with MOM5 master branch, and then merge changes
  • Look into OpenDAP/THREDDS for use with MOM on raijin (Aidan, Nic, Marshall)
  • Nic to present MATM code re-write proposal to TWG for feedback before sign-off. Will then be presented to Andy Hogg for approval.
  • Nic create a discussion document (on COSIMA?) to document current approaches and strategies for future
  • Move FMS to submodule of MOM5 github repo (Marshall). Liase with Nic on implementation?
  • Work up test cases to cover the nudging code (Justin, Mirko) and supply them to Nic.
  • Add new test cases to Jenkins test suite (Nic).
  • Start a new google doc about coupler issues and MATM (Marshall)
  • Ask Dale Roberts about effects of OpenMP for Roger (Marshall)
  • Make a proper plan for model release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model (Marshall and TWG)
  • Blog post around issues with high core count jobs and mxm mtl (Nic)
  • Create document outlining options for configuration sharing (?)

COSIMA Cookbook

One of the goals of the COSIMA community is to create a common framework for evaluating and diagnosing model output. Our initial attempts to do this are encapsulated in the COSIMA Cookbook. The Cookbook aims to:

  • Create standard Jupyter Notebooks for diagnosing model simulations, or groups of simulations;
  • Documenting methods to efficiently interrogate large model datasets;
  • Document model sensitivity studies; and
  • Provide a gallery highlighting commonly used diagnostics.

The COSIMA Cookbook is publicly available on the OceansAus Github repository. We are also in the process of building documentation and the COSIMA gallery.

COSIMA 2 Workshop Report, May 2017.

The second meeting of the Consortium for Ocean Sea Ice Modelling in Australia (COSIMA) was held in Sydney on the 25-26 May 2017. There were 36 attendees (2 via video, 26 gave presentations) representing 9 different institutions. A full participant list and links to presentations is included at the end of this report. The workshop goals and agenda is here.

The consortium has 3 years remaining of 4 years funding from an ARC Linkage Project to build a model configuration which underpins a variety of applications. The consortium also aims to provide a lasting platform that engages and benefits all ocean-sea ice scientists and technicians in Australia. This report begins with an assessment of milestones achieved since the May 2016 meeting, followed by a list of the 2017 workshop highlights and discussion/action items to be undertaken in 2017/2018.

Assessment of  COSIMA Community Action Items from 2016 Workshop

  1. Need to create a website to outline COSIMA activities and developments. [We are in the process of acquiring the domain name cosima.org.au, and ANU will fund hosting and a web development team to put together a skeleton site.]
    • DONE (cosima.org.au).
  2. We will formalise a code of ethics for COSIMA users to abide by, based on the DRAKKAR agreement. [Spence]
    • DONE (http://cosima.org.au/index.php/about-2/ethics/) 
  3. We aim to register members on the website, and create a mailing list.
  4. We will need to formalise the use of technologies to share code configurations, analysis tools and data. It may be possible to have a data project code on NCI to help with this.
    • Substantial Progress: i) hh5 storage for COSIMA model data at nci; ii) cosima cook book github repository in use for data processing scripts; iii) git hub model code repositories in use for some configurations.
  5. We will release flagship configurations that are broadly supported by the community. The goal is to make the naming conventions consistent with the ACCESS community where possible, and to overlap with ACCESS developments where possible.  
    • Substantial Progress. New flagship configurations are under active developement: e.g. JRA-55 forcing of MOM-SIS and MOM-CICE configurations at multiple resolutions, including enhanced vertical resolution. Information regarding flagship simulations will be posted on website this quarter [Spence]. 
  6. COSIMA will hold an annual meeting in the last week of May, for two days. Venue will rotate around the partner institutions. The focus of the meeting will be on science applications of ocean models, but will also include a technical component.
    • DONE. Next meeting in first week of June, 2018, Canberra.
  7. We will aim to have more regular communications, including newsletters and video meetings.
    • Substantial Progress. Open COSIMA video conference meetings were held monthly. Linkage project CI meetings occurred quarterly. Regular Technical Working Group meeting were held. Meeting notes are often posted at cosima.org.au.
  8. We will establish working groups within the community, along with a working group chair. Proposed groups include
    • Sea Ice Modelling [Heil]
      • Limited activity from this group in 2016
    • Technical [Ward]
      • Best communicating group that holds regular meetings, posts notes, has active community and achieves results.
    • Linkage Project [Hogg]
      • Quarterly meetings are held to gage model development progress and funding usage.
  9. The major gap in the community was identified to be sea ice modelling and forecasting. We will all look for opportunities to attract visitors and expertise in this space.
    • Needs more work. 

Assessment of  ARC Linkage Project Action Items from 2016 Workshop

  1. There is a strong need to consider vertical resolution in future high resolution model developments (Stewart).
    • DONE. http://www.sciencedirect.com/science/article/pii/S1463500317300434
  2. Our MOM-CICE implementations using OASIS3-MCT coupling, but it is not clear that this solution will scale to 0.1° and beyond (Hannah)
    • Substantial Progress. Parallel OASIS-CICE now runs as fast as serial FMS-SIS with MOM5. Uncertainties remain regarding MOM6 coupling.
  3. Forced ocean-sea ice models should be transferring to the JRA-55 forcing set when possible (Marsland)
    • DONE. JRA-55 forced configurations are underway in multiple groups. Fabio Dias is running coarse resolution MOM5-OASIS-CICE simulations. The CoE Oceans Group is running MOM5-FMS-SIS and MOM5-OAIS-CICE simulations at multiple resolutions. A relative year forcing data set has been developed from JRA-55 (contact Kyle Stewart)
  4. There was general agreement that the Linkage Project should fund both the development and evaluation of new model configurations. This point implies that we should equally fund the technical and postdoctoral position, despite the partial funding of the program.
    • DONE. Hannah continues to excel at developing the model Linkage project model configurations. A scientist has been hired (starting Aug., 2017) to help develop and evaluate the model simulations.  
  5. In the first year we will look to upgrade current MOM5 implementations, focussing on the vertical grid and the incorporation of CICE.
    • Substantial Progress. A 1/10 degree 75 level ACCESS-OM2 configuration consisting of MOM5 and CICE5 is being developed. Several scalability and performance problems have been overcome and we are now completing a spin-up run. The work has also resulted in improvements to the lower resolution 1 and 1/4 degree configurations and ACCESS coupled models.
  6. In subsequent years we will look to adopt a MOM6 configuration.
    • Progress. Testing with MOM6 adaptive vertical coordinates is underway (Angus Gibson). Given that SIS2 is adopting much of the CICE5 code, uncertainties remain regarding the sea ice and coupling model to be used with MOM6.
  7. As configurations develop and have been properly evaluated, they will be distributed to the community.
    • Substantial Progress. The distribution platform is under development with substantial progress on the back-end. A COSIMA data repository on NCI exists, and currently houses several flagship simulations. A repository for python analysis code that accesses the data repository is in use (https://github.com/OceansAus/cosima-cookbook). Substantial progress has been made in adopting effective big data analysis techniques (James Munroe).

2017 Workshop_Highlights

  • Substantial progress in the development and analysis of coupled WOMBAT biogeochemistry simulations.
  • Exciting opportunities within the newly funded CSIRO Decadal Prediction team.
  • Our technical teams continue to excel at developing and optimising model configurations with effective collaborations. Hannah has made excellent progress in MOM-OASIS-CICE coupling. Ward provides a key conduit to understanding NCI optimisations. Heerdegen is supporting a multitude of operations. Fiedler continues to develop the new bathymetry.
  • The enhanced capabilities of CICE over SIS remain. CICE6 is being developed under a community framework. SIS2 has adopted some of the CICE code.
  • ACCESS is working towards the CMIP6 deadlines.
  • The blue link forecast models are now perceived to be limited more by the ocean model configuration than the data assimilation.
  • New model configurations (MOM5-OASIS-CICE, MOM5-FMS-SIS) using enhanced vertical resolution, JRA forcing (including an RYF forcing), and updated WOA initial conditions are underway.

COSIMA Community Discussion/Action Items from 2017 Workshop

  1. Develop website into more of a community portal to flagship simulations. Provide model configuration and analysis with low barrier to entry for users. (Spence)
  2. Develop the COSIMA Cookbook portal  with a gallery of options (Munroe). Improve the community usage and contribution to the cookbook. Provide a visual gallery of code examples (ALL).
  3. Provide online chat capability for COSIMA model configurations. Look into using Slack teams. (Spence)
  4. Provide list of ongoing experiments, links to data available and diagnostics wanted, contacts for expts. (Spence)
  5. Consider separating models configs into Tier 1 (control flagships), 2 (perturbations from control), 3 (parameter sensitivities) to aid communication. (Spence)
  6. Provide discussion platform for new experiments. Wider discussion of methodologies and diagnostics.
  7. Provide platform for efficient explorations of parameter spaces (sea ice, neutral physics, bgc params, SSS restoring). How to share configurations? Tracking configs with git needs to be embraced by all! (Aidan, Hogg)
  8. Explore CICE efficiency, coupling a wave model to CICE, CICE6-MOM6 capabilities. Be aware of similar progress being made at NCAR. (Heil).
  9. Consider hiring admin support for the COSIMA website (England).

ARC Linkage Project Discussion/Action Items from 2017 Workshop

  1. A new 1/10 degree model bathymetry data set continues to be underdevelopment with substantial progress towards a final product expected with a few months (Russ Fiedler)
  2. Hannah continues to develop the MOM5-OASIS-CICE5 configurations. Expects to provide spinups at 3 model resolutions to the communitu (1, 1/4, 1/10). Discussion regarding the CICE settings (Heil), vertical resolution (Stewart), JRA forcing (Stewart) are needed. Does the model currently have a closed heat budget (Dias)?
  3. Excited to welcome Andrew Kiss as the scientist funded from this ARC Linkage grant. Hoping for a model evaluation paper before the next meeting.

 

Attendees

(Where available, talk files are linked from the presenter’s name.)

Andy Hogg (ANU)
Gary Brassington (BoM)
Nic Hannah (Breakaway Labs)
Dave Bi (CSIRO)
Matthew England (UNSW)
Adele Morrison (ANU)
Ryan Holmes (UNSW)
Jan Zika (UNSW)
Angus Gibson (ANU)
Andy Kiss (ANU)
Veronique Lago (UNSW)
Laurie Meviel(UNSW)
Kaitlin Alexander (UNSW)
David Webb (UNSW)
Peter Dobrotoff (CSIRO)
Leela Frankcombe (UNSW)
James Munroe (Memorial, Canada)
Joan Llort (Utas)
Arnold Sullivan (CSIRO)
Xuebin Zhang (CSIRO)
Terry Okane (CSIRO, Remote)
Peter Oke (CSIRO)
Paul Spence (UNSW)
Richard Matear (CSIRO)
Clothilde Langlais (CSIRO)
Stephen Griffies (GFDL, Remote)
Russel Fiedler (CSIRO)
Marshall Ward (NCI)
Matthew Chamberlain (CSIRO)
Fabio Dias (UTas)
Aidan Heerdegen (ANU)
Petra Heil (AAD & ACE CRC)
Siobhan O’Farrell (Remote, CSIRO)
Xiaobing Zhou (BoM)

Technical Working Group Meeting, May 2017

Minutes

Date: 16th May 2017
Attendees:

  • Marshall Ward (NCI, Chair)
  • Aidan Heerdegen and Andy Hogg (ARCCSS ANU)
  • Scott Wales (ARCCSS Melbourne Uni)
  • Nicholas Hannah (ARCCSS/Double Precision)
  • Russ Fiedler and Matt Chamberlain (CSIRO Hobart)
  • Justin Freeman (BoM)
  • Peter Dobrohotoff and Roger Bodman (CSIRO Aspendale)

Updates

  • Roger is having issues with N95 atmosphere. Marshall says it doesn’t scale past 256 cores. Roger would like to get RTM profiling working. Martin and Peter have got multiple threads working in AMIP. Has spoken to Scott Wales. Something odd happens in decomposition. Marshall will ask Dale Roberts about effects of OpenMP. Has chemistry has been enabled for OpenMP?

Liaison with COSIMA Management

  • Andy is here to get some feedback about our activities. Andy thinks TWG is doing a good job of communicating, a goal is to get more communication amongst COSIMA in general. Management team meets quarterly. No science talk or detail. Have to be better at merging/converging disparate code, TWG is crucial for this.
  • Andy wants a better framework for analysing and post-processing runs, and access others outputs. And we are currently doing some work in this space. The effort it is scattered, no one lead. Nic has done some work in past. Now using ipython notebooks to share analysis. James Munroe working on dashboards. Justin asked if COSIMA will deliver this? Andy: not explicitly funded but will benefit uptake. Need better ways to serve data.
  • Andy would like more engagement around COSIMA website. Blog our progress? Marshall: do you have sample topics? Is it legitimate to put updates on issues with library versions say? Andy: borderline, maybe just TWG for that example. Currently reply on members of TWG to propagate information back to users. Maybe more relevant would be update on scaling of code for example. Don’t want to limit people now. Encourage as much as possible and filter if required. Peter agreed: get useful information up there. Haven’t had anything over the last year. Suggestion to make minutes more report like? Nic thought blog posts are a nice idea, but need deeper insight to be useful and interesting.
  • Andy wants science side to publish results of runs, and point to data.
  • Andy also keen for COSIMA to have information about model versions. ACCESS doesn’t have a way of releasing versions and hosting code. ACCESS is somewhat hobbled with partner disputes. Would like ACCESS-OM releases on COSIMA. Marshall pointed out the TWG was set up to address this. Models were not ready at that stage. Marshall suggested we make a proper plan for a release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model.
  • What are the expectations of TWG for the COSIMA Meeting? Andy: some already giving talks. Interact and discuss with others. Get to know each other and ambitions and look for synergies.
  • Andy will contact Paul to make sure TWG will have a slot to fill in others on progress.
  • Andy would like an email list for COSIMA announcements.

COSIMA Models

  • Nic: Was in Canberra week ago. Had tenth timestepping on <3000 cores. More than 3000 didn’t initialise. MCT couldn’t set up routing tables with more than 3000 cores. Would just hang. After discussions figured out some MPI switches and flags to get it working: mxm mtl makes it work better. Justin suggested this would make a good blog post. Marshall found MOM6 was failing at 3000 cores too. Went away with mxm mtl.
  • Andy: MOM-SIS tenth was also failing. About 30% fails. Russ has had similar. Nic now running on 6K cores in ocean.
  • Nic had discussions in Canberra around CICE halo updates. Made 12+ changes to CICE and MATM code. Made big improvements. For all three model resolutions (1,0.25,0.1) overhead of coupling is 1-2% compared to MOM solo. That is a tiny serial bit of interpolating forcing fields on to ocean. Like 20s/month for quarter degree.
  • Nic: quarter degree 1800s time step should be less than 75min/year. Andy: UNREAL!. 1 deg is also running super fast. 50 years/day. Did a new compile on MCT library to squeeze as much performance as possible.
  • Now have 3 new configs. 1 deg and 0.25 deg could be used. Focussing on tenth at the moment.
  • Andy: 70min MOM-SIS-025  + CORE, JRA55 adds 30%.
  • In old config, all models block waiting for MATM to read files. Now MATM has sent everything. Reduces difference between CORE and JRA55. Nic has not done longer runs as yet. No longer buffering multiple years of MATM output.
  • Agreed Andy should get these configs and do some longer runs.
  • Andy talking to NCAR about JRA55 forcing. CORE used NYF. JRA doesn’t have that. Others have used a single year. Our strategy is May-May forcing with a shock at the end of May. Candidate years are 84/85, 91/92, 03/04. Want to test this at 0.25 deg and 1deg. Should adopt just what Nic has done.
  • These are MOM-SIS as Andy wants a baseline. Doing MOM-SIS from CORE with WOA13. Repeat with JRA55 RYFs. Want to compare to ACCESS-OM config of Nic.
  • Nic and Andy to talk offline and try out a repeat year.
  • Although it is fast, the tenth is inefficient, as there is no ocean masking currently. This is the next priority. Probably beyond Nic’s current contract.
  • Nic can I use unmasked restarts? Russ: yes. Just need to combine them.

COSIMA Workshop

  •  Marshall: can we agree to transfer to CM2? To get on common version of CICE.
  • Andy: will the code we release in OM2 be different in ocean and/or ice? Can we manage it in one codebase? Marshall: should be possible. Set aside time to discuss this at meeting.
  • Discuss moving to common CICE repo for all.
  • Marshall: Justin need some info from us on OM config? Nic and Justin will liase.
  • Justin won’t be at the COSIMA Meeting, does he want us to cover anything? Justin: staying up to date with what we’re doing, will be engaging much more in future.

Updates on previous actions

  • Nic: has updated OceansAus repo to Peter’s CICE. Can Peter look at the code and check it.
  • Marshall: Justin need some info from us on OM config? Nic and Justin will liase.
  • Russ has been doing a lot of clicking for bathymetry. Aus and PNG done. Need help.

Actions

New:

  • Ask Dale Roberts about effects of OpenMP for Roger (Marshall)
  • Make a proper plan for model release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model (Marshall and TWG)
  • Contact Paul Spence about TWG speaking slot at meeting (Andy)
  • Prepare slides for TWG presentation at COSIMA meeting, and present (Aidan and Marshall)
  • Email list for COSIMA announcements (Aidan)
  • Blog post around issues with high core count jobs and mxm mtl (Nic)
  • Do longer runs with Nic’s 1 deg and 0.25 deg ACCESS-OM2-JRA55 configs (Andy and Aidan)
  • Try repeat year forcing with Nic’s configurations (Nic and Andy)

Existing:

  • Nic to present MATM code re-write proposal to TWG for feedback before sign-off. Will then be presented to Andy Hogg for approval.
  • Nic create a discussion document (on COSIMA?) to document current approaches and strategies for future
  • Move FMS to submodule of MOM5 github repo (Marshall). Liase with Nic on implementation?
  • Test Nic’s access-om model config on OceansAus (All)
  • Work up test cases to cover the nudging code (Justin, Mirko) and supply them to Nic.
  • Add new test cases to Jenkins test suite (Nic).
  • Start a new google doc about coupler issues and MATM (Marshall)

COSIMA Linkage Project

The COSIMA consortium is partly supported by an ARC Linkage Project. This project is now active, following sign-off from all partners in December 2016. The management committee for the project formally met for the first time on Friday 9th December. Following are abbreviated minutes from the meeting which outline our plans for 2017:

  1. To advertise for a Postdoctoral/Research Fellow to coordinate COSIMA projects.
  2. To enhance the vertical resolution of existing MOM5-SIS 0.1° model, with no change in horizontal resolution;
  3. To enhance model bathymetry in coastal regions to take advantage of reduced minimum depth stemming from vertical resolution changes;
  4. To couple MOM5 with CICE at 0.1° using OASIS3-MCT (we have a contract with Nic Hannah from Double Precision to perform this work);
  5. To evaluate, refine, optimise and document the new model configuration.

Furthermore, we have confirmed that the 2017 COSIMA meeting will be in Sydney on May 25-26.

Technical Working Group Meeting, October 2016

Minutes

Date: 11th October 2016
Attendees:

  • Marshall Ward (NCI, Chair)
  • Aidan Heerdegen (ARCCSS)
  • Nick Hannah (ARCCSS/Breakaway Labs)
  • Justin Freeman and Mirko Velic (BoM)
  • Russ Fiedler and Matt Chamberlain (CSIRO Hobart)
  • Peter Dobrohotoff (CSIRO Aspendale)

Code submissions

  • Mirko submitted major refactoring update to the nudging code for MOM. Three different options depending on namelist. One just sponge, one does nudging, another does adaptive nudging. Added instantaneous update from datafile. Wanted to reproduce the MOM4 behaviour. Tested, and now works. Was broken previously.
  • Can merge, but we need some testing on other coverage. Currently have a dozen test cases. Not sure any touch this, but will run them anyway. Justin suggests they provide a test case which covers some of these sections.
  • Nic asked that if possible functional and formatting changes be separate commits, as it makes approving pull requests much easier
  • Maybe not merge yet, but get testing working to cover this. Justin will look at adapting an existing MOM test case for this purpose.

Exchange grids and smoothing

  • Justin was talking to Paul Sandery about exchange grids. An issue with tiling as a result of remapping. Was asking about how Russ implemented smoothing.
  • If you took interim fields end up with horrible pattern with convergence of winds with 1st or 2nd order remapping due to discontinuity. Russ wrote some code that does 2D smoothing within the surface boundary condition. Bypasses the exchange grid and used the flux exchange to native grids options(?). GFDL apply an interpolation when they read in via data override. So can use the data override to interpolate to the finer grid and can control this.
  • This is only a problem with conservative remapping with exchange grids.
  • Nic didn’t think this was a problem with standard MOM-SIS runs, but Russ said it should still be visible in the fluxes with coarse (1deg) forcing fields.
  • With ACCESS high res ocean the fields and fluxes are extremely blocky, so Nic smooths on the ice grid, before it comes into the ocean grid, on a tile by tile basis.
  • If you want local conservation, cannot get around this. In ACCESS can use linear interpolation and then post-process to get global conservation. Doesn’t work with local conservation.
  • Marshall suggested we have some test cases that don’t run the model but test coupling and fields
  • These effects most often seen when there is a big difference in resolution between model fields and input fields. Look at wind stress fields. Maybe some of the barotropic fields, height and definitely convergence in barotropic restart file.
  • Paul’s runs do not use conservative remapping. Don’t see the horrible features with some of the other schemes.
  • Nic: do we need a central document discussing this?

OceanMaps 4

  • Justin is trying to prototype OceanMaps 4. Picking up on Paul Sandery’s work. He has been using MOM5-SIS and using bulk fluxes to link the models. Would like to standardise, or make these things available. Not sure how it connects to linkage project.
  • Nic felt it was good to know what Paul does. So far no code changes?.

FMS

  • Aidan got a query from Dave Hutchinson, asking if latest version of FMS was included in the code on MOM5 repo. Marshall has updated FMS in the master branch to Ulm, but not to Verona, the latest version.
  • Move FMS to a submodule of MOM5 rather than manually included inline
  • Goal is for Rui Yang (NCI) to work on parallel netCDF in MOM5

Model release naming and definition

  • Still an issue
  • Nic has put an access-om model on OceansAus. Has version controlled input files and code. Can be downloaded, compiled and run.

CICE

  • ACCESS-OM models are using CICE4.?, but Peter is using 5.1.
  • There are many bug fixes and performance improvements in the version of CICE Nic has been working on that would be beneficial to Peter.
  • Peter is working on a refactoring of CICE5.1
  • We should align our work to the same version of CICE.
  • First step is for Peter’s version of CICE5.1 to be hosted on OceansAus and development work to be based from that so we can work together. Some discussion about the best way to do this.

Actions

  • Just and Mirko work up test cases to cover the nudging code and give them to Nic.
  • Nic to add new test cases to Jenkins test suite.
  • Aidan to add mom-ocean.org and mom-ocean.org.au to uptime monitoring service (Uptime Robot).
  • Add Peter’s CICE5.1 config to OceansAus github repo
  • Nic create a discussion document (on COSIMA?) to document current approaches and strategies for future
  • Marshall to move FMS to submodule of MOM5 github repo. Liase with Nic on implementation?
  • Others test Nic’s access-om model config on OceansAus

COSIMA I: Workshop Report

The first meeting of the Consortium for Ocean Sea Ice Modelling in Australia (COSIMA) was held in Hobart on the 26-27 May 2016. There we 38 attendees (20 of whom gave presentations), representing 10 different institutions. A full participant list is included at the end of this report.

IMG_1150

The goal of the workshop was to formalise a consortium across universities, BoM, AAD and CSIRO to build global model configurations that will form the basis for high-resolution forecasting, reanalysis, process modelling and ultimately coupled climate modelling. The consortium recently received 4 years of funding from an ARC Linkage Project to build a model configuration which underpins a variety of applications.

Workshop Highlights

Presentations included an update on the current status and near-term plans of the major modelling groups, recent scientific advances using ocean and sea ice models and highlights of technical advances in model development. Some edited highlights are listed below:

  • Uptake of the MOM5 model for global applications has progressed well in the last 5 years (Bi). It is currently used broadly across climate and ocean-only configurations at both 1° (O’Kane) and 0.25° resolution (Spence, Holmes). Developments at 0.1° are proceeding (Zhang, Langlais, Matear, Chamberlain, Hogg).
  • Other models with a strong user base in Australia include ROMS (Galton Fenzi) for near-coastal and near-Antarctic applications, MITgcm for regional GFD-style simulations (Nikurashin) and NEMO (Alves) for seasonal prediction.
  • There is a strong need to consider vertical resolution in future high resolution model developments (Stewart).
  • The CICE sea ice model, while no longer under active development at LANL, continues to be the ice model of choice (due primarily to its superior ice physics over SIS 1; Heil, Reid, Bennetts). It is likely that it will remain the model of choice with applications using MOM5.
  • Our MOM-CICE implementations using OASIS3-MCT coupling, but it is not clear that this solution will scale to 0.1° and beyond (Hannah)
  • MOM6 is rapidly developing, and is beginning to gather users within Australia. It looks viable to use MOM6 for global models in the next year or two (Griffies, Gibson).
  • MOM6, being a C-grid model, presents some challenges when working with a B-grid ice model such as CICE. For the time being, it seems that the best ice model to use with MOM6 will be SIS2, which is actively being developed at GFDL to incorporate the vertical physics (Griffies).
  • Computational performance of these models shows that MOM5 is marginally faster than MOM6 (once accounting for vertical resolution differences) and that both models scale well on Raijin. NEMO is faster at small core counts but scales poorly (Ward).
  • Forced ocean-sea ice models should be transferring to the JRA-55 forcing set when possible (Marsland).

The COSIMA Community

There was significant support for the formation of a community of ocean-sea ice modellers around the COSIMA banner:

  • Need to create a website to outline COSIMA activities and developments. [We are in the process of acquiring the domain name cosima.org.au, and ANU will fund hosting and a web development team to put together a skeleton site.]
  • We will formalise a code of ethics for COSIMA users to abide by, based on the DRAKKAR agreement. [Spence]
  • We aim to register members on the website, and create a mailing list
  • We will need to formalise the use of technologies to share code configurations, analysis tools and data. It may be possible to have a data project code on NCI to help with this.
  • We will release flagship configurations that are broadly supported by the community. The goal is to make the naming conventions consistent with the ACCESS community where possible, and to overlap with ACCESS developments where possible.
  • COSIMA will hold an annual meeting in the last week of May, for two days. Venue will rotate around the partner institutions. The focus of the meeting will be on science applications of ocean models, but will also include a technical component.
  • We will aim to have more regular communications, including newsletters and video meetings.
  • We will establish working groups within the community, along with a working group chair. Proposed groups include
    • Sea Ice Modelling [Heil]
    • Technical [Ward]
    • Linkage Project [Hogg]
  • The major gap in the community was identified to be sea ice modelling and forecasting. We will all look for opportunities to attract visitors and expertise in this space.

Linkage Project

One of the workshop goals was to receive advice from the community regarding the ARC Linkage Project designed to support COSIMA development activities. Major items of discussion were:

  • There was general agreement that the Linkage Project should fund both the development and evaluation of new model configurations. This point implies that we should equally fund the technical and postdoctoral position, despite the partial funding of the program.
  • In the first year we will look to upgrade current MOM5 implementations, focussing on the vertical grid and the incorporation of CICE.
  • In subsequent years we will look to adopt a MOM6 configuration.
  • As configurations develop and have been properly evaluated, they will be distributed to the community.
  • Suggestions on evaluation include using ESMVal.

Attendee List

(Where available, talk files are linked from the presenter’s name.)

Andy Hogg (ANU)
Gary Brassington (BoM)
Ben Galton-Fenzi (AAD)
Nic Hannah (Breakaway Labs)
Dave Bi (CSIRO)
Oscar Alves (BoM)
Max Nikurashin (UTas)
Simon Marsland  (CSIRO)
Kial Stewart (ANU)
Xuebin Zhang (CSIRO)
Terry O’Kane (CSIRO)
Paul Spence (UNSW)
Richard Matear (CSIRO)
Ryan Holmes (UNSW)
Clothilde Langlais (CSIRO)
Stephen Griffies (GFDL)
Angus Gibson (ANU)
Russ Fiedler (CSIRO)
Marshall Ward (NCI)
Justin Freeman (BoM)
Phil Reid (BoM)
Luke Bennetts (Adelaide)
Matt Chamberlain (CSIRO)
Scott Carpentier (BoM)
Fabio Dias (UTas)
Stephanie Downes (UTas)
David Gwyther (UTas)
Aidan Heerdegen (ANU)
Petra Heil (AAD & ACE CRC)
Andreas Klocker (UTas)
Andrew Lenton (CSIRO)
Jan Lieser (UTas)
Sebastien Moreau (CSIRO)
Siobhan O’Farrell (CSIRO)
Paul Barker (UNSW)
Mirko Velic (BoM)
Xiaobing Zhou (BoM)
Stefan Zieger (BoM)

 

Southern Ocean Animation

This detailed animation of the movement of the densest and coldest water in the world around Antarctica has been produced using data generated on Australia’s most powerful supercomputer, Raijin.

So much data was used, that it took seven hours to process just one second of the animation.

The visualization has revealed underwater ocean storms generated by eddies, waterfalls of cold dense water that plummet two kilometres off the Antarctic Continental Shelf into the abyss and underwater waves hundreds of metres high.

This latest animation peels back much of the surface layer of the ocean to explore how the cold dense water produced on the Antarctic continental shelf spreads out into every ocean basin in the world.

The movement of this dense water is vital. It is the most oxygenated water in the ocean and its extreme density and coldness drive many of the significant currents in the major ocean basins connected to the Southern Ocean.

The distinctly different densities of water that move around Antarctica also make it important in regards to climate change. Because the most dense water forms near the surface of Antarctica before descending to the ocean floor, any warming that occurs near the surface can be drawn down into the deep ocean.

Importantly, this drives more heat and more carbon into the deep ocean that would otherwise have returned to the atmosphere.