Peter Dobrohotoff and Roger Bodman (CSIRO Aspendale)
Roger is having issues with N95 atmosphere. Marshall says it doesn’t scale past 256 cores. Roger would like to get RTM profiling working. Martin and Peter have got multiple threads working in AMIP. Has spoken to Scott Wales. Something odd happens in decomposition. Marshall will ask Dale Roberts about effects of OpenMP. Has chemistry has been enabled for OpenMP?
Liaison with COSIMA Management
Andy is here to get some feedback about our activities. Andy thinks TWG is doing a good job of communicating, a goal is to get more communication amongst COSIMA in general. Management team meets quarterly. No science talk or detail. Have to be better at merging/converging disparate code, TWG is crucial for this.
Andy wants a better framework for analysing and post-processing runs, and access others outputs. And we are currently doing some work in this space. The effort it is scattered, no one lead. Nic has done some work in past. Now using ipython notebooks to share analysis. James Munroe working on dashboards. Justin asked if COSIMA will deliver this? Andy: not explicitly funded but will benefit uptake. Need better ways to serve data.
Andy would like more engagement around COSIMA website. Blog our progress? Marshall: do you have sample topics? Is it legitimate to put updates on issues with library versions say? Andy: borderline, maybe just TWG for that example. Currently reply on members of TWG to propagate information back to users. Maybe more relevant would be update on scaling of code for example. Don’t want to limit people now. Encourage as much as possible and filter if required. Peter agreed: get useful information up there. Haven’t had anything over the last year. Suggestion to make minutes more report like? Nic thought blog posts are a nice idea, but need deeper insight to be useful and interesting.
Andy wants science side to publish results of runs, and point to data.
Andy also keen for COSIMA to have information about model versions. ACCESS doesn’t have a way of releasing versions and hosting code. ACCESS is somewhat hobbled with partner disputes. Would like ACCESS-OM releases on COSIMA. Marshall pointed out the TWG was set up to address this. Models were not ready at that stage. Marshall suggested we make a proper plan for a release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model.
What are the expectations of TWG for the COSIMA Meeting? Andy: some already giving talks. Interact and discuss with others. Get to know each other and ambitions and look for synergies.
Andy will contact Paul to make sure TWG will have a slot to fill in others on progress.
Andy would like an email list for COSIMA announcements.
Nic: Was in Canberra week ago. Had tenth timestepping on <3000 cores. More than 3000 didn’t initialise. MCT couldn’t set up routing tables with more than 3000 cores. Would just hang. After discussions figured out some MPI switches and flags to get it working: mxm mtl makes it work better. Justin suggested this would make a good blog post. Marshall found MOM6 was failing at 3000 cores too. Went away with mxm mtl.
Andy: MOM-SIS tenth was also failing. About 30% fails. Russ has had similar. Nic now running on 6K cores in ocean.
Nic had discussions in Canberra around CICE halo updates. Made 12+ changes to CICE and MATM code. Made big improvements. For all three model resolutions (1,0.25,0.1) overhead of coupling is 1-2% compared to MOM solo. That is a tiny serial bit of interpolating forcing fields on to ocean. Like 20s/month for quarter degree.
Nic: quarter degree 1800s time step should be less than 75min/year. Andy: UNREAL!. 1 deg is also running super fast. 50 years/day. Did a new compile on MCT library to squeeze as much performance as possible.
Now have 3 new configs. 1 deg and 0.25 deg could be used. Focussing on tenth at the moment.
Andy: 70min MOM-SIS-025 + CORE, JRA55 adds 30%.
In old config, all models block waiting for MATM to read files. Now MATM has sent everything. Reduces difference between CORE and JRA55. Nic has not done longer runs as yet. No longer buffering multiple years of MATM output.
Agreed Andy should get these configs and do some longer runs.
Andy talking to NCAR about JRA55 forcing. CORE used NYF. JRA doesn’t have that. Others have used a single year. Our strategy is May-May forcing with a shock at the end of May. Candidate years are 84/85, 91/92, 03/04. Want to test this at 0.25 deg and 1deg. Should adopt just what Nic has done.
These are MOM-SIS as Andy wants a baseline. Doing MOM-SIS from CORE with WOA13. Repeat with JRA55 RYFs. Want to compare to ACCESS-OM config of Nic.
Nic and Andy to talk offline and try out a repeat year.
Although it is fast, the tenth is inefficient, as there is no ocean masking currently. This is the next priority. Probably beyond Nic’s current contract.
Nic can I use unmasked restarts? Russ: yes. Just need to combine them.
Marshall: can we agree to transfer to CM2? To get on common version of CICE.
Andy: will the code we release in OM2 be different in ocean and/or ice? Can we manage it in one codebase? Marshall: should be possible. Set aside time to discuss this at meeting.
Discuss moving to common CICE repo for all.
Marshall: Justin need some info from us on OM config? Nic and Justin will liase.
Justin won’t be at the COSIMA Meeting, does he want us to cover anything? Justin: staying up to date with what we’re doing, will be engaging much more in future.
Updates on previous actions
Nic: has updated OceansAus repo to Peter’s CICE. Can Peter look at the code and check it.
Marshall: Justin need some info from us on OM config? Nic and Justin will liase.
Russ has been doing a lot of clicking for bathymetry. Aus and PNG done. Need help.
Ask Dale Roberts about effects of OpenMP for Roger (Marshall)
Make a proper plan for model release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model (Marshall and TWG)
Contact Paul Spence about TWG speaking slot at meeting (Andy)
Prepare slides for TWG presentation at COSIMA meeting, and present (Aidan and Marshall)
Email list for COSIMA announcements (Aidan)
Blog post around issues with high core count jobs and mxm mtl (Nic)
Do longer runs with Nic’s 1 deg and 0.25 deg ACCESS-OM2-JRA55 configs (Andy and Aidan)
Try repeat year forcing with Nic’s configurations (Nic and Andy)
Nic to present MATM code re-write proposal to TWG for feedback before sign-off. Will then be presented to Andy Hogg for approval.
Nic create a discussion document (on COSIMA?) to document current approaches and strategies for future
Move FMS to submodule of MOM5 github repo (Marshall). Liase with Nic on implementation?
Test Nic’s access-om model config on OceansAus (All)
Work up test cases to cover the nudging code (Justin, Mirko) and supply them to Nic.
Add new test cases to Jenkins test suite (Nic).
Start a new google doc about coupler issues and MATM (Marshall)
The COSIMA consortium is partly supported by an ARC Linkage Project. This project is now active, following sign-off from all partners in December 2016. The management committee for the project formally met for the first time on Friday 9th December. Following are abbreviated minutes from the meeting which outline our plans for 2017:
To advertise for a Postdoctoral/Research Fellow to coordinate COSIMA projects.
To enhance the vertical resolution of existing MOM5-SIS 0.1° model, with no change in horizontal resolution;
To enhance model bathymetry in coastal regions to take advantage of reduced minimum depth stemming from vertical resolution changes;
To couple MOM5 with CICE at 0.1° using OASIS3-MCT (we have a contract with Nic Hannah from Double Precision to perform this work);
To evaluate, refine, optimise and document the new model configuration.
Furthermore, we have confirmed that the 2017 COSIMA meeting will be in Sydney on May 25-26.
Mirko submitted major refactoring update to the nudging code for MOM. Three different options depending on namelist. One just sponge, one does nudging, another does adaptive nudging. Added instantaneous update from datafile. Wanted to reproduce the MOM4 behaviour. Tested, and now works. Was broken previously.
Can merge, but we need some testing on other coverage. Currently have a dozen test cases. Not sure any touch this, but will run them anyway. Justin suggests they provide a test case which covers some of these sections.
Nic asked that if possible functional and formatting changes be separate commits, as it makes approving pull requests much easier
Maybe not merge yet, but get testing working to cover this. Justin will look at adapting an existing MOM test case for this purpose.
Exchange grids and smoothing
Justin was talking to Paul Sandery about exchange grids. An issue with tiling as a result of remapping. Was asking about how Russ implemented smoothing.
If you took interim fields end up with horrible pattern with convergence of winds with 1st or 2nd order remapping due to discontinuity. Russ wrote some code that does 2D smoothing within the surface boundary condition. Bypasses the exchange grid and used the flux exchange to native grids options(?). GFDL apply an interpolation when they read in via data override. So can use the data override to interpolate to the finer grid and can control this.
This is only a problem with conservative remapping with exchange grids.
Nic didn’t think this was a problem with standard MOM-SIS runs, but Russ said it should still be visible in the fluxes with coarse (1deg) forcing fields.
With ACCESS high res ocean the fields and fluxes are extremely blocky, so Nic smooths on the ice grid, before it comes into the ocean grid, on a tile by tile basis.
If you want local conservation, cannot get around this. In ACCESS can use linear interpolation and then post-process to get global conservation. Doesn’t work with local conservation.
Marshall suggested we have some test cases that don’t run the model but test coupling and fields
These effects most often seen when there is a big difference in resolution between model fields and input fields. Look at wind stress fields. Maybe some of the barotropic fields, height and definitely convergence in barotropic restart file.
Paul’s runs do not use conservative remapping. Don’t see the horrible features with some of the other schemes.
Nic: do we need a central document discussing this?
Justin is trying to prototype OceanMaps 4. Picking up on Paul Sandery’s work. He has been using MOM5-SIS and using bulk fluxes to link the models. Would like to standardise, or make these things available. Not sure how it connects to linkage project.
Nic felt it was good to know what Paul does. So far no code changes?.
Aidan got a query from Dave Hutchinson, asking if latest version of FMS was included in the code on MOM5 repo. Marshall has updated FMS in the master branch to Ulm, but not to Verona, the latest version.
Move FMS to a submodule of MOM5 rather than manually included inline
Goal is for Rui Yang (NCI) to work on parallel netCDF in MOM5
Model release naming and definition
Still an issue
Nic has put an access-om model on OceansAus. Has version controlled input files and code. Can be downloaded, compiled and run.
ACCESS-OM models are using CICE4.?, but Peter is using 5.1.
There are many bug fixes and performance improvements in the version of CICE Nic has been working on that would be beneficial to Peter.
Peter is working on a refactoring of CICE5.1
We should align our work to the same version of CICE.
First step is for Peter’s version of CICE5.1 to be hosted on OceansAus and development work to be based from that so we can work together. Some discussion about the best way to do this.
Just and Mirko work up test cases to cover the nudging code and give them to Nic.
Nic to add new test cases to Jenkins test suite.
Aidan to add mom-ocean.org and mom-ocean.org.au to uptime monitoring service (Uptime Robot).
Add Peter’s CICE5.1 config to OceansAus github repo
Nic create a discussion document (on COSIMA?) to document current approaches and strategies for future
Marshall to move FMS to submodule of MOM5 github repo. Liase with Nic on implementation?
Others test Nic’s access-om model config on OceansAus
The first meeting of the Consortium for Ocean Sea Ice Modelling in Australia (COSIMA) was held in Hobart on the 26-27 May 2016. There we 38 attendees (20 of whom gave presentations), representing 10 different institutions. A full participant list is included at the end of this report.
The goal of the workshop was to formalise a consortium across universities, BoM, AAD and CSIRO to build global model configurations that will form the basis for high-resolution forecasting, reanalysis, process modelling and ultimately coupled climate modelling. The consortium recently received 4 years of funding from an ARC Linkage Project to build a model configuration which underpins a variety of applications.
Presentations included an update on the current status and near-term plans of the major modelling groups, recent scientific advances using ocean and sea ice models and highlights of technical advances in model development. Some edited highlights are listed below:
Uptake of the MOM5 model for global applications has progressed well in the last 5 years (Bi). It is currently used broadly across climate and ocean-only configurations at both 1° (O’Kane) and 0.25° resolution (Spence, Holmes). Developments at 0.1° are proceeding (Zhang, Langlais, Matear, Chamberlain, Hogg).
Other models with a strong user base in Australia include ROMS (Galton Fenzi) for near-coastal and near-Antarctic applications, MITgcm for regional GFD-style simulations (Nikurashin) and NEMO (Alves) for seasonal prediction.
There is a strong need to consider vertical resolution in future high resolution model developments (Stewart).
The CICE sea ice model, while no longer under active development at LANL, continues to be the ice model of choice (due primarily to its superior ice physics over SIS 1; Heil, Reid, Bennetts). It is likely that it will remain the model of choice with applications using MOM5.
Our MOM-CICE implementations using OASIS3-MCT coupling, but it is not clear that this solution will scale to 0.1° and beyond (Hannah)
MOM6 is rapidly developing, and is beginning to gather users within Australia. It looks viable to use MOM6 for global models in the next year or two (Griffies, Gibson).
MOM6, being a C-grid model, presents some challenges when working with a B-grid ice model such as CICE. For the time being, it seems that the best ice model to use with MOM6 will be SIS2, which is actively being developed at GFDL to incorporate the vertical physics (Griffies).
Computational performance of these models shows that MOM5 is marginally faster than MOM6 (once accounting for vertical resolution differences) and that both models scale well on Raijin. NEMO is faster at small core counts but scales poorly (Ward).
Forced ocean-sea ice models should be transferring to the JRA-55 forcing set when possible (Marsland).
The COSIMA Community
There was significant support for the formation of a community of ocean-sea ice modellers around the COSIMA banner:
Need to create a website to outline COSIMA activities and developments. [We are in the process of acquiring the domain name cosima.org.au, and ANU will fund hosting and a web development team to put together a skeleton site.]
We will formalise a code of ethics for COSIMA users to abide by, based on the DRAKKAR agreement. [Spence]
We aim to register members on the website, and create a mailing list
We will need to formalise the use of technologies to share code configurations, analysis tools and data. It may be possible to have a data project code on NCI to help with this.
We will release flagship configurations that are broadly supported by the community. The goal is to make the naming conventions consistent with the ACCESS community where possible, and to overlap with ACCESS developments where possible.
COSIMA will hold an annual meeting in the last week of May, for two days. Venue will rotate around the partner institutions. The focus of the meeting will be on science applications of ocean models, but will also include a technical component.
We will aim to have more regular communications, including newsletters and video meetings.
We will establish working groups within the community, along with a working group chair. Proposed groups include
Sea Ice Modelling [Heil]
Linkage Project [Hogg]
The major gap in the community was identified to be sea ice modelling and forecasting. We will all look for opportunities to attract visitors and expertise in this space.
One of the workshop goals was to receive advice from the community regarding the ARC Linkage Project designed to support COSIMA development activities. Major items of discussion were:
There was general agreement that the Linkage Project should fund both the development and evaluation of new model configurations. This point implies that we should equally fund the technical and postdoctoral position, despite the partial funding of the program.
In the first year we will look to upgrade current MOM5 implementations, focussing on the vertical grid and the incorporation of CICE.
In subsequent years we will look to adopt a MOM6 configuration.
As configurations develop and have been properly evaluated, they will be distributed to the community.
Suggestions on evaluation include using ESMVal.
(Where available, talk files are linked from the presenter’s name.)
This detailed animation of the movement of the densest and coldest water in the world around Antarctica has been produced using data generated on Australia’s most powerful supercomputer, Raijin.
So much data was used, that it took seven hours to process just one second of the animation.
The visualization has revealed underwater ocean storms generated by eddies, waterfalls of cold dense water that plummet two kilometres off the Antarctic Continental Shelf into the abyss and underwater waves hundreds of metres high.
This latest animation peels back much of the surface layer of the ocean to explore how the cold dense water produced on the Antarctic continental shelf spreads out into every ocean basin in the world.
The movement of this dense water is vital. It is the most oxygenated water in the ocean and its extreme density and coldness drive many of the significant currents in the major ocean basins connected to the Southern Ocean.
The distinctly different densities of water that move around Antarctica also make it important in regards to climate change. Because the most dense water forms near the surface of Antarctica before descending to the ocean floor, any warming that occurs near the surface can be drawn down into the deep ocean.
Importantly, this drives more heat and more carbon into the deep ocean that would otherwise have returned to the atmosphere.