Technical Working Group Meeting, November 2017

Minutes

Date: 14th November 2017
Attendees:

  • Aidan Heerdegen (Chair), Andrew Kiss (ARCCSS ANU)
  • Fanghua Wu (National Climate Center, China Meteorological Administration, Visitor ANU)
  • James Munroe (Memorial University of Newfoundland, Visitor ANU)
  • Nicholas Hannah, Anthony (Double Precision)
  • Russ Fiedler and Matt Chamberlain (CSIRO Hobart)

COSIMA Models

  • Discussion around publicising 1/10th model spin up, in case interested parties would like diagnostics saved.
  • Bluelink are interested in full JRA55-do IAF style spin up, and would want 15-20 years of daily full 3D U,V,T,S and eta fields from that. What is required to construct ensembles/climatologies.
  • Nic looking into ACCESS-OM2-01 performance issues. Lots of time in ice coupling field halo updates. In serial so holding up ocean when it does this. Definite target for optimisation. Should use OASIS to fill the halos when it does the coupling step? Russ disagrees. OASIS shouldn’t know anything about what goes on in models. Gridding using block trains, a 1:1 mapping between grids. If you do this have a 1:many mapping. No longer have identical grids when put in halo information, might break optimisations. When Russ looked at 1/4 deg, hold up was due to synchro just before that. Not sure about 1/10th. Want a barrier just before calling clock before halo update. See if synchronisation issue, or actual time take with halo distribution. 5 halo distributions being done. Heaps more done in CICE itself. Nic: land imbalance between ice processors? Russ: yes my hunch. Load imbalances change a lot with resolution and processor layout. Nic: a problem doing halo updates without considering where field is used. Russ: agree. Velocities need updating, not sure about tracers.
  • Fanghua has been running the new tenth bathymetry with the MOM-SIS-KDS75 config. With JRA55 RYF forcing time step now 450s (from 150s initially). Runoff data now a problem with very low salinities in the arctic at about 7m depth, even with 150s timestep. Created new runoff data, spread more into the ocean but still have issues. Russ saw very high salinities in the Arctic (Laptev Sea). Might be brine rejection from forming sea ice from ice free start. Suggests decreasing salinity restoring timescale from current 60 days to 10 days or even 1 day, to get the model over the initialisation. Andrew suggested issue could be resolved with initial sea ice climatology. There were issues with these files and not been used for a long time. Recent poster to mom users google group has identified some of the problems.
  • Nic’s online runoff redistribution may help, as it is possible to specify maximum runoff per cell, which can help in these areas with very large runoff. Would require ACCESS-OM2-01 config.
  • Nic currently working on getting ACCESS-OM2-01 working with Russ’ new bathymetry. Had a couple of attempts. Getting close, various technical glitches with masks and so on.
  • Andy Hogg has MATM issue when running ACCESS-OM2-1deg for more than 4 years at a time. There is an error on netCDF open call, which comes from HDF layer. Nic ran valgrind, found a bunch of errors, and so recommends everyone update their MATM, but this did not fix the 5 year issue. Determined this was not a memory errors, but an HDF library error. Russ suggested using some HDF library calls to try and determine why the crash occurred. Also try different versions of the netcdf library.
  • Nic suggested we could change MATM to make few file open calls. Aidan has a new payu feature that allows multiple runs per PBS submission, so decided not a priority as MATM needs complete rewrite.
  • Regridding. Nic: need to choose which interpolation schemes to use for which fields. 2nd order cons for everything? Russ: Velocity should not be conservative. Momentum is not conserved. Patch for velocities, T and S. Will give smooth flux fields. Nic: 2nd order cons will be very smooth. Russ: do whatever is cheaper for T, S. U,V should be as smooth as possible. Patch should be 1st order cons, possible 1nd order.  AK: 1st order cons is piecewise constant (bad for wind stress curl). 2nd order is piecewise linear? So similar to bilinear. Need to go to patch for smoother. Russ: tried 2nd order cons, see problems at corners, nodes and edges with wind stress curl. Coarse to fine get artefacts. Patch should work. AK: half of the fields are fluxes. Those should be conservative (2nd order ideally). The remaining are not fluxes, don’t see strong argument for conservative. Is there an issue with different interpolation schemes from different fields? Will bulk formula at fine scale be an issue? Russ: will get jumps in some of the calculated fields. Quantities like T, S should be done with patch, end up with smooth fluxes. AK: Surface stress bulk formula, does it take atmosphere stability into account? Any drag coefficient? Russ: it does. Looks at a profile, figure out a profile. AK: Use SST and 10m T to determine stability? MC: Yes. Say warm atmosphere sitting over cold surface, that’s stable so air would slide over. Daytime, warm surface, near neutral stability so not so sensitive. Possible for temperature and humidity to have small effect on drag coefficient. AK: If we use different interpolation method for 10m winds/T, will it cause issues? Russ: Small jump in sensible heat maybe? Just go with patch or bilinear for all scalar quantities. Velocity go for patch. How will it take into account rotation in tripolar? Presume it is handled  well? AK: only an issue with velocity. Checked with current forcing fields and was ok. Will check new fields the same way.
  • AK: Final decision:
    • patch (the smoothest available) for u_10, v_10
    • 2nd order conservative for fluxes (rain, rdls, rsds, runoff_all and snow)
    • patch or bilinear for non-flux scalars (q_10, slp, t_10) suggest trying patch and only using bilinear if performance with patch is bad
  • Nic: what does MOM-SIS do? Aidan: Thought Steve said bicubic, used to use bilinear but wasn’t smooth enough. Smoother the better.
  • AK: Should WOA salinity restoring fields be smooth in the same way? Nic: What do we currently do? Nic: bilinear? Aidan did it. Russ: not a big issue if salinity restoring not too strong.

Task follow ups

  • Should be using GFDL FMS code directly. Would work better to collaborate with GFDL. Use same code, submit bug reports easily.
  • Once we have FMS as submodule, use all pre/post processing code from GFDL. Make MOM5 leaner, easier to keep updated. Russ: what is the latest FMS version? Aidan: don’t know, and it is hard to tell. Russ: noticed there are new features, like new diagnostic output options, e.g. RMS on the fly, statistics. So things like diag_manager has been updated. Could be some other powerful tools.
  • Aidan: Currently huge step to upgrade. Small step, but could be really good. Not sure how Marshall did it, but not simple.
  • Nic has updated the access-om2 repo structure. Every single test case/experiment is in it’s own repo. Makes it easier for users to grab config without worrying about other configs and source code. OceansAus now has more experiment repos. Aidan: Andy has an issue with git clashes with multiple runs in a single repo. This will fix this.
  • Blog posts?

Actions

New:

  • Will have a December meeting. Tue 12th.
  • Determine if COSIMA intend to do IAF JRA55 spinup of tenth model (Aidan)
  • Send link to spinup diagnostics spreadsheet to Russ (Andrew Kiss)
  • Nic add MPI barrier before ice halo updates timer to check if slow timing issues are just ice load imbalances that appear as longer times due to synchronisation.
  • Test Andy’s 5 year config with different netcdf library versions to check MATM error is not a just a library issue (Aidan)
  • Check current sea surface salinity restoring smoothing (Aidan)

Existing:

  • Russ to add all his ocean bathymetry code to OceansAus repo.
  • Nic to help Peter get his MOM repo up to date with MOM5 master branch, and then merge changes
  • Look into OpenDAP/THREDDS for use with MOM on raijin (Aidan, Nic, Marshall)
  • Nic to present MATM code re-write proposal to TWG for feedback before sign-off. Will then be presented to Andy Hogg for approval.
  • Nic create a discussion document (on COSIMA?) to document current approaches and strategies for future
  • Move FMS to submodule of MOM5 github repo (Marshall). Liase with Nic on implementation?
  • Work up test cases to cover the nudging code (Justin, Mirko) and supply them to Nic.
  • Add new test cases to Jenkins test suite (Nic).
  • Start a new google doc about coupler issues and MATM (Marshall)
  • Ask Dale Roberts about effects of OpenMP for Roger (Marshall)
  • Make a proper plan for model release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model (Marshall and TWG)
  • Blog post around issues with high core count jobs and mxm mtl (Nic)
  • Create document outlining options for configuration sharing (?)