Technical Working Group Meeting, November 2019


Date: 27th November, 2019
  • Aidan Heerdegen (AH) CLEX ANU, Angus Gibson (AG) ANU,  Andrew Kiss (AK)  COSIMA ANU
  • Russ Fiedler (RF), Matt Chamberlain (MC) CSIRO Hobart
  • Rui Yang (RY), Paul Leopardi (PL) NCI
  • Nic Hannah (NH) Double Precision
  • Marshall Ward (MW) GFDL

ACCESS-OM2 on gadi

PL: Submodules not updated (#176). Reported bug from CICE5 but not being built. AK: not sure how to release this. Sometimes model components updated but not tested. AH: gadi transition branch? AK: Yes. PL: Science bug.
PL: To test had to copy files around. Needed to update config.yaml and atmosphere.json. Made fork of 1deg_JRA55_RYF for testing. Had to move to non-public places as don’t have access to public places. Will send details in an email.
PL: conda/analysis3-unstable needs to be updated, payu not working on gadi. AH: Did update, still not working. Update only tested on interactive job. PBS job strips out environment. Wanted to consult with Marshall about why payu works as it does currently. Difficult to debug as payu-run as it does not have the same environment as “payu run”. PL: Work-around to add -V option to qsub_flags in config.yaml. AH: This is what I am considering to change payu to by default. Not sure. Currently looking into this.
PL: nccmp module not on gadi. Been using for reproducibility testing. In backlog. RY: Can install personally, don’t have to wait for system install.
PL: Running on gadi. Got 1 deg RYF55 finished. Did not have mppnccombine compiled. Will have to do this to get this working correctly. Got something for baseline for comparison. Report by the end of the week.
RY: gadi 48 cores. Default based on broadwell (28 cores). Do you have an up to date config? Paul currently changes core count in his config, but is it done in official config?
AH: I was in the process of making an official configuration for gadi. Copied all inputs that were in /short/public to the ik11 project. Once directory structure finalised will make a config that runs, update on GitHub, and look at making the same changes for other configs. Make an exemplar config with those changes. RY: Should work on same configs.
RY: Anyone else running on gadi? AH: No.
AH: What are the impediments to others updating ACCESS-OM2 on GitHub? People not sure if they can? How they should go about tit? AK: Put my hand up to do this. Other model components also need updating. AH: Maybe dev branch that everyone pulls from. Easier to make changes without worrying about breaking. So everyone working from the same version and don’t have to re-fix known bugs.
AH: Environment stuff? MW: Something about python exec command. Nuance? Wholesale copy everything? Wanted to create idealised processes, rather  than depend on what users haves stop. payu run submits job to PBS with whole new environment. Explicitly give environment variables.
AH: Drawback payu-run does not use same environment as payu run. MW: Not launching a process. payu run submits to PBS and starts posix process with defined environment. Exception when explicitly give it environment variables. AH: One work-around is to make list of environment variables want to keep. Losing MODULEPATH variables. PL: module env being used by payu required modules 3. Modules 4 works differently. Python code from modules 4 may work better.
MW: Fixed? AH: Thought I had, but was fooled because using payu-run. MW: If you set MODULEPATH locally, it won’t be exported to payu run process.
PL: What is the fix? MW: On raijin there was a bootstrap script in init dir, which sets everything. I duplicated those commands and put them in the payu module that did equivalent bootstrap. If moving to gadi and it is different none of that bootstrap script works. PL: Bootstrap script there, but completely different. MW: Was old version, and never actually used the bootstrap script. Maybe exec the bootstrap script they provide? AH: Or pass through environment variables that are set already. MW: Do whatever you think is best. Did try and make it so ‘payu run’ job was clean and always looked the same regardless of who submits. If we take entire ENV and submit to run, every run will be different. One variable is a controlled solution. Solution should be possible to have job on submitted node can set it up on it’s own. Should get it going and not be held up by my purist notions. AH: Try/except blocks can be used to support multiple approaches. MW: Definitely need to bootstrap the modules. PL: Sent through email with details.

OpenMPI/4.0.1 on gadi

AH: Angus reported openmpi/4.0.1 seems broken. Has this been fixed?
AG: Any wrapped commands (mpicc, mpifort) will print whitespace before output. In most cases ok, but can break configure scripts. Ben M knows about it, but not why.
PL: Divide by zero error in MPI_Init. MW: Remember that one UCX back-end, FP exception. Evaluates a log function when evaluating binary tree when working out communication. Ben M told them about it, but got nothing back. We use FP exception checking, but can’t ignore for just MPI. PL: Work-around like turn off UCX? MW: Could turn off FP exceptions. A race condition, so not every job sees it. RY: Can turn off UCX. Can use ob1 instead of UCX. Also try that. PL: Wasn’t sure it would work on gadi.
AH: Maybe 4.0.1 not a good candidate for testing? Get intermittent crashes.

Russ update on model performance on gadi

RF: Been testing OFAM bluelink, compiled as MOM-SIS without doing ice. Performance was fantastic. 2x faster than Sandy Bridge. Don’t get hammered with extra cost on new CPUs. Initialisation was very fast. A lot of files, so might be a low load issue. Dropped from 100s to 8s. Doing data assimilation runs, run 3 days at a time. 25% of the run time was init. Now pretty much zero. MOM5 performance was really good.
RF: Did notice some variation on start up of CM4. Still a lot faster. Reads in a lot more files and a lot more data. Still considerably faster than on raijin. MW: MOM has IO timers, do you have those on? FMS timers. Rui used them a lot. RF: No, didn’t turn them on.
RF: Running CM4 was about 15% faster than Broadwell. Improved but will cost a lot more for decadal prediction. RY: 15% is normal. Martin report UM is 30% quicker. RF: SIS2 load balance is bad. Probably a bunch of things being covered up. Needs more testing.
MW: Bob has never talked about SIS2 load imbalance. Presumably oblivious to them. RF: Would have to be. Regular layout would lead to many redundant processors. MW: Alistair has done some iceberg code load balance improvements. RF: Doesn’t take much time. Had to turn off iceberg stuff on raijin. netcdf stuff broke it. Might turn back on. Time spent in iceberg code minimal.

Stack array errors and heap array option

RF: When compiling need to set heap-arrays option in compiler, otherwise get segfaults with stack, even when stack set to unlimited. Wasn’t an issue on raijin. Happened for both MOM5 and CM4. PL: Dale mentioned about stack size limited to 8MB. RF: I unlimited stack size, so shouldn’t have been an issue. Got all sorts of issues with unmapped addresses. First one saw it was automatic so tried moving to allocatable, moved error. Then tried different heap-arrays size options, which moved error again. MOM5 dropped to heap-arrays 5KB. Same for CM4 but set to zero for SIS2 and it got through. Different models, seems ubiquitous. MW: Intel fortran?
MW: When compile and run on CRAY machines stack vars use malloc, so heap variables not stack. Same model, same compiler on laptop (gcc), same variables are stack variables. Is it possible moving from raijin to gadi something different about malloc. RY: CentOS 7 v 8 makes some difference. MW: Is kernel making some decisions on malloc? RY: Had similar issues with UM. Stacksize unlimited seemed to fix for UM. But Dale talked about this in ACCESS meeting, kernel changed something that caused this problem.
NH: Intel compiler has heap always arrays option. Useful in some cases. Models can have array bounds overruns, and easier to track when trash heap compared to stack. RY: Slower? NH: Depends. Doesn’t do it for everything, just the larger arrays. RF: If you just set heap-arrays, all on heap. Can control it. MW: In MOM6 explicit places we declare variables we know we won’t use, contingent on assumption they are stack vars. Can’t make those assumptions any longer.
NH: Surprised to hear linux kernel. Would think it was Fortran runtime or compiler. MW: runtime or libc. Couldn’t figure out why different results with same compiler on different platforms. NH: Calculating variables addresses, compiler computes stack offsets. Looking at the executable there are static offsets. Needs to be done at compile time. MW: Shouldn’t be running models that need to use heap. Should be resilient to either choice. No? NH: Comes down to algorithms used to manage memory. Heap has algorithm to minimise fragmentation. Don’t have an answer, will need to think about it.
MW: Can you send a bug report for SIS2? RF: Could be everywhere that has run out of stack space. Just the first one I tried to fix this.
AH: What OS are you running on your laptop? MW: Archlinux. Comparing them to the travis VMs. AH: At some point the compiler has to query the system to see what resources are available? MW: The fact that you’re typing stacksize unlimited shows you accessing the kernel. AH: Seems strange, system has plenty of memory. MW: I’m interested in this problem. AH: Problem should be reported to relevant NCI people (Dale/Ben?). Potentially affecting a lot of codes. Not tenable that everyone who has this issue have to debug it themselves. MW: Bad memory explicit in stack, buried in the heap? NH: Can make a huge difference. Layout of memory is different. More likely something on HEAP won’t affect other variables. More fragmented on stack. Heap memory more tightly packed. MW: Fixed a couple of dozen memory access bugs in MOM6 and they take it seriously. RF: Old versions I’m using with CM4 release. Happens with MOM5. Only FMS common. MW: Wondering if this is a bug that is hidden moving from stack to heap.
MW: Using GCC9.0 to find these. Few flags to find stuff. Initialise with NaNs. malloc-perturb is an environment variables you can turn on and that helps. Turns on signal NaNs. Any FP op generates an error now. Finds a lot of zeroes in bad memory accesses that didn’t trigger errors. Trying to not use valgrind, but that would work also.
RF: Switch in GCC that does something similar to valgrind. Puts in guards around arrays. MW: Don’t know the explicit option, using -Wall, turns it on for me. GCC9.0 is very aggressive at finding issues in a way that 5/6/7 were not.
AH: Same compiler on raijin and gadi, see if gadi only issue. RF: Not sure if it was the same version of 2019 I was using. AG: One overlapping compiler 2019.3. RF: Recently recompiled MOM-SIS build. Will look and see if it is the same. AH: Useful data point if same issue is gadi specific.

Update on BGC

AH: Andy Hogg has asked for an update. People at Melbourne would like to us eit. RF: On my desk with Hakase. Been promising. Will prioritise. Almost there for a while. Been distracted with gadi. On to-do list.
MC: Do we know who in Melbourne wants to use it? AH: A student, not sure who.

New projects to support COSIMA and ACCESS-OM2 on gadi

AH: /g/data/ik11 is where inputs that were on /short/public will now live. Not sure exactly how this will be organised. Will mostly likely have input and output directories. Might be some pre-published COSIMA datasets there. Part of a publishing pipeline. AK: Moving data from scratch to this as a holding area? AH: People were using datasets from hh5 that had no status, not sure how to reference them.
AK: Control directories are separate, and not well connected to the data on hh5. Nice to have ways to link things more firmly. AH: To-do for payu is have experiment tracking IDs. Generate UUIDs as unique identifiers for experiments. Will go in metadata file. Not linked to git hash. If they don’t exist, make new ones. AK: Have data on hh5 and the control directories have been moved or deleted. Lose the git history of the runs that were used to generate the output. AH: Nothing to stop that all being in the same directory. Nic has advocated this for some time. Could change the way we do things. AK: Not sure on solution, but flagging as an issue.
AH: Published dataset from the COSIMA paper is almost ready. New location for COSIMA published data will be cj50. To do this publishing have created a python/xarray tool to create published dataset from raw model data. Splits data into separate files for each variable, a year per file in most cases. Needs a specific naming convention for THREDDS publishing. Using xarray  it doesn’t matter what the temporal range of each model output file. Uses pandas style resampling to generate outputs. In theory simple, in practice there are many many exceptions and specific tweaks to be standards compliant. Same tool can handle MOM and CICE outputs, which are different models, and radically different file metadata and layout. If you have something that you might find it useful for it is called splitvar. Also made a tool called addmeta for adding metadata. Do the metadata modification as a separate step as it is always fiddly. Uses yaml formatted files to define metadata. The metadata for the COSIMA data publishing is available.
PL: Published data is netCDF format with all the correct metadata? AH: MOM doesn’t put much metadata in the files. To make this better connection between runs and outputs is to insert the experiment tracking id mentioned above into the files. Would be nice to put that into a namelist so that MOM could put it in the file. Best option, and if anyone knows how would like to know. Another option is a post-processing step, on all the tiled outputs. MOM isn’t the only model we run. Not all output netCDF. Would be nice if there was a consistent way for payu to do this. COSIMA published data should be up before the end of the year.
PL: Will ik11 replace hh5 and v45. AH: hh5 is storage space that is part of a ARC LIEF grant from the Australian climate community. The COE CMS team was tasked with managing this, and people could ask for temporary storage allocations. In practice it is harder to get people to remove their data. COSIMA was one of the first to ask for an allocation, but it somewhat outgrown the original intent of hh5, as it has been there for a long time and grown quite large. hh5 might still be used for some models outputs. Not sure. ik11 started because we needed somewhere to put common model inputs/exes because /short/public went away and /scratch/public is ephemeral. /scratch space is difficult to utilise because of the ephemeral nature. NH: Have some experienced /scratch space on Pawsey. Once you lose data you make sure you have a better system to make sure your data is backed up. Possibly a good thing. AH: Doesn’t suit the workflow people currently use, where they come back and run some more of a model after a break. Suits workflows that create large amounts of data and then do a massive reduction and only save the reduced dataset. Maybe suits ensemble guys. Our models everything we create we want to keep. NH: Doesn’t all the model output go to scratch. AH: Yes, but model output doesn’t get reduced, so end up having to mirror the data.

Technical Working Group Meeting, September 2019


Date: 11th September, 2019

  • Aidan Heerdegen (AH) CLEX ANU,  Andrew Kiss (AK)  COSIMA ANU
  • Russ Fiedler (RF) CSIRO Hobart
  • Rui Yang (RY) NCI
  • Nic Hannah (NH), Double Precision


AK: JRA55 v1.4 splits runoff into liquid and solid. Most elegant way to support? Have a flag in accessom2 namelist to enable combining these runoffs. NH: Is it a problem in terms of physics? Have to melt it? AK: Had previously ignored this anyway, so ok to continue. NH: Backward compatibility!
AK: Some interest in multiplicative scaling and additive perturbations to allow for model perturbation runs. NH: Look at existing code. Might not be too hard. AK: Test framework for libaccessom2? NH: When did scaling did longer to write test than make the code change. All there, could use as an example. Worth to run tests, don’t want to get it wrong. AK: Not familiar with pytest. NH: In this case just copying scaling test, modify, and get pytest to run just that test. Once got just  that test running and passing you’re done.
AK: New JRA55 now in Input4MIPS. Used JRA v1.3 from that directory and didn’t reproduce. AH: Correct. Didn’t work out why it wasn’t reproducing. AK: Ingesting the wrong files? Should be identical. AH: Never figured out what was wrong. Didn’t match checksums from historical runs. Next step was to regenerate those checksums to make sure the historical ones were correct. Could have been ok, but didn’t get that far.
AH: JRA55-do is now on the automatic download list, should be kept up to date by NCI. If it isn’t let us know.
NH: Liquid and frozen runoff backwards compat, but what about future? AK: Some desire to perturb solid and liquid separately, and/or distribute solid runoff. NH: Can we just put it somewhere and allow model to deal with it. AK: In terms of distributing it, not sure. Some people are waiting on this for CMIP6 OMIP run. Leave open for the future. NH: MOM5 doesn’t have icebergs? AK: No. Depoorter et al. has written a paper for meltwater distribution. Maybe use a map to distribute. RF: What they use for ACCESS-CM2. Read in from a file.
AK: Naming convention for JRA55 v1.4 has year+1 fields. Put in a PR some time ago. AH: Problem with operator in token? NH: Should be fine as long as within quotes. AK: Just a string search shouldn’t make a difference.
AK: Can’t get libaccessom2 to compile and link to correct netcdf library. Ben Menadue tried and worked ok for him. Problem with findnetCDF plugin for CMake. Not properly supported on NCI. Edited the CMake file to remove this, could find netCDF, but used different versions for include than linking. Should move to a newer version of netCDF. v4.7.1 has just been released. Have requested this be installed on NCI NH: Does supported include CMake infrastructure around library? If getting findnetCDF working was NCI responsibility that would be great. Difficult getting system library stuff working properly with CMake. CMake isn’t well supported in HPC environments. AK: Ben suggested adding logic to check and not use on NCI. NH: Definitely upgrade, to 4,7 if they install it.
AH: Didn’t Ben Menadue login as AK and it ran ok? AK: No, he didn’t do that as far as I know. AH: Definitely check there is nothing in .bashrc. Also worth checking if there is a csh login file that is sourced by the the csh build scripts.

OpenMPI testing

RY: OpenMPI 2,3,4 and Intel 2019. Consistent results between for all OpenMPI versions. 1, 0.25 and 0.1. Some differences between Intel 2017, not from MPI library. Not sure if difference is acceptable or not? Would like some help to check differences.
Just looking at access-om2.out differences. Maybe need to look at output file like RF: Need to compile with strict floating point precision to get repro results. MOM is pretty good. Don’t know about CICE. Can’t use standard compilation options. fp-precise at a minimum.
RY: If this difference is not acceptable need to use flags to check difference between 2017 and 2019? RF: Once get a bit change, chaos and get divergence. RY: Intel 2017 still on new system. AH: So not only newest versions of modules on gadi? RY: 2017 will be there, but no system software built with it. AH: Done a lot of testing. Should be possible to just use 1 degree as a test to get 2017 and 2019 to agree. There are repro build targets in some of those build files. Could try and find them. RY: Yes please.
AK: Any difference in performance? RY: No big difference. NH: New machine? RY: No, old machine, with broadwell.
RY: NCI recently sent out gadi update and blog and webpage. 48 cores/node. NH: Did we think it was 64 cores/node? AH: Still 150K cores in gadi, with 30K of broadwell+skylake. Maybe have to change some decompositions. RY: Not the same as any existing processors.
AH: Two week overlap with gadi, then short will be read only on gadi. RF: There was panic in ACCESS due to an email that said short would disappear in mid October. AH: Easy to misread those dates.

accessom2 release strategy

AK: Harmonising accessom2 configurations. Somewhat haphazard release strategy, but not tested. Maybe master branch that is known good, and have a dev branch people can try if they want? Any thoughts?
NH: Good way is really time consuming and labor intensive. Would mean testing every new configuration. Not sure if we can do that. Tried to keep master of parent repo only references master of all the control experiments. Not sure if necessary or desirable? Maybe makes more sense to develop freely on own experiment and keep everything in control stable? Not sure. If all control experiments are stable and working, can be a bit slow to update. Just update your experiment.
AK: Some people are cloning directly from experiment repos, some cloning all of access-om2. Would reduce confusion if control directories under accessom2 are kept up to date with latest known good version. NH: Does make sense I guess. Shame for people to clone something that is broken which has already been fixed. There is some python code in utils directory which can update everything. Builds everything at all resolutions, copies to public space, updates all exes in config.yaml and does something with input directories. AK: I ended up writing up something like that myself.
AH: Should split out control dirs from access-om2 repo. Is a support burden to keep them synched. Not all users need entire repository, as using precompiled binaries. Tends to confuse people. NH: Did need a way for config to reference source code and vice versa. AH: Required to “publish” code? Maybe worth looking into. NH: Ideally from the experiment directories need to know what code you’re using. Probably got that covered. In config.yaml do reference the code and it’s in the executable as well. When run executable it prints out the hash from the source code. Enough to link them?
AH: I recall NH wanted to flip it around and have the source code part of the experiment. NH: Probably too confusing for users. AH: True, but a useful idea to help refine a goal and best way to achieve it.
AH: A dev branch is a good idea. Then you have the idea that this is the version that will replace the current master. Can then possibly entrain others into the testing. Users who want updates can test stuff, you can make a PR and detail testing that has been done.
NH: Good idea. Some documentation that says experiments have stable and dev. When people are aware and have a problem, wonder if they can go to dev, see if it fixes. AK: Bug fixes should go into master ASAP. Feature development is not so urgent. A bit gray, as sometimes people need a feature but they can work off dev. AH: Now have some process for this: hot fixes that go straight in. Other branches are dev/feature branches. Maybe always accumulate changes into dev. Any organisation helps.
NH: Re: Removing experiment repositories: namelists depend on source code. AK: Covered by executables defined in config.yaml. NH: Yes ok.


RF: Did it work? It’s got a lot of merges. RF: Just two lines. Did a merge and pushed it to my branches on GitHub. AH: I’ll merge it in. Just wanted to check. AH: Can always make a new master branch that tracks the origin, check that out and pull in code from other branches. RF: Have a lot of other branches. AH: Can get very confusing.

payu restart issue

AH: Issue has resurfaced. I commented on #193, but didn’t look into the source of the problem. Should look into it rather than talk about it here.

FMS subrepo

AH: Still not done the testing on this. Been sick. Will try and get back to it.

Tenth update

AK: Andy done 50 years with RYF 90/91. Running stably. AH: What tilmestep? RF: Think he was using 600s. AK: 3 months / submit. Should ask for longer wall time limit. RF: Depends on how queues will be on new machine, what limits and what performance. AH: Talking about high temporal res output. AK: Putting out 3D daily prognostic fields. Want it for particle tracking. Including vertical velocity. Slowed it down a little bit. RF: More slowdown through ice. AK: No daily outputs from CICE.


NH: Still in progress. AK: Also requires newer version of netCDF? NH: Requires specific version of netCDF. Needs parallel version. Not a parallel build for every version. AK: Has parallel for 4.6.1. RF: Bug in HDF5 library which it is linked to. Documented in PIO. Probably a bug we’re not going to trip. Doing a collective write, and some of the processors not taking part/writing no data. Fixed next version of HDF5 1.10.4? AH: Not a netCDF version so much as the HDF library it links to. RF: Yes. AH: So should make sure we ask for a version of netCDF that doesn’t have this bug? AK: Add to request.
RY: If want parallel version, use OpenMPI 3 or 4? AH: Good question! RY: All dependencies will be available and very easy to use. AH: This using spack? RY: Above spack and other stuff. Automatic builds with all possible combinations. AH: Using it for your builds? RY: We are requested to test and are now using. Difficult to create new versions currently. In transition difficult, but in new system should be fixed quite easily. AH: Should fix the various versions of OpenMPI with different compilers. RY: Yes. AH: Will have a compiler/OpenMPI toolchain? RY: Will automatically use correct MPI and compiler. AH: Any documentation? RY: Some preliminary, but not released. When gadi is up all this should be available.
AK: Should I ask for a specific version of MPI? RY: If don’t specify, will be built with 3 or 4. Do you gave a preference? AK: No, just want the version with performance and stability we need. Do we need to use the same MPI version across all components. RY: Not necessarily. Good time to try OpenMPI3. No performance benefit as system hardware is still old hardware.

Technical Working Group Meeting, August 2019


Date: 14th August, 2019

  • Aidan Heerdegen (AH) CLEX ANU, Angus Gibson (AG) RSES ANU, Andrew Kiss (AK)  COSIMA ANU
  • Russ Fiedler (RF), Matt Chamberlain (MC) CSIRO Hobart
  • Rui Yang (RY) NCI
  • Marshall Ward (MW) GFDL
  • Nich Hannah (NH), Double Precision
  • James Munroe (JM), COSIMA

PIO work with CICE

NH: PIO code in CICE not as complete or thorough as netCDF code. Nothing to suggest it won’t work. Relies on NCAR PIO library, and a CESM utility library. Dependencies which are not part of CICE. Built PIO dependency on raijin, ran into CESM dependency. Can either remove dependency or remove code.
NH: Initially thought to use the MOM approach. Tile and collate. Russ’ comments encouraged to try PIO. Will be supported in future and will be supported in CICE6. Nothing working, but will soon test with 1 degree.
RF: Real bottleneck with high freq output. Worth a go. Attempt to put this into FMS by Hartnett. AH: Different to parallel netCDF? NH: PIO is wrapper around parallel netcdf. Written by NCAR to simplify parallel netcdf. Another layer. On GitHub, continuing to be maintained. RY: Wrapper that does work to match computing to IO domain. Not so useful for MOM5 as it has io_layout already.
MW: Harntnett motivated by FE3 (forecast model) rather than ocean. Not sure what project even involved in.
NH: Big test is handling interesting CICE layout, difference between cartesian grid and PE layout. MW: PIO will support explicit decomposition and other approaches.
NH: Parallel netCDF version on raijin only links with OpenMPI3.0. RY: New machine launched soon. OpenMPI 1.* will be dropped. No new software depending on 1. MW: OpenMPI 2 is not good. Should use 3.
NH: Probably have to test this with OpenMPI 3.0 RY: 3.1.3. Switch everything to that. Good test for new machine. AH: Working now? RY: My fault. Used unmatched openMPI library. Everything looks fine. OpenMPI 2/3/4 with Intel 19. All working. 1 deg & 0.25 deg working. Tenth not working. MW: I was able to run tenth with 3.1.2/3.1.3.
MW: One of the intel compilers broke MOM. A compiler bug with types in types.
AH: Should  start an issue for testing RY: Will email MW directly. RY: Not a MOM bug.
MW: Tried MOM-SIS tenth? Good test. RY: From earlier this year do have this working. This is testing for new machine, so ACCESS-OM2.

OMIP date restart protocol

RF: Talked to Griffies. GFDL take ensemble approach. Run for N years using true dates. At finish reset back to start date with correct calendar. Storing new stuff in different directory. End up with 5 sequences of 55 years. All dates are correct. No issues with leap years going wrong. Think this is the best way to go.
AK: Came to conclusion that this was right way to go, mostly due to leap year issue. Problem is, can we get the model to do that, but Maurice and Ryan had issues. Issue with CICE getting the correct date. CICE has a flag “use_restart_dates”. Suggested set this to false, and set the dates in access_restart.nml, but CICE is not picking up dates. Looks like libaccessom2 is not passing them on to CICE. Some confusion about exactly what they have done. Some instructions on Wiki for restarting, from restarting IAF from RYF at tenth, but doesn’t work for other people. NH: I’ll look at it. AK: Will send issue. NH: Didn’t realise it was happening. CICE date handling is not great.
AH: Downside with ensemble, difficult to get metrics across the whole time series. RF: Need extra meta-data added in. Maybe which cycle you’re in. An extra variable which gives the actual number of days since the start of the run. Down with post-processing. Might be able to concatenate files using extra meta-data. AH: Always have issues with missing leap years if it spans a century. But only daily is an issue. AK: Cookbook do something. MC: Pretend it is no leap? JM: Data looking at as time series? AH: Extra metadata, say offset day is a good idea. RF: Add buffer in netCDF file so don’t need copies. mppnccombine can add padding. usually done with nccreate, make sure the header has some space. hbuf?

Strategy for CICE updates for flexibly adding fields

RF: Way CICE drivers work, variables you want are either hard coded, or muck around with pre-processing to compile them in and out. Wondering if anyone looked at doing it on the fly. Using error codes coming back when setting up variables, so have flexible number of variables passed in and out. Would like this to pass total wind speed, to harmonise code. Also Hakase wants it for some BGC stuff. Phytoplankton through to the ice. So specify the variables, work out if they’re there or not.
NH: Would want the exe to handle configuration with different sets of coupling fields. Sometimes include total wind speed, sometimes not. RF: would know complete set, if not there skip it. Currently have to be hard wired in, or make another driver. NH: Way to do it, start with superset in namcouple, and code would exclude certain variables. RF: Maybe if variable not in namcouple, return an error code, but ignore error. NH: Shouldn’t be too hard to do. NH: OASIS does return error codes that could be used. Either abort or return error code. If aborting could change that. AH: Restart fields? NH: Should do behind the scenes.

Paths for JRA55-do forcing files. Some changes to support v1.4

AH: JRA55-do not part of Input4MIPs, part of CMIP6. Have to use the copy that is CMIP6. Encodes all the metadata in filename, consequently doesn’t currently work with YATM. Circumvented by creating symbolic links that worked with YATM. When I did this couldn’t reproduce. Not sure if this is actually an issue with the fields being different or not.
AH: Tried to use testing framework NH developed for this using jenkins. The historical test that tests against known checksums doesn’t seem to actually compare them. Not sure if that is intentional. Would like to use framework, as NH has done a great job with it.
MH: MOM6 has diag_mediator, supports CMOR name alongside internal model name. Porting to MOM5 is a big task, but idea is good and saved them a lot of work. Could create a thin wrapper to translate to CMOR name if that helps. AK: How integrate with YATM? MW: Don’t know. At FMS level, so only help with 1 model (MOM). AK: YATM access the JRA files. So libaccessom2 change. AH: Looked at YATM code. Generates filename form date. Input4MIPS has current year and next year, so would require code changes. Might just be easier to create a file with date->filename mapping? AH: Possible to do. Would need to add a token for year+1. Possible to do. Probably best to do it that way.
AK: Also need code changes with v1.4. Solid and liquid runoff are separate. What to do with solid runoff? Griffies either use iceberg model, or melt them and add them to runoff. Take account latent heat of fusion? Assuming solid runoff is at zero, which could be a problem. Put in a request to download v1.4. Scripts they have should automatically download it, but not. MW: Think GFDL only has v1.3.
MW: Fields go to end of 2017, is 2018 downloaded? Looking in wrong place? Looking in ua8. AK: Should look in qv56. AK: qv56 up to feb 2018. AH: If not automatically downloading, we should ask. What does the OMIP protocol say about end date? AK: JRA55 can find out about 2018. RF: It is specified, but would like latest for ongoing runs.

Testing FMS merge

AH: Putting FMS in as a sub-repo. Just needs testing. If it reproduces checksums for a month we’re sure it is ok? Is that sufficient?
NH: When Marshall upgraded FMS, went through every MOM test. Including 0.25. Can’t recall how strict we were. AH: Testing framework still there? NH: It is there. Because it never gets used, might be rotted a bit. Can give Jenkins URL of PR and it would do it. We should work together to get that working.

New NCI HPC hardware announcement

RY: System by end of the year. 2 phases, install new machine with Cascade Lake nodes. Short period gabi and raijin run simultaneously. After that skylake and broadwell will be merged with new machine and SandyBridge nodes removed. 100 GPU installed. 16 skylake k-80 nodes. PBS pro again. Storage and network infiniband. 200GB/s transfer speed. OS is CentOS 8. AH: Trying to figure out total core count for new machine. Do you know what core count will be? RY: Not clear on exact number. Can check with system guys if they know the exact number. If 32 cores/node, 150+K processors. AH: Will runtimes be extended for new machine. Find 5 hours too low for high core count jobs. Reduces flexibility. RY: Queue time limits are per project. Quite flexible. Contact NCI help. AH: Have asked for time limit changes in past, but usually time limited. RY: Have been asked by other users, not sure about the policy. Good time to ask and get a better policy for the new machine.

Technical Working Group Meeting, July 2019


Date: 1tth July, 2019

  • Aidan Heerdegen (AH) CLEX ANU, Angus Gibson (AG) RSES ANU, Andrew Kiss (AK)  COSIMA ANU
  • Russ Fiedler (RF) CSIRO Hobart
  • Rui Yang (RY) NCI
  • Peter Dobrohotoff (PD), CSIRO Aspendale
  • Marshall Ward (MW) GFDL
  • Nich Hannah (NH), Double Precision

Config checking

AH: Made payu configuration checker. Include safety checks for synching scripts in BASH scripts. Interested in checking for bad namelist options. Russ any specific bad ones.
RF: Kpp kbl standard method should be false, red sea fix used in access-cm. For a CM model maybe warn. OM model just not allowed. AK: nprocs and ncpus driver issue?
NH: diag_step checks? To frequent ok for low res, bad for high. RF: For production runs don’t want? But how you diagnose problems. Best way to find how things are going wrong. Maurice’s issue was trivial to spot. AH: Definitely say don’t want debug_this_module turned on. RF: diag_table debug turned on, should be turned off. Creates huge numbers of messages. AK: Setting up new updated configs. Ten configs. Make them more homogenised. Fixing all these things as they go. AH: These things will get changed by mistake. Don’t have enough people to keep checking things. Doesn’t scale. This will allow new users to submit a config that at least passes these checks, also gives others confidence to change config knowing they have produced something that meets some minimum standard, appropriate for public facing production.

Tenth Run

AK: Andy Hogg is running ACCESS-OM2-01 with JRA-55-do RYF90/91, seems to have smaller biases than previous repeat year (84/85). Currently 13 years. When it does die it is CICE CFL problem. Sometimes the same date, subsequent years didn’t occur. Checked dates. Storm goes near tripole. Not currently messing with forcing winds. Did this with 85/85 but this doesn’t seem as bad, so haven’t done it so far. One drawback doesn’t run at dt=600s, but takes 2.55 hours to do 3 months. With dt=600s could do 6 month submits, which would mean less queue wait. Should be straightforward to fix the winds to enable this. AH: Not a priority considering extra SU cost. AK: About 10%. Cost of losing 6 month submit halfway through > 10%. AH: Shame it is the tail wagging the dog. AK: Could ask for 6 hour limit from NCI? AH: Worth trying. Done it before, and have seen others with increased limits. Prefer to do it, but time limited, and just for one project. AK: Hopefully limits will change with new machine. Currently 65KSU/3mths. RF: MOM or CICE bound? AK: Fraction of time MOM is waiting 2-3%. RF: Not greatly MOM bound. Throw a few more processors at MOM to get it to run less than 2.5 hours?
AK: 40 years. IAF not split, just start from climatology. AH: When will IAF start? AK: No plans, not simultaneously run RYF and IAF.

NCI update

AH: Attended an NCI scheme manager meeting. Mostly about new storage scheme for short term storage. Push came from CSIRO to change to scratch model, but some others in CSIRO not happy. PD: Wasn’t aware that was being driven from this end. Maybe further up the food chain.
AH: Change to time-limited scratch, or a tidal model deleting oldest data first. Maybe a split scheme with old style short on one disk, time limited on another, but not a lot of appetite for that.
RY: First stage November. Our group look for HPC application for new machine. Already have ACCESS-OM from Andy Hogg to look into software state. New machine some old library will not be maintained.


Recently used ACCESS-OM2 with OpenMPI 3.0. Seems to hang? Know this issue? Or avoid 3.0? Some work required to run on new machine. Spend some time on this work.
AH: Marshall any ideas? MW: Have tried 3.0.0 3.0.1, maybe 3.1.1. Earlier ones didn’t work then got fixed. Newest 3.x should work. RY: Tried 3.1.3, MOM keeps hanging until finish of job. Should finish at 40min. Keeps hanging. 1.10.2 works. 3.1.3 hanging.  MW: Sure I got it running. Will make sure they are in repo. RY: Catch up with your personally? MW: where it hung should tell you. RY: Talk later offline.
AH: Definitely need it working on the new machine. MW: No work needed to be done, it just worked. AH: What changes would you have made? MW: Just versions, environment file and flags. Maybe using some of the alltoallw changes, but I don’t think that was a deal-breaker.
AH: What is the minimum version OpenMPI supported on new machine. RY: Under discussion. System guys will decide. Haven to prepare for any. Not sure OpenMPI 1.10 will still be supported. Don’t know. AH: Likely to be OpenMPI 3.x+? RY: New machine with new architecture. Performance enhancements with new architecture. MW: What arch? RY: Now have skylake. Newer than Skylake. MW: Intel architecture, not Ryzen. AVX512 can’t benefit. fma which we already have AH: AVX512 because can’t vectorise enough? MW: Currently vectorising, but bandwidth limited. Ryzen has better bandwidth. RY: Not announced. No idea. AH: At scheme managers meeting it was an Intel chip. Told it was November when they commission new nodes, take equivalent raijin nodes offline. Iron out the bugs, and early next year will turn off the rest of raijin and turn on the rest of the new machine and at that point it will be larger than raijin now, but not a huge increase in compute. AH: Thanks for bringing that up Rui, as we definitely need to keep an eye on this for the new machine.
MW: Apparently used 3.0.3. Maybe a reference point to start with. RY: Start with 3.0.3? MW: Whole space is volatile, some 3.0.* series work some don’t. But start with 3.0.3 and Intel19.
AH: Would be nice to have a spack like build tool so can say for certain what was run. MW: payu build! AH: spack written by a smart guy from TACC, and lots people use it, and they still have a lot of issues. Not an easy problem to solve. MW: Dale was keen on it. AH: When we met with Dale he was thinking to have spack as a tool preconfigured with compiler toolchains that we can build our tools from. RY: Dale is very busy getting new for the new machine.

Splitting off FMS

AH: Been working on Cmake to compile FMS separately from MOM. Been using the FMS fork in mom-ocean repo with your alltoallw changes. MW: Also a branch on the GFDL repo with those changes.
AH: How to organise the FMS fork? Have a branch that tracks GFDL and master contains our local changes? Could have a branch called gfdlmaster, could have our master branch exactly track the GFDL FMS. Any opinions on how to organise this? MW: Don’t want to use GFDL FMS? AH: I want an easy way to update FMS without touching MOM source tree. MW: Want to get FMS out of MOM? AH: Yes. MW: And want to know how to refer to FMS you want to use? AH: FMS we want to use is a fork on mom-ocean. Gives flexibility to add changes when we need to.
MW: Best to have your own FMS fork. GFDL don’t want to support anything but for GFDL, including MOM5. Don’t really want to get involved in supporting other projects. Will be receptive. No harm in using FMS repo straight, but if doing anything with FMS better off maintaining own version and update as see fit. Don’t see compatibility with older models as priority. Planning a big IO rewrite. Wouldn’t be surprised if it starts breaking and not salvageable.
AH: alltoallw we definitely want on our architecture as we’ve had issues in the past? MW: A lot of work, return not what I’d hope. Latest MPI version bigger impact. Are cases with speed up, but such an infrequent operation not such a big deal. AH: Stopped initialisation hangs? MW: Yes, some rare scenarios where they did alltoall with point to points that broke a lot. In OpenMPI 2.0/3.0 and later they changed something, scenario no longer happened. Segfaulted before, now properly checking. Only necessary for 1.10. It is better, as collectives are generally more responsible. May become necessary, assuming 3.0.3 works.
AH: If want alltoallw, would keep a branch with those changes and rebase on to gfdl master. This would be a well documented branch, or branches, and a well documented way of applying those changes when an update is required.
MW: Can CMake build as libfms and link to MOM when you build it. No submodules, rely on Cmake. Does that work? AH: FMS is not suitable to be a loadable module. Get OpenMPI conflicts, best to build at the same time with the same compiler toolchain. There is a new Cmake tool called FetchContent that can grab a repository and it behaves like it is physically in the source tree. Works well, but not great versioning. MW: Isn’t Nic already doing something like this for ACCESS-OM2 to pull in specific versions of son-fortran. AH: Yes, you can specify a library git hash. The only thing stopping it from working is relocating the versioning string stuff Nic did as it is currently sitting in the FMS directory, and that is going to disappear. Needs it’s own directory, maybe ocean_shared? RF: ocean_shared is used for other tracers. MW: should not use that name. AH: Ok, will make a new directory called version. Can recreate the sed script functionality that is currently in the build script in Cmake using template files. Quite a clean solution. I have a cmake branch on the MOM5 repo and FMS fork on mom-ocean, will get them compiling properly and working properly together. There is a way forward.
MW: Alistair is pretty interested, might be a template for MOM6. AH: Angus already did this for MOM6? MW: Angus, is what you did still viable? AG: Haven’t tried recently, don’t know why it wouldn’t work. Replicating mkmf process in CMake. MW: Automake is not good and won’t touch it. AH: Surprised there was no way to build FMS from the FMS repo. Relies on being imported into another project that knows how to build it. Not sure it is great that a project can’t build itself. MW: CMake support not widespread enough? Not available everywhere? AG: Updates frequently, can have features that break old versions. Used in a lot of projects. Surprised if it went away. AH: Cmake can be brilliant, but also terrible, but better than mkmf. MW: mkfmf is doing two jobs, importing stuff and working out dependencies. Does work well for the latter job. Set a high bar. AH: Haven’t done proper comparisons, but Cmake seems to better for dependencies. Can do parallel builds with Cmake you can’t with mkmf. MW: mkmf just generates a makefile, which is already parallel. AH: So does cmake AG: doesn’t seem like a good makefile, don’t know if the dependency tree is deficient. Rebuilds too much even after touching a single file. MW: if CMake intelligently supports mod files then it is fantastic. AG: Has native fortran support. AH: Speed point of view, Cmake is better. Generated correct dependencies so that parallel compilation worked. Couldn’t do that with mkmf. Also had compilation cascade issues. MW: I build 5 exes at once, so it always looks fast to me. AG: MW same makefile gen as mkmf. MW: More readable makefile than automake? AG: Yes. More readable than automake. AH: When the magic works Cmake is great, when it doesn’t it is a pain, but the magic is worth it. Also supports multiple architectures.


RF: Aidan can you approve change to FAFMIP. Starting to get conflicts. Ryans changes put it all in conflict. Riccardo has disappeared, but Fabio’s changes so it is all the same bit for bit. AH: Current conflict in ocean_frazil. RF: Because you put Ryan’s changes in. AH: Sorry. Could rebase on Ryan’s changes. Maybe pull in Ryan’s changes. AG: Could check out the branch, make changes and push to the branch. AH: I’ll try doing it directly on GitHub, get back to you about it. RF: Get that done and I can finish up some of the WOMBAT stuff. With the ESM model I also have to make some changes to CICE. A couple of design things with the number of fields that are passed. Hard wired at the moment. A couple of issues there. Have a chat at a later stage. Rather than hard wire fields, flexibility, test error codes, make compatible with namcouple, so can be done on the fly. Also feed into BGC Hakase is putting into CICE. Need to pass BGC fields between the two modules. Rather than having a plethora of drivers, or CPP directives, better ways to do it.
AH: Made that change on GitHub and merged it. Once checks are finished will accept the PR.
MW: Been working on a test with MOM6, where we turn of every diagnostic, fantastic for finding bugs. Found nearly 2 dozen bugs. Don’t actually register the diagnostics with FMS, just spoof the whole thing at the diag_mediator level, which is a wrapper around the diag manager. Interesting if this could be translated to MOM5. Don’t know a natural way to do it, but might be worth some thought at some point. RF: Code you’re putting into MOM6, not the diagnostic manager? MW: Yes. FMS moves too slow, very conservative, don’t have a robust test framework so are worried about putting in changes. There are some hints that maybe this code could be shared with MOM5. Lots more in there than just this. Just raising it as food for thought. AK: Put as an issue? MW: Opposed to those sorts of issues, but you can if you want.
AK: Want to set up new vanilla reference versions of the 1 and 0.25 deg ACCESS-OM2 models. The forcing on those use 2nd order conservative interpolation. There are overshoots for some fields which have to be positive definite. Would like 1st order conservative for some fields. Do they exist? NH: They should be there, we were using 1st order for a long time, and should be in the input directory. Not sure how well they are named. Should say in the filename, have a look and if you can’t find them we can recreate them.

Technical Working Group Meeting, June 2019


Date: 19th June, 2019

  • Aidan Heerdegen (AH) CLEX, Andrew Kiss (AK)  COSIMA, ANU
  • Russ Fiedler (RF), Matt Chamberlain(MC) CSIRO Hobart
  • Rui Yang (RY) NCI
  • Peter Dobrohotoff (PD), CSIRO Aspendale
  • James Monroe (JM) Memorial University


RF: FAFMIP into MOM. Riccardo will do his tests. Don’t expect issues. AH: Did Fabio notice problems? RF: Started with ice formation used by ACCESS wasn’t coded up. Did that and then noticed way things were being done didn’t match with what was in the literature. Mismatch between what Griffies did and Riccardo wrote. Now at a stage where that is now consistent. Talking with Trevor McDougall about equation of state. Coded in MOM not totally consistent with what protocol says should be done. All groups do it a little differently. How badly can we violate the freezing condition and still get reasonable results. If you do this incorrectly can fall below freezing and not form frazil. Behaves ok down to -3 degrees. Hopefully won’t get that far. Are other approaches, have to have a think about that. Will stick with what is done currently. AH: Modifications? RF: Look at other mods to see if we can do it more consistently. AK: More consistently without additional tracer? RF: Still need additional tracer, but more consistent, temp and redistributed heat tracers see the same values of frazil. The way Griffies et al constructed get slightly different values. Not completely clean. Can’t get runwaway with one of the tracers. Safe but not right way. Other ways: fix problem with implicit diffusion. Code as it stands is at least consistent with what has been written up. MC: None of these full TEOS-10? RF: Yes TEOS-10. Also had to fix the conversions to potential temperature. MC: Dealing with salinity etc? RF: Simplified version. Need these changes to do FAFMIP correctly.
AH: Any other ramifications? RF: None. All changes only take place in this style of experiment. Everything separate from other experiments. Only issue was prognostic versus pot temp.
AH: Merged independent of the WOMBAT stuff? RF: No. WOMBAT stuff relies on changes on ocean_sbc. Have to rebase. Get FAFMIP in first.


 RF: Haven’t had a chance to sit with Matear and test it properly. Just a few changes needed from current code. Hopefully pin down Matear. AH: Hakase with WOMBAT  in tenth? RF: Yes. Hakase will test. Currently inputting winds via a file rather than in through coupler. MC: Richard Matear is working directly with Hakase.
RF: Few lines in the coupler that I have to add and a namelist item. In namcouple file need to pass 10m winds. It is in CM2 code, but not in OM2. AH: Can Hakase work with ice BGC stuff in his current setup? Is this slowing him down? RF: No idea.
AH: Few weeks? RF: Have to rebase WOMBAT stuff.

CICE Mushy ice

RF: Code suddenly got changed and altered and no-one knew why? AH: Nick been keeping our codebase up to CICE6. RF: He made other changes that caused problems. That code also moved to CICE5 svn repository. AH: Backporting to CICE5? A lot of assumed logic in those code changes. RF: Have to familiar with POP code makes salinity changes. Doesn’t go through the surface like MOM. The clause where


“this is done elsewhere”, not true for all models. Nowhere in the code those salt fluxes are being calculated. AK: Proof in runs, results show drift. RF: Looking at it, needs that if clause removed for coupling to MOM. AH: We’re not part of any CICE6 test suite so they can’t spot errors. AK: Elizabeth Hunke said consortium was open, anyone can join. Have a comprehensive testing regimen. Get more involved so they test our use cases? AH: Definitely need more oversight on code changes into CICE. JM: Any testing when code changes added to CICE5? AH: Not currently no. Nic has some scheduled Jenkins tests but not sure on the status of those.

AK: Hit problem as using mushy ice. Wouldn’t see it otherwise. Using to overcome bug in other scheme, but don’t really want to use it. Slow, don’t need. AH: Can we fix it? AK: Iterative solver fails in high res case. Happens in fresh water regions with low ice concentration. Had intended to dig down more. AH: Would struggle to find this bug anyway as we wouldn’t routinely test tenth.
AH: Fixed now. AK: Not sure about any other problems with changing parameter setting. Took a lot of digging. AH: don’t want science changes without reason

Ob runoff

AK: Not sure how important this is. Shows how runoff code can fail. Cut away a lot of the Ob estuary due to small grid cells causing instabilities. Runoff is done on the fly. Find all runoff that is on land, move to  nearest coastline. Then check for high runoff and spread out if over threshold. Some runoff goes to embayment to the west. Changes to the Ob means that is the nearest bit of ocean. GitHub issue
Not sure how important it is. Similar issue with spreading out. Uses kdtree to find neighbouring points. Doesn’t account if there is land between those points. JM: Can tunnel. AK: What could be done to make land impassable.
JM: Resolution on that discussion?
AK: Not sure high enough priority to spend time on. AH: Use connectivity? Like used to find isolated water bodies. Move land runoff to nearest connected wet cell. AK: Depends on runoff being ocean in the first place? AH: Yes. RF: If can get to right place and just smear it out and use neighbouring ocean points. AH: Is all JRA55 runoff currently on a wet cell on the JRA55 grid? AK: Don’t know if it is a wet cell, it is on the coast. AH: Need to look into that.
AH: How important? AK: Not paying close attention Arctic. Correct volume of fresh water, just in slightly wrong location. Already severe liberties at that location.. Points to failure mode of this method. Can cross land.

Splitting FMS and other components

AH: You want to talk about other components as well Russ?
RF: If we start doing things like that to MOM repo. Will that affect anyone else who already has stuff from there? Cause problems if they want to update if we move to different setup?
AK:  Proposal to put FMS codebase into different repo? AH: Yes. AH: Can’t compile without pulling from another repo. RF: Not sure how it would all work. Use submodules? JM: In submodule right now? RF: Not for MOM5.
AH: I proposed to use CMake to create an alternate way of compiling to pull in those libraries from external repos. Could keep the FMS directory in the repo, but at some point the MOM5 code may use features in an updated FMS that are incompatible. However, they can always pull from a previous commit. Could tag a commit as the last one that had FMS included. Marshall did update FMS in the past. Desirable to go this way, to have a tighter coupling with changes in FMS, put in pull requests to main repo for features we want.
AH: Got CMake working for half the builds. Super simple to swap out external library, already compile it separately. Will finish this so people can test as proof of concept.

Langmuir KPP

AK: Progress with ACCESS-CM2. Turned on langmuir param for kpp and improved Antarctic intermediate water. Should we turn it on for OM2? RF: Our coupled runs got improvement in southern ocean. Getting shallow summer mixed layers. Helped deepen them a little bit. Different types of simulations, but work in the right direction.
RF: Not sure if that is an issue mixed layers in southern ocean over summer? If shallow, could be good. AH: Turn on/off or parameter? RF: Just turn on/off. Pretty sure I changed ACCESSOM2 to get wind coming through. Might need change in namcouple. AK: Need wind velocity as well as stress through compiler? RF: Two ways. Both have been enabled. Standard to pass 10m winds as well as stresses. Other way, if don’t pass winds, flag in kpp scheme can derive 10m winds. MOM6 does it that way. Pass through stress and calculated 10m. AH: Would still work without passing winds? RF: If forcing model with stresses and don’t have winds, this is an alternate way. Not being used currently as most models can pass wind.
AK: Might be a good time to compare OM2 and CM2. Perhaps there are beneficial changes from one or the other? Might just be model specific changes?  AH: How would this happen? AK: Maybe a meeting. Sent an email to Dave and Peter. Look at the namelists and input files.

Other updates

PD: Not up too much. Interested in getting models aligned and best outcomes for both. Maybe have a small VC and discuss. A fairly complicated set of outputs, suites etc. Can be difficult navigating this structure. Definitely encourage talking about it.
AH: What is the status of your runs? PD: PI control is up to year 950. A lot of that is pinup. Historical forked around yr 900, and a 4x historical. This is CM2. No carbon cycle. Two submissions, ACCESS-ESM-1.5. Old atmosphere, cice, updated MOM. ACCESS-CM2 is much newer atmosphere, full aerosol scheme, 5-6x slower, but no carbon cycle. ESM is a lot further along. CM2 is  not as advanced. Took some time to reach equilibrium. AH: Happy with results? PD: Yeah, seems pretty  good. Climate sensitivity seems about right. Sensitivity is a lot higher for CMIP6 than CMIP5.
JM: Will attend meetings going forward. To complement some stuff Angus is doing on the cookbook.

Technical Working Group Meeting, May 2019


Date: 15th May, 2019

  • Aidan Heerdegen (AH) CLEX, Andrew Kiss (AK)  COSIMA, ANU
  • Marshall Ward (MW) GFDL
  • Russ Fiedler (RF), Matt Chamberlain(MC) CSIRO Hobart
  • Nic Hannah  (NH) Double Precision
  • Rui Yang (RY) NCI


– Follow up on migrating FMS to an external library
– WOMBAT in harmonised MOM update and testing
– Tenth load balancing
– CICE IO bound in high core counts

CICE IO bound in high core counts

AK: Runs with new CICE executables NH compiled a while ago. Performance slowdown with compression level 5. Tested with level1 few % larger in size, 2500s -> 1800s for IO time. 1300s without compression. Compresses well with low value because a lot of missing data with ice.
NH: Went from netCDF3 to netCDF4. Might be worth trying no compression. AK: have a run with compression level zero. RF: Does impact on walltime. MOM is waiting. Usually have CICE waiting on MOM, but when outputting is the other way. MW: Compressing MOM before, now both? NH: Compressing and daily output an issue. AH: What is the chunking? RF: Uses default. AH: Some libraries chose weird values for time value? RF: No funny business, all sensible. RF: All these point to point gather, maybe not efficient. MW: Do you know where the time taken is? RF: Slowdown, but not sure split between gather and write. NH: Breaking new ground, daily output and running at scale, and unusual tile distribution. Increases the COMMS to gather. So many different new things. MW: On sect robin still? AH: 10% of total runtime.
NH: With MOM do all this with post-processing to get performance of model best as possible. Anything we do slowing model as whole, should post-process. Didn’t think about that option when put change in. If slowing down as a whole, back out change and work out post-processing step. AK: Half the data in daily files is static. Totally unnecessary. Made issue to maybe output static data to a file once. RF: Aggregate daily files to monthly? AK: Slows down output from model. Less compressible? RF: Highly correlated, will compress easily. AH: How much extra wait time? RF: The whole write time. AK: 25 or 18% in MOM runtime. AH: Monthly output issue disappears? RF: Yes. RY: CICE write to single file? RF: Yes through one processor. RY: Can we do it like MOM, each processor writes data to it’s own file. NH: Yes, good idea, but more complicated than MOM. CICE tiles are not located close to each other in space. RF: Could use PIO interface. Not compatible with centrally installed netCDF libraries. Bugs in version of HDF. Need OpenMPI > 1.10.4  and netCDF > 4.6.1. MW: PIO good candidate, RY can help. CICE developers looking into this? Stayed in touch with them? NH: Look at CICE6 GitHub. RF: Looked, but no active development on IO in any fundamental way.
NH: If we did decide to go that way, good opportunity to feed that back to CICE community.
MW: NCAR as a developer of PIO, keen to get it into other models. If CICE is on their radar might get some feedback there. RY: MOM has IO layer a bit like PIO. MW: Not a good idea to use PIO in MOM6.
RY: Tried PIO in MOM and found it was not a good candidate. MW: Yeah, MOM6 was already doing something like that.
RY: Parallel compression will be supported in future in netCDF.
RY: Been experimenting with my own version of library and got some positive results.
End result: take compression out, take out static fields. Post processing. Is anyone using daily fields. RF: We’re interested in daily ice fields. Using data assimilation. MW: Shorter runs though? RF: 20 years.
NH: Instead of writing individual daily files, should write to a single file, static fields won’t be replicated, maybe benefit from some netCDF buffering. AH: Big code change? NH: Not sure. AK: Has a file naming convention for different frequencies. Frequency part of filename. NH: Saying could already output daily into monthly files? AK: No, filename encodes time and frequency. Doesn’t seem to write repeatedly to any of it ’s output files. AH: Define unlimited dimension.
NH: Make a GitHub issue. If high priority could get some time. MW: Make the issue in the CICE repo, inform them what we’re doing. They mentioned an NCAR community board.
AH: Make a namelist option and recompile? Compression level as option?

Tenth load balancing

AK: RF suggested a smaller core count of 799. Doesn’t change wall time which is a win. How low can we go? RF: Worked out a few more configs. Slight change of tile size, 720 would be ok. 36×36 or 40×30.. Running some quick tests with tool under /short/v45/masking. Run and output masks and where tiles get located. Also number of processors/blocks you need. AH: Put code on COSIMA GitHub? RF: Just a quick little thing. AH:  Yes but useful.
AH: Down from 1380. Big win. Total core count? AK: not sure. RF: Total just over 5000. AH: Still running on normalbw? AK: Yes. AH: Wait on normal crazy. RF: Look at skylake? Usually empty. RY: Yes new nodes, not large total core count. AK: Get 6mo/submit without daily outputs. Daily over by 30/45mins with ice. dt=600s.
NH: If no-one else to fix, and no-one else to fix, assign NH to issue.


RF: Got Matear up to speed. Ran a few tests. One or two bugs yet to be fixed. A couple of fields that weren’t coming through from OASIS properly. Was the ice field, wasn’ t coming through correctly. Got it going with external fields forcing it. Figured out changes to get it running properly with full ACCESS mode. Running some tests cases after bugs fixed. MC: Now running with calculated gas exchange coefficients. RF: The way it was originally written the way fields were ingested into MOM. MC: Using the same wind field in BGC and wind mixing? RF: Yes, all together. MC: Level of the wind? In ACCESS-ESM was getting lowest atmospheric wind. MC: CICE will send a 10m wind through OASIS? RF: Not FMS coupler, this is just OASIS 10m wind. MC: ACCESS-ESM case?
AH: Hakase could be used as a guinea pig. Any of these changes affect ACCESS-CM2? RF: Shouldn’t. AH: Do we need to do any bit repro tests? RF: Shouldn’t change anything.

migrating FMS to an external library

AH: I put my hand up to do the change and test.
MW: FMS updated to Xanadu a couple of weeks ago. AH: So a good time to try it out. MW: Already tried it, put some MOM patches in to fix some issues. AH: On the GFDL FMS repo? MW: They have opted not to take the parallel netCDF using MPI IO patch RY and I worked on. Have set up a branch with parallel IO, and Xanadu has been merged into that branch. May want to use branch with parallel netCDF extensions. Ongoing conversation with this. They may merge it in. Can use what you want. Your call as to what to use.
RF: Any whitespace issues? MW: FMS and MOM6 live on different planets. They don’t interact much. Don’t collaborate with FMS guys.
MW: Alistair getting miffed at the red buttons on the jenkins server. He/I will look at some GFDL independent solution. Happy for NH to be involved as much or as a little as he wants. NH: They should be more blue than red. MW: Happened in March due to checksumming? NH: Bitrot, Jenkins is fragile. Scott often fixes it. Good idea, happy to help in any way. May be easier to set up on raijin. Does one qsub and runs them all under one sub. MW: slurm is sort of designed to do that. NH: slurm is awesome. MW: slurm is better. NH: like it a lot more. MW: Good for running multiple jobs per submission. Blurs the line between MPI and scheduler. Some sort of meta-scheduling. Place jobs on ranks within the request. AH: More flexibility.


  • Update MOM build to use external FMS library (CMake) – AH
  • Finish WOMBAT integration – RF
  • Make CICE compression issues – AK

Technical Working Group Meeting, April 2019


Date: 10th April, 2019

  • Aidan Heerdegen (AH) CLEX, Andrew Kiss (AK)  COSIMA, ANU
  • Marshall Ward (MW) GFDL
  • Russ Fiedler (RF), Matt Chamberlain(MC) CSIRO Hobart
  • Nic Hannah (Double Precision)


MW: Discovered travis test GFDL uses for FMS has been failing for six months. MW fault. Introduced new MPI function. Function doesn’t exist openMPI 1.6, which is what travis uses. Doesn’t show up in MOM5 as changes not in there. Solution was to switch to MPICH. Bleeding edge travis only uses openMPI 1.10.


AH: Link FMS rather than have in repo. NH: I agree. MW: Still think subtree best solution. Now FMS has dedicated automake build can formally install as module? MW: Had a long chat about this with Alistair. Not hot on submodule/subtree. NH: Just have in CMake or a script. AH: Makes sense.
RF: One of my jobs with decadal project is to have MOM5 linked with AM4. Uses a more recent version of FMS. Will be useful for RF. Have to get MOM5 talking with FMS used in AM4. MW: Have run MOM5 with latest FMS. RF: Just making sure no surprises, changes of interfaces. Not just FMS, also other bits and pieces. AH: Auto testing with multiple FMS.
MW: If we go path of building independent libraries, not sure how C world tracks this? ABI changes? How do you manage binary compatibilities? NH: Did that with OASIS, and not sure it was worthwhile. MW: C programs don’t seem to have these issues. NH: Using precompiled libs necessary in linux, for us not reason not to compile FMS when compiling MOM. MW: Not keeping public library? NH: More complexity than necessary. We’re just talking about splitting source code out into separate repo. Good idea. MW: MOM6 has FMS repo, and a macro repo above that the builds everything. Not sure we want to go that way. NH: access om2 works that way too but experiment repos are separate. MW: Maybe submodules / subtrees aren’t so bad. NH: MOM5 repo can have a build script that references a build script for FMS. MW: Doesn’t CMake have some functionality to check it out for you.
AH: Finish off CMake build scripts and add in FMS stuff.
MW: What they do with MOM6 is having issues. Will bring up with them. Maybe some convergence on library dependencies. AH: Don’t favour central lib install with MPI dependencies.


RF: Not much to report. Make sure WOMBAT can be called in MOM-SIS. Only outstanding issue. Just changing a few if statements. MC: Comfortable that it will run in ESM framework? RF: Not sure who is going to test? MC: OM2? RF: Should run in OM2.
MC: Richard Matear went to visit AH. I haven’t run anything yet. With experiments running under payu ready to go. OM2 test with WOMBAT.
AH: Is there a PR for these code changes? Make a PR. RF: Maybe said to do that. Split up testing to avoid duplication.
MC: Hakase wants to run this too I believe?

Tenth Model

AK: Set up RYF for Spence. Run 20 years. Looking at test bed for improved config. Improved bathy from RF. Conservative temp. Running at half the cost of previous config. 10Mh/yr, 60-65 KSU. Speed up from higher time ocean tilmestep, and ice is now 2 time steps per ocean timestep, compared to 3. Due to removal of fine cells in bathymetry in tripole. Wanted to use non-mushy ice, but low ice con in Baltic fails to converge thermodynamic temp profile. Should converge in a few steps, but limit at 100 and still doesn’t converge. Paul is using mushy. Had a run with non-mushy up top crash. Spinup7 is mushy spinup8 is non-mushy. 10-15% extra cost for mushy ice. Not sure if we’re CICE or MOM bound. Other resolutions are not using mushy. Want to set up an IAF tenth run starting in 1958. Can afford with cheaper model.
AK: TEOS-10, not sure if we want to use. Need absolute salinity and cons temp.
AK: Noticed gyres are much too weak in all resolutions. Looks have to careful with JRA55. Did a test with 0.25 with abs wind rather than relative. No change. Florida current 65%, EAC about 70%. Gulf stream is not separating properly. Mean position ok, but to variable. Maybe insufficient momentum. Doesn’t go around grand banks properly. Causes SST biases. Not sure how much to fix before IAF.
NH: All resolutions? AK: All resolutions are too weak. Gulf stream separartion is ok in tenth in average, but too much variance. Mean position in 0.25 is really bad. Biases around grand banks similar in all resolutions. NH: Improving separation improve biases? AK: Maybe. SSH is localised in model, but stretched out in obs.
NH: Is this specific to JRA55? Does it happen with CORE forcing as well? AK: Don’t know. Griffies said others find gyres a bit weak with JRA55. It uses scatterometer winds, which are relative to an eddying ocean. Not in the same location as a model. JRA55 paper suggest adding climatological mean current to the wind to force the ocean. AH: Should that be in the product? AK: Griffies says people aren’t too keen on Sujino suggestion. AH: Diagnose wind stress from 0l25 test? AK: yes. 10-20% change in stress in western boundary currents and southern ocean where large mean currents. Stress changes are in quite small areas, not a big effect on gyres.
MC: Do you recall what the EAC numbers were in model compared to obs. I thought we had 20Sv which is similar to OFAM/BRAN. AK: Obs: 18.7, 17.5 and 17.2 Sv about 2000m in models. 22.1 pm 7.5 from a mooring. Florida current is 30% too low. Well observed.
NH: What is big challenge in future? AK: Not sure how much to change before next IAF. Will put out a call for diagnostics. Also explain config and see if people have an issue with that.
AH: Doesn’t MOM5 not fully support TEOS-10? RF: Not obvious to user that can use TEOS-10. Kind of fudged. Proper way is to carry an extra tracer. Have preformed salinity and an adjustment factor to create abs salinity. Another way is to have abs salinity as a single variable and adjustment factor is zero. To use full TEOS-10 in MOM5, need 2 tracers. If you do it the same as the rest of the world would have a zero tracer. Don’t want a wasteful tracer.
RF: There is a newer way to parameterise the equation of state. Need updated. AH: New module? RF: Yes, just switch.


RF: Frazil not being redistributed. Needs fixing. AH: Affect other runs? RF: No just FAFMIP.

Technical Working Group Meeting, March 2019


Date: 12th March, 2019

  • Marshall Ward (MW) (Chair) NCI
  • Aidan Heerdegen (AH) CLEX, Andrew Kiss (AK)  COSIMA, ANU
  • Russ Fiedler (RF), Matt Chamberlain(MC) CSIRO Hobart


MW: Submitted parallel IO FMS patch. New automake made PR more complicated. FMS now buildable by automake. If we add new files/dependency build will fail. Not very auto. Works well. Some tuning with lustre. Just need a good io_layout with large contiguous chunks.

AH: Compression? MW: IO runtimes double with compression. Testing some of the newer algos and getting better numbers. AH: How does he get other compression into netCDF? MW: Custom libraries. Will make accessible when time comes. Have been using it. Good. MW: Haven’t heard back from GFDL yet. AH: will be needed for further high res models.

MW: RF big help with all the fill value stuff. RF: You put missing values in missing tiles. There is a mppnccombine bug which stuffs up. MW: Used netCDF fill value for restarts, MOM sets it to CMOR fill value.
automake goal is to make FMS into a library that can be centrally installed.
AH: Should just link against libFMS rather than have in the repo.
RF: part of decadal project is to update FMS. Wants to use it in AR4. MW: make it loadable module? AH: dynamic linking? AH: need access to compiled module files?
MW: Issues with GNU/Intel. Dale roberts wooing on this. Also module versions are an issue. Intel 18 can’t read 19 mod files for example. Don’t think there are any includes. If there are, should be fixed. RF: There is include mpi.f. Not sure how they include netcdf.
RF: Advanced with getting WOMBAT into CM2 harmonised MOM5. Fixed a few issues with redundant ifdefs. Hopefully sometime this week can finish. Pretty close by the end of the day.
MC: Might have an executable with WOMBAT compiled into it? RF: Yeah, just changed a few interfaces to get rid off ifdefs, changed to optional arguments for WOMBAT arguments.
MC: If Matear is there tomorrow, can try something.
AH: Now have a suite of models. Different resolutions, now with BGC as well. Great position to be in.

Technical Working Group Meeting, February 2019


Date: 14th February, 2019

  • Marshall Ward (MW) (Chair) NCI
  • Aidan Heerdegen (AH) CLEX, Andrew Kiss (AK)  COSIMA, ANU
  • Russ Fiedler (RF), Matt Chamberlain(MC) CSIRO Hobart
  • Peter Dobrohotoff (PD), CSIRO Aspendale

TWG Meta Stuff

AH will redo MOM5 governance doc for next meeting.
AH finding minutes a burden, MW suggested exploring other options.
MW: Will leave at the end of March. Will maybe try and attend. Given time.
Even in anarchy someone has to send out the email.

CICE Meeting

MW: CICE meeting. AK going. Going to Hobart Ocean Workshop? MC & RF not registered, might drop in.
MW: Also a VC: chat with Elizabeth Hunke. Who is going to attend? Just me? AK: Yes. MW: AH  come too? Ben Evans asked Rui to come. Not sure about NH. Assume interested.
MW: What to ask her about? Agenda? What motivated it? AK: Just that Elizabeth is around and could chat. AK: Any point me turning up a day before, more talking about Petra, but Petra thought it might be useful. Could show her how we set stuff up, some results?
RF: Anyone from Aspendale coming down? PD: Not sure. MC: Simon Marsland and Siobhan on the attendance list.
MW: Might ask about using latest GitHub branch (cice6). If we were to use it what should we do? Incorporate changes from OM2 codebase? Others more interested in physics?
AK: Might be interested in scaling work. Hoping to put some in my talk. MW: Fine with me.
MW: Not done as much as Tony Craig (?) on load balancing.
Monday 18th @3pm with Elizabeth (2 hours)
AK: Valuable networking opportunity.
MW: Would be great for NH to come.
MW: Maybe AK give a run down of some of the runs, start from there.

MOM5 Pull Requests

MW: RF been busy
RF: Bug in one of those in GW scheme. Was testing temperature in the wrong direction. Also something odd happens to temp rebinning at the bottom of a level compared to density. Missing value is zero. Interpolates first non-zero temperature to below bottom level. Because density in the rock is zero, can’t get a bounding. Problem with the way the diagnostic is originally done.
RF: Calculates transport in density one don’t account for transport in lower half of bottom cell, but temperature remapping you do. MW: Haven’t looked at the patch yet. Is this what Ryan Holmes was asking about? RF: This would speed up Ryan’s remapping. His PR was different. Trying to remap onto different levels. He sort of fudged the code. Take code from remapping onto density levels, and made something spoof, pretends neutral density is temp or salt. Don’t like what he’s done. Probably works, but not totally sure, but my optimisations might break some of the things he does. AH: Your optimisations are field dependent? RF: Yes. Assume it is density, with assumption density increases as you get deeper.  MW: He added a neutral density thing? RF: Trying to trick the code into something else.
RF: Can’t do it on more than one variable.
AH: Might be worth telling Ryan this might break his code.
RF: I thought he had put the commit in there. AH: No deleted the PR. He doesn’t have commit rights.
MW: Has a hard coded neutral density point that he has defined.
AH: RF still thought worthwhile? RF: Yeah, have a general thing, remap to level? A lot of code would be copy/paste. Could be a lot of work. AH: classes of rebinning?
MW: Not sure I understand exactly what RF’s commit does. Not sure I can add value.
RF: Just a lot faster.
AH: How did you pick up the error? RF: Was worried about it. Hadn’t checked rebinning to temperature. Wasn’t sure I had accounted for reverse in signs. In transport beta _ gm. Neutral physics utilities module. Checking for maximum and minimum temperatures on wrong levels. Hadn’t tested that diagnostic. Missed temperature. When tested failed. Doesn’t alter results of simulation, diagnostic slightly wrong. Other things were bit repro, all checksums were identical.
AH: So when code changes are made to diagnostics make sure those diagnostics. Make sure we paste in pics of diagnostics. Made sure to double precision in `diag_table`.

MOM5 Governance

Last month agreed to tackle PRs. MW: Paul never answered. Other didn’t answer. AH: AK didn’t answer! 😉
MW: A lot of weird hard constants in FMS. Data structures are weird.
MW: Other PRs when we got no answer? Ask for an update without interaction without a month? Have some policy? Paul looks more valuable. Other one is more FMS. Could call phone.
General approach for non-responding PRs: Get in contact again. Warn it will be closed. Close and say they can reopen.
MW: Sometimes got good ideas with poor implementation, accepted and completed reimplemented. RF: Short one best to redo a different way, and reject the FMS stuff. Contact Paul and get it done?
MW: No answer after prolonged time, incorporate good ideas in a different branch.
AH: Why coding now? RF: Had these ideas for ages, but noticed low hanging fruit. Remapping and submeso scale. Knew we could make significant time savings. Knew about these ages ago. Similar with tidal mixing. AH: Uses MOM timings? RF: It was slow, and looked at it and wondered about looping. MC: With changes what improvements? RF: 20-30% in each module. I run short cases, so data writing might dominate a bit. Will depend on the size of the model. Time spend on each tile proportional to mixed layer. MW: Shallow levels will be a big improvement? RF: yes. MW: Not  iterating where there aren’t values? RF: Yes. Two types of tests, check if entire tile can be topped, other times if a latitude can be stopped. RF: Did test of 1200 cpu job on OFAM grid too 30% off those routines.
AH: submeso is 10% of total ocean runtime.
RF: Starting a big run, good time to get it in.
MW; Sometimes said MOM was well balanced. Aggressively masks everything.
RF: Imbalance comes through the parameterisation code. KPP, Tidal mixing. Found another weird thing in the barotropic routines. Takes a lot of time. eta and pbot diagnose. No reason to diagnose the pressure at bottom on a u cell. Except if you’re writing the diagnostic. AH: standard for the code to check if diagnostic used before calculating? RF: Required for restart file. Check at restart stage and write it out that time. AH: don;’t restarts have to be field_table? RF: No
AH: If they don’t affect science can add to 0.1 at any time.
Ocean eta and pbot diagnose 10% of runtime.
AH: should we prioritise any changes. RF: just the ones I have put in. Others not so much. I’ll fix up the PR. Just got compiled and testing.

netCDF Parallel MPI IO

MW: Parallel IO stuff looking good and nearly done. Getting parallel IO without collation. Even restarts. A few masked cases where things look odd  with completely missing values.
MW: Fill value versus zero over land? If I do mppnccombine intelligently turns zero over land into missing values.
RF: When MOM sends diagnostics sends a mask with the call.
MW: Should land be zero or fill value? RF: should be fill. MC: What about restarts? RF: Used to have zero and then changed. Turned up in the density restarts.
AH: Performance?
MW: As fast as the number of disks. Can be subtle to configure. Have to balance the nodes with io_layout with ncpus on node. Negligible with 0.25 deg. Write speeds at about speed of lustre (half speed x number of disks).
PD: Fan of missing_value stuff. Parallel IO work from Dale.
MW: Rui will know about timing variance. Worried GFDL will find it slow and reject. Rui looked into compressed parallel IO. Interesting results. Reasonably fast. It’s half the speed of non-compressed. What is the serial (offline) compression time? No idea. AK: Is speed MB/s. Or twice as slow for total data file? MW: Twice as slow as the entire dataset.
MW: Currently uncompressed. Can then compress.RF: Need to work for regional output. MW: Do at FMS level. AH: Should test for regional output. RF: Regional output done by geographic rather than index. If by index would make it easier. MW: If you can get that for a test.



  • Amend MOM5 governance doc (AH)
  • Feedback to RF PRs (MW+AH)
  • Check back on Paul’s PR (MW)


  • Shared google doc on reproducibility strategy (AH)
  • Pull request for WOMBAT changes into MOM5 repo (MC, MW)
  • After FMS moved to submodule, incorporate MPI-IO changes into FMS (MW)
  • Incorporate WOMBAT into CM2.5 decadal prediction codebase and publish to Github (RF)
  • Move FMS to submodule of MOM5 github repo (MW)
  • Make a proper plan for model release — discuss at COSIMA meeting. Ask students/researchers what they need to get started with a model (MW and TWG)
  • Blog post around issues with high core count jobs and mxm mtl (NH)
  • Look into OpenDAP/THREDDS for use with MOM on raijin (AH, NH)
  • Add RF ocean bathymetry code to OceansAus repo (RF)
  • Add MPI barrier before ice halo updates timer to check if slow timing issues are just ice load imbalances that appear as longer times due to synchronisation (NH).
  • CICE and MATM need to output namelists for metadata crawling (AK)
  • Provide 1 deg RYF ACCESS-OM-1.0 config to MC (AK)

Technical Working Group Meeting, January 2019


Date: 15th January, 2019

  • Marshall Ward (MW) (Chair) NCI
  • Aidan Heerdegen (AH) CLEX, Andrew Kiss (AK)  COSIMA, ANU
  • Russ Fiedler (RF) CSIRO Hobart
  • Nic Hannah (NH) Double Precision
  • Peter Dobrohotoff (PD), CSIRO Aspendale

MOM5 CM2 code harmonisation

PD: Stopping an 18 year run with harmonised code. Seems successful. Not losing summer Antarctic sea ice, which had been an issue. Dave Bi gave his approval.

PD: Looking at new bug fixes on GitHub. Didn’t appear to be in the CSIRO code.

RF: The fixes I added weren’t to do with harmonisation. Diagnostic for transport on density levels. PD: Won’t affect our model run? RF: Yes. PD: Maybe should keep the 18 year run going. RF: There is also a fix to a submeso scale smoothing that you’re probably not using. 99% likely you’re not using that. PD: Could check by looking at the namelist? RF: It’s smooth hblt, or something, but also a note in the code specifying the namelist that shouldn’t be used because of this error. PD: Could you send me the namelist value? RF: Should be in GitHub issue/pull-request. AH: I’ll put links from your commits on to the slack channel.

AH: hblt_smooth

RF: If you applied the smoothing could smooth into rock. PD: Keep getting current going? Want to get an ENSO spectrum. RF: Shouldn’t make a difference, but will get noise due to changes in red sea fix. Statistics will be the same

AH: In the release candidate code we reproduced a red sea fix timing bug to make the comparison as clean as possible, using a namelist option. That has been stripped out before merging into the master branch. If you continue with this test run and it becomes your spin up run then you will not be able to do a clean comparison if you then change something else from the master branch version. Is this a test run or will it become a spin up?

PD: Sometimes test runs become real runs, but I’ll say this is a test.

AH: If at any point you start a spin up you need to be on a commit on the master branch. Currently running from a commit on a pull request that no longer exists. There is no comparable commit on the MOM master branch repo because of removing the salinity time unfix option, and merging in RF’s bug fixes.

NH: Second that. Also important if we want to continue to be harmonised, this is a divergence and if we carry on with that we’re diverging immediately.

PD: Alright. Will leave this run going to test ENSO spectrum, but will start a new parallel run for safety. Don’t want to be in the position where the test run gets turned into a spin-up because of lack of time.

AH: Calendar time not compute time is your constraint? PD: Yes. AH: Definitely agree with that strategy. PD: Never have enough compute time, but always have that trade off.

AH: Will tag the code with CM2 version which can use to identify the code. PD: Should tag straight away and I will clone and let people know this is the correct code.

NH: Reproducibility is important, and the current MOM code does not reproduce. PD: On restarts. NH: Yes, so would be a shame to lose that by not using the merged code.

AH: Not only is it reproducible, but NH is running tests for this. NH: That’s right. Just a simple 2 day versus 2×1 day runs. To do that test presently turns off red sea fix. Now I can turn it back on? RF: It should now reproduce. AH: Turn it back on and check it reproduces? AH: This is reproducible between runs, not necessarily reproducible to before RF’s fix. PD: Which is reproducible on restarts and which isn’t? AH: Current harmonised code in the MOM5 master is reproducible between restarts, the code from the pull request that PD is currently running is potentially reproducible if you turn off the flag we introduced which emulated the incorrect behaviour of the old MOM5 CM2 code for testing purposes. PD: So the harmonised code in the pull request is different to the master branch? AH: Yes, in that the hack to emulate the incorrect timing behaviour of the red sea fix has been removed, and a couple of RF’s bug fixes added. PD: Going forward will there be a harmonised branch and a main branch? AH: No, there will be a tag identifying where you get your code from. If you need to pull in updates but don’t want all the updates in master, then you might start a new branch, and cherry pick those updates. At that point you might have a different branch, but not currently. In general better off not having a separate branch, as it just starts diverging again. I don’t know how you guys work and I’m guessing you don’t want code changes, but if, for example, you just wanted to add code changes that added diagnostics, you could add those in, and have something to compare against. PD: Just trying to figure out how it will work, if there are different branches, and if down the track we want to develop the harmonised code again. AH: Yes, and if you have some testing then you an add code and test to see if there are differences in the output, and so add code changes with confidence.

MW: Highlights that we have not been tagging MOM for some time. Maybe we need more regular tagging.


ESM code harmonisation

PD: In an ESM meeting on Friday Tilo said it was too late to include changed code for CMIP6.

AH: Whatever we did he wouldn’t put it in CMIP6? PD: Yes. AH: Invested too much time on spin ups? PD: Yes. AH: So no particular rush to do this. PD: I thought it was good to pass that on. AH: Good to know, thanks. MW: There ESM for CMIP6, but also in the CoE. Will they use what you’re working on? AH: Yes, but shame, as we’re not then using the same code as the CMIP6 submission. MW: I thought that was your interest in ESM. AH: Yes, not sure.

MOM5 Governance model

MW: Not sure how much we can do without Steve being present, or anyone from GFDL.

AH: Made some notes, hoped to get some feedback and others make changes, additions. MW did add some useful points about defining domain experts.

AH: The MOM5 repository and community is not welcoming to those outside the current clique of COSIMA. Pull requests languish for years without attention because it is no-one’s responsibility, no-one has a defined role. Even if some people put their hands up to monitor pull requests.

Have a, to tell people how to contribute to the code. There are users outside this group who use it, see them on the MOM users mailing list, and can use that channel to advertise it.

MW: That was my experience as a grad student. Not sure who is charge of the code. Tried to contribute code and found it intimidating. Not a fan of overly prescriptive instructions on how pull requests must look, or how code must be written. That has made me less likely to report bugs. Would rather get bad contributions than no contributions. AH: Poor quality contributions require effort from us to work through them. MW: Yes, just want to make sure they know it is ok to screw-up. A lot of projects require a lot of environment information etc, which can be onerous for new users. Now tend to ignore those instructions and wait for devs to ask for it. Do think that governance model is good, but want to encourage contributions and emphasise it is the effort that matters rather than the quality of the contribution.

NH: I think I agree. Want to be more friendly to outsiders, also agree with MW, want to put as few roadblocks as possible. People have little incentive to contribute to the repository. The harder we make it, the less likely they are to contribute. With Pull Requests, in order to merge we need testing, but people aren’t going to do those tests. They test for themselves to satisfy their scientific goals and that’s it. Not sure how to reconcile that.

MW: Automated code coverage tests help. A lot of CI services panic too much when code coverage drops after a contribution. This can be useful if it then spurs contributors to improve units tests to improve that metric. AH: Currently have 0% code coverage, so it can’t get worse!

MW: Yes, one issue is we have no code coverage presently. But looking more broadly if we have some automated tests that can tell us what is broken that can help a lot. I know we have testing. AH: Only compilation testing.

AH: If we do this, it will be a burden, want to minimise this, hence the document defined some steps for assessing a PR:

  1. PR assigned to committer
  2. Committer checks PR conforms to guidelines
  3. PR passes CI checks
  4. Iterate until correct or time limit passes (say developer does not update PR as needed)
  5. Testing? Do we push that burden on to contributor to demonstrate bit repro? What about performance?
  6. Accept or reject (rejection is automatic if contributor does not meet expectations above and time limit is exceeded, so rejection at this point would be unusual, but might occur if the efficacy or efficiency of the code is questionable)


MW: James Munroe has experience contributing to dask, managed by experienced software engineers. He had to do a lot of testing, as well as style and engineering changes. They had someone asking him to make changes until it was deemed good enough. We could do this too, but it would require engagement from us. Also, are we overthinking this? We hardly get any pull requests.

AH: Yes, but some of this stuff is useful for us to do too. A lot of it is just good communication. It may sound onerous, but in many ways it makes it easier, so people aren’t trying to guess the right thing to do, e.g. with code style it may be as simple as pointing to a particularly well written section/module and say “seek to emulate this”

MW: GFDL is struggling with this right now. I sent them a beefy FMS patch and they did not how to handle it. They didn’t want it, but they didn’t want to just reject it. Might be worth figuring out how they are dealing with it. AH: They might copy what we end up doing. MW: Yes, they might end up copying this.

AH: First up, should we consider roles, e.g.

Contributors: Anyone who wants to contribute code

Committer: Anyone with commit access

Maintainers: Committers who check PRs, assign PRs to other committers and maybe do some other admin tasks.

Admins: Do we need another layer? Currently only Steve, Nic and I are Organisation Owners and Admins on the MOM5 repo. Need at least 2 admins at all times (run over by bus scenario).

Sponsoring institutions: Acknowledge role of institutions which provide time for code development?

BDFL: Steve? (Does he even want to be a Benevolent Dictator?)


AH: Do we want to do something like this? NH: Yes. It is a great idea. Write it down and we can use it ourselves, and our collaborators.

AH: Are we happy with the roles? NH: Yes, need to look more closely. Looks good. Keep it simple.

MW: Maintainers should be admins, at least for now. I wonder if Steve would prefer to not have a formal role, as he is transitioning out of the coding. AH: Would be want to be BDFL? MW: Would probably accept it, but maybe he doesn’t want to

AH: I think it is good to keep him there. At some point we will transition to MOM6 and it will no longer be my role to look after this. Others will move on too, so it would be good to have Steve still there in that eventuality.

MW: He could be a decider. AH: Yes, like having a Queen. Just have to host him in visits every now and then. MW: Head of State, but not the Head of Government.

AH: Ok, get rid of admin/maintainer distinction. Just make them Maintainers.

MW: Sponsoring institutions is interesting. AH: It was to acknowledge that institutions that pay people like me might have a reasonable expectation that the people they pay to help maintain the code could have some say in the way it is run. RF: I don’t like that, it is a community code. MW: I agree with RF, but there is some reality. AH: More about expectations. If I leave my job, the CoE might have some expectation that my replacement would be able to become a committer/admin. MW: I worry about sponsoring institutions insisting on having some oversight. AH: I always thought CSIRO was a bit keener on the whole “official stamp of approval” thing, but if we don’t like it I’ll get rid of it. No worries.

AH: Ok. Lets codify the roles, and look into a to sit at the top of the repo to tell people how to contribute code. This all started with wanting to have timely responses to Pull Requests. Who wants to be what?

MW: What is the

NH: Happy to be a maintainer, but COSIMA would have to pay for my time. AH:

MW: Don’t have an explicit list of users.

RF: How far back do you go with sponsoring institutions? Back to 2009?

MW: I would prefer not to involve them.

RF: Just leave it. Keep it simpler.

MW: When I say sponsor, I mean someone who can’t function without this code. Need a maintainer.

AH: It is literally my job, and happy to volunteer as a maintainer.

AH: Maintainers check PRs, and then assign them to people, we need others to be willing to have a PR assigned to them. MW: Happy to do it, but feel like others are more invested. AH: Doesn’t have to be just you. It is just having someone respond to a PR, check progress, feedback what is required to get it merged, close it if it doesn’t meet standards and the submitted doesn’t respond to queries to fix it.

MW: Yes we need that role, at the moment it is AH and NH. If you want to add me I’m happy to do it, just not sure where I will be in the near future.

AH: Ok how do people get these roles? Commit access is currently Steve, NH and AH. Do other people need it, or want it. How do we decide who should have it?

MW: Best to have one person responsible for commits/merges. AH: Steve recently merged a commit that needed to be reverted. MW: Which is what prompted this. AH: Yes, and he wouldn’t have felt the need to do it if he knew it was being handled.

AH: Do we need to do this now? Specify who is on “the committee”?

MW: Contributor is obvious. Pull committers from contributors. Pull maintainers from committers. If you don’t get enough commits, then a project just dies. AH: Don’t need to codify how someone becomes a committer? MW: No. We’re too small. Look on the issues, and PRs, represents very few people. AH: Ok, so no need right now.

MW: Quite premature to codify this.

AH: Need a ``, how to contribute.

MW: Any role for governance is getting the word out. Find the user base. There are a lot of MOM users out there. Steve cited a thousand MOM users in 2010. Governance should be about where are these people, and then get the community. AH: You run the danger of “hey guys come and contribute” and not have processes in place. MW: A good problem to have. AH: What happens if you got 10 PRs tomorrow what would you do with them? MW: Deal with them one at a time. AH: But who would deal with it? We have existing PRs that people aren’t being dealt with, one has been there for a year. MW: Yeah, but it is a bit stupid. Changed an integer to an 8-byte. RF: I would have rejected it because it used a specific kind. Change was a good idea, but badly implemented. MW: Ok, let’s look at this and learn some lessons and produce a document.

MW: So lets look at this PR. What are we going to do about it? So who do we assign this to? I would assign to Russ as he has expressed a strong opinion. AH: Ok. MW: What would you say about this PR RF? RF: I would split into two, as they are unrelated. If you do submit these things, try and keep to individual issues. Specifically about this, first change is non-portable, which can easily be fixed. Would we do the fix, or would he do the fix? All it needs is a selected_integer_kind. MW: This is a portability issue. RF: Yes, anything from now on that gets contributed should be standard. MW: Lesson should be an evolving document about what we want. RF: Yes, standards compliant, independent commits. MW: Would you be ok with being assigned this? AH: Already done it. RF: Second part could be a real bug in remapping vectors. Maybe get back to him and ask if he can show a test case. AH: Ideally this would be associated with an Issue which explains the problem. AK: Really ideally would include a test case that fails and that subsequently passes with the fix in the PR. MW: I would argue against pedantry. GitHub treats Issues and PRs similarly, so while an Issue with a test would be ideal, shouldn’t be mandatory. If you would rather formalise then fine.

RF: Have to leave

Subsequent discussion:

AK: Who feels they have responsibility, so things don’t languish, and who has authority to say what is going to happen. NH: And who has time. MW: Good not to dump maintainer responsibilities on NH.

AH: If suggestion is prescriptive so as not to waste time wondering how to do things. Not mandatory, but good to give guidance.

MW: Current touchy point is assigning stuff to RF. Had a meeting and asked it was ok, but won’t scale. Let’s clean up the pull requests.

MW: I have different idea of purpose of Issues page to AH for payu. You see as dumping idea for ideas (AH: Yes), and I see it as something to keep as small as possible. Ok for a small group, but with a larger group it can balloon and lose track of good ideas. MW: Yes bad idea, or not enough time should be a criteria for closing an issue, without losing the idea. Would be great to get issues down to 1-2 pages, would reveal a lot of lessons.

AK: Paul’s PR is pretty substantial. A whole new file, ocean_basal_tracer. MW: But bungled the build script. Also made a timer that does nothing. This is a good issue with a terrible title. This is a good example.

MW: These projects work best when they’re community driven, that we agree to do it. AH: If you put up your hand to be assigned stuff, you will get assigned stuff. If they don’t, then people won’t. AK: If someone has that role you feel ok assigning to them.

MW: When we talk about governance, lets not worry about users, lets figure out how to assign issues, that is a great place to start. Not worry so much about how users should make PRs.

NH: I agree, but maybe still work on guidelines for contributors. There was a document called “How to contribute”, based on technical specifications. It was a markdown doc in the original website. AH: Maybe that exists somewhere. NH: It’ll be around somewhere, outlines how to do a PR and things like that. AH: Should definitely start with that.

MW: Make it a goal this week to deal with those two PRs. Assign me Paul’s if you want.

AH: Willing to be a maintainer? NH: Yes, I like being part of that community. Hard to know how to dedicate time to it sometimes. We need to do a certain amount of maintenance. Make think of it as pro-bono professional work. Matt England is keen on that sort of stuff, he might be willing to support it/sanction it/pay for it.

Agreed: Maintainers: Aidan, Nic, Marshall. Ask Steve if willing to have BDFL status.


AK: Modified topography to remove terraces (bug). Smoothed a seamount near Sverny Island, and eliminated very small cells near tripoles. 54KSU / model year. This is repeat year forcing. Time step is 720s. Did a test run with 900s and it crashed. 720s is a factor of 1800 (coupling time step). No factors in between.

AK: MOM was going unstable, generating 18 m/s velocities, CICE was the first thing to crash with ice remapping error. Was using ndtd (number of CICE time steps per MOM baroclinic timestep). When using ntdt=3, making CICE stable with vaguely unstable MOM. At one point in the run I added Rayleigh damping in Kara Strait as I had done in IAF run. MOM was going unstable in Kara Strait, moving ICE around too quickly, and CICE crashed. Once I stabilised MOM, could get away with ndtd=2, so now the model is CICE bound rather than MOM bound. AH: What is runtime? AK: 2 hours, so can do 2 months/submit. MW: What were the fails at the beginning? Related? AK: Showing out from a run-summarising tool. Just shows when runs failed. Not necessarily crashes, might have just stopped to change something. AH: So when you say MOM went unstable, not so unstable it crashed, but unphysical velocities? AK: Haven’t looked in detail, but imagine it would be grid scale alternation and that sort of thing. AH: Does that mean, in order to compatible with CICE, maybe we need to reduce some of MOM’s thresholds for warnings so they are better matched? AK: Yes. CICE has a CFL condition that is tighter than MOM. MW: They are solving different equations. AH: Not saying they should, or could be, exactly the same, but MOM is more permissive than CICE, but the crashes happen in CICE. So if you reduce the limits in MOM will diagnose the problem in the correct place, rather than indirectly through CICE crashing. A new user would then get the right information.

AK: Haven’t examined the fields. It is right on the limit of the barotropic CFL limit. Splitting factor of 80. The outputs show it is right on the limit. MW: Griffies would hate that. AH: Pretty routine to change that when timestep changes. AK: Had to go 100 for dt=900s for it to pass that check. Might be wise to up that number.

MW: This is tenth degree? Really dropped your CPU count. AK: This is a minimal config. Dropped from 6K to 2K, getting same throughput as larger model. MW: But with lower ndtd and higher timestep. AK: Yes. AH: timestep is like a superpower.

NH: Great! Well done. AK: Thanks for your work. On the number of CPUs, it is good to run minimal configs, as they are inherently more load balanced. Each CPU is doing more work and a greater variety of work. A good reason this minimal config is more efficient. MW: Won’t it run slower if you reduced timestep? Doesn’t make it more efficient necessarily? NH: I think it does.

NH: How does this compare to MOM-SIS? We can’t really compare without running with the same grid? MW: Can usually use the MOM timing and add 15%. I went back and checked this numbers. AH: I think I was getting six months / submit with MOM-SIS at a nominal 7200 core layout, with dt=600s. The larger ACCESS-OM2 config has 6K cores in total, but only 4300 MOM cores.

PD: Have to go now. Bye.

NH: Are you going to try getting the bigger config running? AH: 6 months per submit is very interesting. Could do some decent 100 year runs. Could be worth looking at.

MW: According to the numbers I have been posting, could speed MOM up by a factor of 4 with more cores. CICE ice step is also scaling well. Don’t know if switched to sectrobin, or missing a coupling bottleneck. This model could scale well. Has anyone checked with sectrobin? NH: This is using it. MW: I am talking about much larger, I get improvement up to 12K CICE cores. AH: Load balancing not an issue? MW: Maybe, but I’m not seeing it with the function I am testing. Is this new? Or was this always the case and I’m not seeing what slows the model down? If it is new, maybe we should experiment with throwing more cores at CICE. Even with the low config, maybe you could ramp the CICE cores up to 600? NH: That one is MOM bound. AK: The IAF was using round robin, and dt=450s. MW: Right so, that is your difference right there. AK: Dropping ndtd to 2 makes it MOM bound.

NH: Let me know if there is anything I can do to improve the load balancing in larger config. Pretty sure there is larger config using sectrobin.

MW: I’ve seen consistent improvements in all CICE configs, including 1 deg. Doubling cpus in CICE and MOM individually in 1 degree saw improvement in both, but got errors when I tried to both. Gave up because 1 degree not so important. Will try again in tenth degree.

NH: Sounds like you’re discovering what the best config should be using some evidence based mechanism. The 1 degree config was pulling numbers out of hat that balanced ok with decent efficiency. Maybe want a small/medium/large config, but not for all resolutions. 1 degree maybe just want max throughput. Would be great to capture what you’ve discovered, and actually start to create those configurations.

MW: MOM numbers are clear. Time per step is very informative. Want numbers around 0.1s/step. Allows us to tackle this independent of time step. Surprised CICE is scaling. Reminds me of the last COSIMA talk which everyone said was wrong because I didn’t have enough ice. Will make a figure, which will show strong scaling numbers for CICE. Looks like it a strong scaling model.

AK: On CICE, there is a workshop in Hobart in late Feb. It is a CICE modelling workshop, and Elizabeth Hunke is visiting. AH: Are they up to CICE6? MW: Dealing with a fork. Los Alamos has a version no-one is allowed to see. NH: Worth looking at the CICE repo. Our CICE5 version contains all changes up to CICE6 tag, We’re as up to date as you can be on CICE5. AH: Have you back-ported stuff? NH: Yes, it wasn’t very hard. This is CICE6, but it has CICE5 in the repo. AH: They have a cice6 branch. MW: What is ice-pack? NH: They have split out the column physics to make it more portable. It is just code from CICE5 but repackaged. AH: Icepack is included in CICE6 as a submodule.

AH: Ruth has been having queuing issues, anyone else having problems? Might just be someone using broadwell heavily. Do you know if we have to use broadwell for minimal? NH: My runs used to crash. AK: My runs are ok normal. Running the same as Ruth. AH: Using less than 1GB/processor? NH: I put something in config.yaml so when running in normal it used the high memory nodes. Did you take that out? AK: Yes I removed that. NH: Who knows. So much has changed since then. Memory can spike.AK: Ruth should be able to run on normal without requesting extra memory. MW: What made you think it was a memory issue? NH: Crashed at a particular place on initialisation in MOM where I know there is a big jump in memory usage. I have made an issue about it. It is in FMS. As soon as I went to the high memory node and it went away. Doesn’t print out specific memory errors, but as soon I as I increased memory it ran fine. Good to go back if it has gone away. It is something that gets done on the MOM root PE on initialisation that increase memory usage a lot. AH: Shame we no longer have access to the PBS logs as there was a lot of information in there on this sort of thing.

MW: Intel 19 gives awesome error messages. Not only are tracebacks more readable, but they give 7 lines of code each side of the error. Now getting MPI trackbacks without even using mpi-debug. Can’t use openmpi/1.10 with Intel 19.

Parameter sweep runs

AH: To use extra time at the end of last quarter suggested we could do a parameter sweep, which would use up a lot of compute time in short order. It didn’t end up happening, but Andy Hogg thought it would be a good capability to develop in case we did want to deploy it. I understand AK has done this a lot? AK: Well sweeps of 2 parameters, yes.

AH: Not a big deal, using payu, maybe have a YaML file specifying parameters and how to sweep them, create a bunch of payu configs programmatically and run them. MW: Always wanted to add this a a feature in payu. Spin off 20 runs with one file. AH: Any particular ideas, or anyone want to do it, feel free, otherwise will go on my very long to-do list. NH: I couldn’t think of anything. I don’t know enough about the science. AH: Me too. I have no idea what parameters I would want to change, so that is where it stopped. The parameter search could mean you always have something to run if there is spare time and add it to the database. If you have a database of runs, you effectively have a parameter search if you change anything.

AK: Have a current grant to do a study on parameter sensitivity in CICE. This capability could be useful.

NH: I have written something which can divide your grid by an integer number, creates initial conditions and everything. Could be interesting to programatically create a bunch of configs and run them with something like this.