Technical Working Group Meeting, January 2018

MPI Errors on raijin

• Marshall: MPI updates on raijin to try and improve MPI performance by turning on all mellanox accelerators. Mellanox can be hit and miss. Had to recompile as fabric was updated. OpenMPI has direct calls to mellanox in library?
• Andrew: tenth job hung. Raised issue. Ben said turn off all mca flags. Then reading the conf file. Lustre issue. Should be fixed.
• Marshall: fca is now turned off. MOM has few collectives.
• NCI changed default config settings which may have let other jobs work. Has now turned everything off. Maybe try and get some UM/MOM testing into NCI upgrade testing suite to avoid these issues.
• Nic: if we can’t get tenth running, fairly sure it is mxm stuff. Marshall: mxm reported too many retries error (jobs become list). Andrew’s jobs are hanging in MPI_Init(). Nic: experienced hang on gather in OASIS without mxm.
• Russ: working on CM2.5. Getting sat-vap pressure errors in first timestep. Which is why Russ implemented traceback update in MOM5 code. Sat vap errors written to stdout. Anything written to stdout not on root PE don’t get output. FMS has stdlog, stdout and stderr. Unless recompiled will not make it to stdout. Most gets redirected to /dev/null in FMS for every PE except root PE. Performance issue.
• mpirun can report which rank is reporting which line (tag output)
• Nic: so any “print *” statements will have a major performance hit. Used to be a lot of them in MOM5.
• Marshall: will have to assess all accelerators separately from now. 5 accelerators, 2 never implemented, all off by default. Now all 5 available, all off by default now, but initially were on but caused instabilities.
• Marshall: NCI are continuing to push openmpi 2/3. Still have speed issues. Need to get in top of it. Memory pinning has changed. Either explicit or changed. This is a critical performance issue for us. We need to be able to say why we don’t use openmpi 2.
• Peter: need a broader discussion around older versions, compiled against older libraries. Papers have been published with these executables. Do we continue to support software which is still being used for science. Scientists aren’t always interested in same things as HPC people. Bigger discussion needs to be had. To what extent does NCI as a partner need to be supporting these deprecated libraries?
• Marshall: our jobs to communicate our issues to program leaders
• Peter: is reproducibility off the table once hardware changes? Yes. By 2nd quarter next year clean slate.
• James: do we need to run these models on other hardware for reproducibility? Marshall: NCI have some secret stuff (other architectures)! Not on the normally accessible queues, but maybe could be used for testing?
• If we have our jobs in MOM6 tests we get cross platform tests for free.