Setup a CESM 1.0.x benchmark run on a generic system

This is a cookbook to setup a CESM 1.0 benchmark run on a generic (Linux) system.
See also porting CESM in the CESM user's guide: http://www.cesm.ucar.edu/models/cesm1.0/cesm/cesm_doc/c2161.html

System requirements

• Compilers known to work: intel 10.1, pgi 7.2, 8.0, 9.0, (pathscale 3.2)
• MPI implementations known to work: openmpi 1.4, 1.5, mvapich2 1.4, 1.5
• tcsh CESM setup scripts are written in tcsh

Compile NETCDF (Requirement)

• See also http://www.ncl.ucar.edu/Download/build_from_src.shtml#NetCDF

• Download NETCDF

  wget http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-4.1.1.tar.gz
  tar xfvz netcdf-4.1.1.tar.gz
  cd netcdf-4.1.1
  

• Compile netcdf with the compiler you use later to compile the model. For example for intel compiler do:

  export FC=ifort
  export F77=ifort
  export F90=ifort
  export CPPFLAGS="-fPIC -DpgiFortran"
  ./configure --prefix=/usr/local/netcdf-4.1.1-intel --disable-netcdf-4 --disable-dap
  make
  make test
  

• Install NETCDF

  make install
  

Download CESM source code

• Download from NCAR SVN. A password is needed to checkout. Please register at: http://www.cesm.ucar.edu/models/cesm1.0/register/register_cesm1.0.cgi In case you don't have time to register ask urs.beyerle@env.ethz.ch

  svn export https://svn-ccsm-release.cgd.ucar.edu/model_versions/cesm1_0_2 cesm1_0_2
  

Adapt configuration files

• Change to Machines directory

  cd cesm1_0_2
  cd scripts/ccsm_utils/Machines/
  

• Meaning of filenames
  
  

  Filename	Purpose
  env_machopts.*	Set environment: Can be used to set paths to compiler, MPI library, NETCDF library
  Macros.*	Set compiler name and paths to MPI library, NETCDF library. Set compiler options
  mkbatch.*	Setting for queuing system

  where * corresponds to a machine.

• As starting point take configuration files of a machine that is close to your environment. For example if you run on Linux, have a look at brutus_io, brutus_im, brutus_po or brutus_pm where i=intel, p=pgi, o=openmpi, m=mvapich2. You can get a list of pre-configured machines with

  cd cesm1_0_2/scripts
  ./create_newcase -l
  

• Let's assume you use intel and openmpi, start with *.brutus_io files:

  cp env_machopts.brutus_io env_machopts.your_machine
  cp Macros.brutus_io       Macros.your_machine
  cp mkbatch.brutus_io      mkbatch.your_machine
  

• Add to config_machines.xml a configuration tag for your machine (your_machine) - only the important lines are listed below

  <machine MACH="your_machine"
           DESC="Test System"
           EXEROOT="/scratch/$CCSMUSER/$CASE"
           OBJROOT="$EXEROOT"
           INCROOT="$EXEROOT/lib/include" 
           DIN_LOC_ROOT_CSMDATA="/scratch/cesm1/inputdata"
           DIN_LOC_ROOT_CLMQIAN="/scratch/cesm1/inputdata/atm/datm7/atm_forcing.datm7.Qian.T62.c080727"
           BATCHQUERY="qstat -f"
           BATCHSUBMIT="qsub" 
           GMAKE_J="4" 
           MAX_TASKS_PER_NODE="4"
           MPISERIAL_SUPPORT="FALSE" />
  

  please set the following variables:

  EXEROOT=                     # working directory, final location of binary and output files
  DIN_LOC_ROOT_CSMDATA=        # input data, date will be downloaded on the fly
  DIN_LOC_ROOT_CLMQIAN=        # input data, data will be downloaded on the fly
  MAX_TASKS_PER_NODE=          # define cores per node
  

• Configure mpirun execution: Search in mkbatch.your_machine for the line starting the executable ccsm.exe and replace it with the correct mpirun command for your system, for example something like

  mpirun -np ${maxtasks} ./ccsm.exe >&! ccsm.log.\$LID
  # or
  mpirun -x LD_LIBRARY_PATH -np ${maxtasks} ./ccsm.exe >&! ccsm.log.\$LID
  

Create a new case

• Change to scripts directory

  cd cesm1_0_2
  cd scripts
  

• Define the machine type (MACH), resolution (RES), compset (COMP=B: fully coupled model)

  MACH=your_machine
  COMP=B
  RES=1.9x2.5_gx1v6
  

  Possible values for RES are T31_gx3v7 (~3°), 1.9x2.5_gx1v6 (2°) or 0.9x1.25_gx1v6 (1°).

• Define a case name (in principle, can be any name)

  CASE=$RES-$COMP-benchmark
  

• Create case

  ./create_newcase -res $RES -compset $COMP -mach $MACH -case $CASE
  

Change the layout

• The layout will define which model component (ATM, LND, ICE, OCN, CPL) will run on how many cores.
• To change the layout you don't have to recreate the case (but you can if you like).
• Change into case directory

  cd $CASE
  

• Clean the case

  ./configure -cleanmach
  

• Configure layout (all components will run on all cores)

  NTASKS=16 #or# NTASKS=32 #or# NTASKS=64 #or# NTASKS=128 #or# NTASKS=256 #or# NTASKS=512
  
  ./xmlchange -file env_mach_pes.xml -id NTASKS_ATM -val $NTASKS
  ./xmlchange -file env_mach_pes.xml -id NTASKS_LND -val $NTASKS
  ./xmlchange -file env_mach_pes.xml -id NTASKS_ICE -val $NTASKS
  ./xmlchange -file env_mach_pes.xml -id NTASKS_OCN -val $NTASKS
  ./xmlchange -file env_mach_pes.xml -id NTASKS_CPL -val $NTASKS
  ./xmlchange -file env_mach_pes.xml -id NTASKS_GLC -val 1
  ./xmlchange -file env_mach_pes.xml -id TOTALPES   -val $NTASKS
  ./xmlchange -file env_mach_pes.xml -id ROOTPE_OCN -val 0
  

• Exception: For resolution T31_gx3v7 with NTASKS=32, NTASKS=64

  NTASKS=32 #or# NTASKS=64
  
  NTASKS_ATM=24
  NTASKS_OCN=8
  NTASKS_LND=24
  NTASKS_ICE=20
  NTASKS_CPL=24
  NTASKS_TOTAL=32
  F=$(( $NTASKS / $NTASKS_TOTAL ))
  ./xmlchange -file env_mach_pes.xml -id ROOTPE_GLC -val 0
  ./xmlchange -file env_mach_pes.xml -id NTASKS_GLC -val 1
  ./xmlchange -file env_mach_pes.xml -id ROOTPE_ATM -val 0
  ./xmlchange -file env_mach_pes.xml -id NTASKS_ATM -val $(( $NTASKS_ATM * $F ))
  ./xmlchange -file env_mach_pes.xml -id ROOTPE_LND -val 0
  ./xmlchange -file env_mach_pes.xml -id NTASKS_LND -val $(( $NTASKS_LND * $F ))
  ./xmlchange -file env_mach_pes.xml -id ROOTPE_ICE -val 0
  ./xmlchange -file env_mach_pes.xml -id NTASKS_ICE -val $(( $NTASKS_ICE * $F ))
  ./xmlchange -file env_mach_pes.xml -id ROOTPE_OCN -val $(( $NTASKS_ATM * $F ))
  ./xmlchange -file env_mach_pes.xml -id NTASKS_OCN -val $(( $NTASKS_OCN * $F ))
  ./xmlchange -file env_mach_pes.xml -id ROOTPE_CPL -val 0
  ./xmlchange -file env_mach_pes.xml -id NTASKS_CPL -val $(( $NTASKS_CPL * $F ))
  ./xmlchange -file env_mach_pes.xml -id TOTALPES   -val $NTASKS
  

Set simulation length

• In general, CESM is hardwired to generate monthly average data. In principle this can be turned of but needs a lot of code changes. Therefore it's not considered here. The following two cases are suggested instead:

• CASE 1: Run a short simulation (10 days) with producing no restart files (REST_OPTION=never) and no archiving (DOUT_S=FALSE)

  ./xmlchange -file env_run.xml -id STOP_OPTION -val 'ndays'
  ./xmlchange -file env_run.xml -id STOP_N      -val '10'
  ./xmlchange -file env_run.xml -id REST_OPTION -val 'never'
  ./xmlchange -file env_run.xml -id DOUT_S      -val 'FALSE'
  

• CASE 2: Run a longer simulation (1 months) with producing restart files at the end

  ./xmlchange -file env_run.xml -id STOP_OPTION -val 'nmonths'
  ./xmlchange -file env_run.xml -id STOP_N      -val '1'
  ./xmlchange -file env_run.xml -id REST_OPTION -val '$STOP_OPTION'
  ./xmlchange -file env_run.xml -id DOUT_S      -val 'FALSE'
  

Configure the case

• Configure case

  ./configure -case
  

Change history file output frequency of atmosphere model (cam)

• CASE 1: leave the default which is monthly output data
• CASE 2: change history file output frequency to one hour. Edit Buildconf/cam.buildnml.csh and set just after &cam_inparm the variable nhtfrq = -1

  cat Buildconf/cam.buildnml.csh
  ...
  &cam_inparm
   nhtfrq  = -1
   absems_data            = '$DIN_LOC_ROOT/atm/cam/rad/abs_ems_factors_fastvx.c030508.nc'
  ...
  

  In case you change Buildconf/cam.buildnml.csh, you can not go back by just deleting the lines. You have configure an new case with with ./create_newcase !

Build the case

• Build and compile the model

  ./$CASE.$MACH.build
  

Run the model

• Run the model, for example with LSF queuing system

  bsub < $CASE.$MACH.run
  

• To start without a queuing system just execute:

  ./$CASE.$MACH.run
  

• Timing results can be found after the run has been successfully completed in folder timing

  cat timing/ccsm_timing.$CASE.*
  ...
  Model Throughput:         6.39   simulated_years/day 
  ...
  

Change the resolution

• Recommended resolutions are T31_gx3v7 (~3°), 1.9x2.5_gx1v6 (2°), 0.9x1.25_gx1v6 (1°)
• Create for each resolutions a separate cases !

Produce a summary

• Create performance matrix for CASE 1 and CASE 2. Fill in Model Throughput in simulated_years/day

• CASE 1: Run a short simulation with producing almost no output (I/O) - Values are from brutus.ethz.ch
  
  

  resolution / layout (NTASKS)	16	32	64	128	192	256	512
  T31_gx3v7	12.0	18.3	26.9	--	--	--	--
  1.9x2.5_gx1v6	--	2.27/2.60					--
  0.9x1.25_gx1v6	--	--	1.88

• CASE 2: Run a longer simulation with producing hourly output data - Values are from brutus.ethz.ch
  
  

  resolution / layout (NTASKS)	16	32	64	128	192	256	512
  T31_gx3v7	7.9	10.8	13.6	--	--	--	--
  1.9x2.5_gx1v6	--						--
  0.9x1.25_gx1v6	--	--