Bern 2.5D Model

How to run several thc-runs in parallel ProjectBern25Dparallel
Bugs, limitations, ToDo's ProjectBern25Dbugs

Source code

SVN repository https://svn.iac.ethz.ch/repos/projects/bern2.5d (classic view)

Or browse the source code https://svn.iac.ethz.ch/websvn/repos/wsvn/projects.bern2.5d (nice view)

Check out latest version (trunk) - the trunk is work in progress!

svn co https://svn.iac.ethz.ch/repos/projects/bern2.5d/trunk bern2.5d-trunk

If you have once checked out the trunk version you can always update your working copy of the trunk to the latest trunk version on the svn server by executing inside your trunk folder bern2.5d-trunk the command
```
svn update
```

View change log of latest version (trunk) https://svn.iac.ethz.ch/websvn/repos/wsvn/projects.bern2.5d/?op=log&rev=0&isdir=1 or
```
svn log
```

Check out version 1.0, original code from Kasper Plattner, compiles with gfortran, pgf90, ifort
```
svn co https://svn.iac.ethz.ch/repos/projects/bern2.5d/tags/1.0 bern2.5d-1.0
```

Check out original code from Kasper Plattner

svn co https://svn.iac.ethz.ch/repos/projects/bern2.5d/tags/original_plattner_code bern2.5d-original

svn export https://svn.iac.ethz.ch/repos/projects/bern2.5d/tar/bern2.5d_linux_knutti.tar .

For more information about how to work with subversion and the svn command see ServiceSvn.

Getting started

Checkout latest version

svn co https://svn.iac.ethz.ch/repos/projects/bern2.5d/trunk bern2.5d-trunk

Read the provided readme files

cd bern2.5d-trunk
less README.txt
less COMPILE.txt
less MACHINES.txt

Compile

See also COMPILE.txt and MACHINES.txt

Define the following environmental variables. This can be for example done by loading modules, see below. Or with the export (bash) or setenv (tcsh) command.
- FC : Fortran Compiler
- NETCDF : path to your netcdf installation, libnetcdf.a should be in $NETCDF/lib, include files in $NETCDF/include
- FFLAGS : additional compiler flags, for example set FFLAGS=-g for debugging

On IAC systems you can use modules to set these variables correctly

Environment for pgf90
```
module load pgi netcdf/3.6.3-pgf90
```

Environment for ifort
```
module load ifort netcdf/3.6.3-ifort
```

Environment for gfortran
```
module load gfortran netcdf/3.6.3
```

Compile
```
make clean
make
```

Scripts

List of scripts

Scriptname	Purpose
thc.sh	Run the Bern 2.5 D model
thc_run.sh	Run a spinup run or transient run
create_case.sh	Create a spinup or transient case
create_restart_file.sh	Create a restart file
list_files.sh	List thc related files
edit_files.sh	Edit and show variables in spinup and transient startfiles
crestart.sh	Simple wrapper script to run crestart (obsolete)

All scripts come with an internal help, just run the script with option -h or -help, for example
```
./list_files.sh -h
```

Run test case

Run the test case model:
```
./thc.sh -s test
```

Run a spinup case

Run default susciar4 spinup model :
```
./thc.sh -s susciar4_default
```
Note: susciar4_default is equal to susciar4_knum1_iadv1

Run a transient case

Run transient model (the spinup run has to be done in advance):
```
./thc.sh -r susciar4_default -t ar4_sres_a2_ar4_default
```
In this case the transient run (-t) ar4_sres_a2_ar4_default is restarted (-r) from the spinup run susciar4_default

Run a spinup run followed by a transient run

Do a spinup run followed by a transient run

./thc.sh -s susciar4_default -t ar4_sres_a2_ar4_default

Modify startfiles

See edit_files.sh script
```
./edit_files.sh -h
```

Run thc with modified startfiles

You can modify the startfiles within the thc.sh command
```
./thc.sh -set kvnum=1,khnum=1,iadv=1 -s susciar4
```
This will set kvnum=1, khnum=1 and iadv=1. Note: This is equal to
```
./thc.sh -s susciar4_default
```

Run a transient thc with on other forcing file

Place your forcing file in the folder forcing
The name and path of the forcing file for a transient run is defined in the transient startup file in a line similar to
```
'forcing/start_ar4_sres_a2_ar4.dat'                      forcing (chfile_indus)
```
Use option -set to change the name of the forcing file. IMPORTANT: Use a backslash \ to escape the ' sign !
```
./thc.sh -set forcing="\'forcing/file1.dat\'" -r susciar4_default -t ar4_sres_a2_ar4_default
```

How the option -set defines the name of the startfile and the case name

If you modify the startfiles with the option -set, a MD5-string will be appended to the name of startfile and therefore to the name of the new case.
The MD5-string is calculate from the parameters given by the -set option. Run the script ./md5.sh to find out the MD5 sum. For example
```
./md5.sh kvnum=1,khnum=1,iadv=1
```

Note that the MD5 sum is independent of the order of the parameters

./md5.sh kvnum=1,khnum=1,iadv=1
MD5c3478ce6723a70b7fb3a4c2e57c97737
./md5.sh iadv=1,kvnum=1,khnum=1
MD5c3478ce6723a70b7fb3a4c2e57c97737

If you don't want to choose the string which is appended, use the option -add, for example
```
./thc.sh -set kvnum=1,khnum=1,iadv=1 -add setup1 -s susciar4
```
This will create the case susciar4_setup1.spinup instead of susciar4_MD5c3478ce6723a70b7fb3a4c2e57c97737.spinup

More options

If you want to re-do a run, use the option -f (force) to overwrite everything
```
./thc.sh -f -s susciar4_default
```
Run thc with nice 19 (-n 19)
```
./thc.sh -n 19 -s susciar4_default
```
Run thc quiet (-q) - with no output
```
./thc.sh -q -s susciar4_default
```
See all the available option of thc.sh by running
```
./thc.sh -h
```

Create only the case directory

Create case directory for a spinup case
```
./create_case.sh -s susciar4_default
```

Create case directory for a transient case

./create_case.sh -r susciar4_default -t ar4_sres_a2_ar4_default

List thc related files

List available spinup start files
```
./list_files.sh -ls
```

List available restart files (-lr) and available transient (-lt) start files
```
./list_files.sh -lr -lt
```

Run thc on brutus batch system

Compile thc, see also MACHINES.txt

module purge
module load pgi netcdf/4.0.1                  # in case you want to use Portland compiler
module load intel netcdf/4.0.1                # in case you want to use Intel compiler
export FC=gfortran; module load netcdf/4.0.1  # in case you want to use GNU gfortran compiler 
make clean
make

Create a case, for example for spinup susciar4_default
```
./create_case.sh -s susciar4_default
```

Submit batch job

bsub < ./cases/susciar4_default.spinup/susciar4_default.spinup.lsf

Afterwards LSF log files are in the case folder, for example

cases/susciar4_default.spinup/susciar4_default.spinup-out.JOBID
cases/susciar4_default.spinup/susciar4_default.spinup-err.JOBID

Run several thc in parallel on brutus

Use the LSF jobfile par_thc.lsf and the script par_thc.sh to distribute thc-runs over several nodes. For more info see ProjectBern25Dparallel

Setup for a crash

Run spinup case susciar4
```
./thc.sh -s susciar4
```

1. Case: Run transient case sres_a2_BernCC_targwfb2.5progipccar4jan09

./thc.sh -r susciar4 -t sres_a2_BernCC_targwfb2.5progipccar4jan09

Time: 21021.2 yr      dt:   20.3 d     dta:   16.8 h
./thc.sh: line 167:  2139 Floating point exception$THC $ex_trans $ex_restart
ERROR: thc exit with an error.

2. Case: Run transient case sres_a2_test

./thc.sh -r susciar4 -t sres_a2_test

Time: 20326.2 yr      dt:   20.3 d     dta:   16.8 h
./thc.sh: line 167:  2241 Floating point exception$THC $ex_trans $ex_restart
ERROR: thc exit with an error.

3. Case: Run transient case ar4_sres_a2_ar4_gwfb_3.2

./thc.sh -r susciar4 -t ar4_sres_a2_ar4_gwfb_3.2

Time: 20310.2 yr      dt:   20.3 d     dta:   16.8 h
./thc.sh: line 169: 25445 Floating point exception$THC $ex_trans $ex_restart
ERROR: thc exit with an error.

The crash can be avoided, if you start with spinup susciar4_iadv1, susciar4_knum1 or susciar4_knum1_iadv1 instead of susciar4 !

Crash Matrix

Spinup	Transient	Runs without crash
susciar4	sres_a2_BernCC_targwfb2.5progipccar4jan09
susciar4	sres_a2_test
susciar4	ar4_sres_a2_ar4_gwfb_3.2
susciar4_iadv1	sres_a2_BernCC_targwfb2.5progipccar4jan09
susciar4_iadv1	sres_a2_test
susciar4_iadv1	ar4_sres_a2_ar4_gwfb_3.2
susciar4_knum1	sres_a2_BernCC_targwfb2.5progipccar4jan09
susciar4_knum1	sres_a2_test
susciar4_knum1	ar4_sres_a2_ar4_gwfb_3.2
susciar4_knum1_iadv1	sres_a2_BernCC_targwfb2.5progipccar4jan09
susciar4_knum1_iadv1	sres_a2_test
susciar4_knum1_iadv1	ar4_sres_a2_ar4_gwfb_3.2

Benchmarks

Summary: Best performance reached with intel compiler

Single thc run. Minimum time of

time ./thc.sh -f -s susciar4 >/dev/null

Compiler	Flags	System	Time
gfortran 4.8.1	-O3 -ffpe-trap=invalid,zero,overflow -fno-automatic	rasperypi	270m 28s
ifort 10.1	-O3 -fpe0	firebolt	18m 26s
pgf90 7.2-3	-O3 -Ktrap=fp	firebolt	20m 10s
gfortran 4.1.2	-O3 -ffpe-trap=invalid,zero,overflow -fno-automatic	firebolt	21m 39s
ifort 10.1	-O3 -fpe0	iacdipl-2	11m 50s
pgf90 7.2-3	-O3 -Ktrap=fp	iacdipl-2	16m 19s
gfortran 4.3.2	-O3 -ffpe-trap=invalid,zero,overflow -fno-automatic	iacdipl-2	15m 09s
ifort 10.1	-O3 -fpe0	fluffy	8m 19s
pgf90 7.2-3	-O3 -Ktrap=fp	fluffy	9m 25s
gfortran 4.3.2	-O3 -ffpe-trap=invalid,zero,overflow -fno-automatic	fluffy	11m 31s
pgf90 9.0-1	-O3 -Ktrap=fp	brutus3 (login node)	14m 59s
ifort 10.1.018	-O3 -fpe0	brutus3 (login node)	10m 01s
ifort 10.1	-O3 -fpe0	Xeon X5690 3.47GHz (atmos)	6m 02s
ifort 10.1	-O3 -fpe0	Xeon E5-2690 2.90GHz (kryo)	6m 30s
ifort 10.1	-O3 -fpe0	Xeon E3-1275 3.50 GHz	4m 08s
ifort 10.1	-O3 -fpe0	i7-2600 3.40GHz	4m 56s
ifort 10.1	-O3 -fpe0	i5-3570 3.40GHz	4m 15s
ifort 13.1.3	-O3 -fpe0	Xeon E5-2697 v2 2.70GHz	5m 32s
ifort 13.1.3	-O3 -fpe0	Xeon E5-2680 v2 2.80GHz	4m 23s
ifort 13.1.3	-O3 -fpe0	Xeon E5-2670 v2 2.50GHz	4m 46s
ifort 13.1.3	-O3 -fpe0	Xeon E5-2660 v2 2.20GHz	5m 17s
ifort 13.1.3	-O3 -fpe0	Xeon E5-2650 v2 2.60GHz	4m 38s
ifort 10.1	-O3 -fpe0	Xeon E3-1245 v3 3.40GHz (FS W540)	3m 57s

Benchmarks of 400 parallel runs

./thc.sh -set iadv=1 -add iadv_1 -s susciar4
for i in $( seq 0 399 ); do 
   var=$( printf "%03d\n" $i) 
   echo "./thc.sh -f -set gwfb=2.$var -r susciar4_iadv_1 -t ar4_sres_a2_ar4"
done > joblist; echo END >> joblist
time ./par_thc.sh -g 2

Compiler	Flags	System	cores	Time	*Time core**
ifort 10.1	-O3 -fpe0	fluffy	8	8m 27s	68m
ifort 10.1	-O3 -fpe0	iacdipl-2	16	6m 47s	109m
ifort 10.1	-O3 -fpe0	firebolt	8	21m 0s	168m

Directory Layout

|-- cdf_2d                    (source code of cdf_2d.a)
|-- src                       (source code of thc)
|-- thc.sh                    (script to run thc *** main script ***)
|-- thc_run.sh                (script to run spinup and transient cases)
|-- create_case.sh            (script to create a new case)
|-- create_restart_file.sh    (script to create a restart file)
|-- list_files.sh             (script to list thc related files)
|-- edit_files.sh             (edit/show starfiles)
|-- crestart.sh               (points to old src/crestart)
|-- data                      (directory of data files, ???)
|-- forcing                   (directory of data files, forcing)
|-- start_files               (directory of start files)
|-- cases                     (directory containing runs/cases)
|   |-- susciar4.spinup        (case directory for susciar4 case, spinup run)
|   |   |-- input               (input date)
|   |   `-- output              (output data)
|   |-- susciar4.sres_a2_test  (case directory for transient run sres_a2_test,
|   ...                         spinup from susciar4 run)
|
|-- batch-template.lsf        (template for batch job script)
|-- COMPILE.txt               (read how to compile the model)
|-- MACHINES.txt              (machine specific documentation)
|-- README.txt                (general readme file)
`-- Makefile                  (main Makefile)

Debugging the model

For debugging pgf90 compiled code, use Portland Group Debugger, see /usr/local/pgi/linux86/7.2-3/doc/pgi72tools.pdf
or http://www.pgroup.com/doc/pgitools.pdf

Load the compiler and corresponding the netcdf library, if not yet done
```
module load pgi netcdf/3.6.3-pgf90
```

Compile with option -g
```
FFLAGS="-g"
make clean
make
```

Run thc inside pgdbg
```
pgdbg src/thc 
```

Press Run inside the pgdbg gui

(old description for original code) Compile and run on firebolt

ProjectBern25Dold.

Access

Set DENYTOPICVIEW =