CCSM 3.5 Benchmarks Results for 1.9x2.5

Summary

Comping and Running

Working Matrix (compiling / running)

Compiler/MPI	pgi/7.2-5	pgi/8.0-5	pgi/8.0-6	pgi/9.0-1	intel/10.1.018	intel/11.0.081	sun_studio/12.1
quadrics_mpi/stable	/	? / ?	? / ?	? / ?	/	? / ?	-
open_mpi/1.2.7	/	? / ?	? / ?	/	/	? / ?	-
open_mpi/1.2.9	/	? / ?	? / ?	/	/	? / ?	-
open_mpi/1.3.2	/	/	? / ?	/	/	/	-
open_mpi/1.3.3	/	? / ?	? / ?	? / ?	/	? / ?	/
mvapich2/1.4rc2	/	? / ?	? / ?	/	/	/	-
mvapich2/1.2p1	? / ?	? / ?	? / ?	? / ?	/	? / ?	-

Code compiles with most combinations of compilers and openmpi versions. It does not compile with sun_studio/12.1.
Intel 10.1.018 or pgi 7.2-5 with open_mpi/1.2.7 are working well.
Intel 10.1.018 or pgi 7.2-5 with mvapich2/1.4rc2 are working well.
Best performance with mvapich2/1.4rc2 for not more than 256 cores.
Using pgi 9.0 ccsm not running at all (with open_mpi 1.2.7 and 1.3.2)
Using intel/10.1.018 and open_mpi 1.2.9 ccsm is not running at all.
Many crashes with open_mpi 1.3.2 and 1.3.3.
More info see below -> Problem Section

Performance

Performance on infiniband nodes is as good or even better than on quadrics nodes.
Best performance for 256 cores or less shows intel/10.1.018 and mvapich2/1.4rc2.
Intel 10.1.018 and open_mpi/1.2.7 has best performance above 400 cores about 6-7% faster than pgi 7.2-5 / open_mpi/1.2.7
Good scaling until 400-500 cores on infiniband.
lower performance beyond 64 cores of pgi and open_mpi/1.3.2 at the moment.

Best performance on ib

cores	theoretical	intel 10.1.018 open_mpi/1.2.7	pgi 7.2-5 open_mpi/1.2.7	pgi 7.2-5 open_mpi/1.2.9	pgi 7.2-5 mvapich2/1.4rc2	intel/10.1.018 mvapich2/1.4rc2	pgi 7.2-5 open_mpi/1.3.3
64	5	5.0	4.5	4.5	4.5	5.1
128	10	10.0	8.1	8.2	9.3	10.5
256	20	19.0	16.5	17.2	19.8	21.6
408	31.9	28.9	27.0	26.2	23.9	23.8	19.9

Best performance on ib-beta

cores	theoretical	intel 10.1.018 quadrics_mpi/stable	intel 10.1.018 open_mpi/1.2.7	pgi 7.2-5 open_mpi/1.2.7	pgi 7.2-5 open_mpi/1.3.2	pgi 8.0-5 open_mpi/1.3.2	intel/10.1.018 open_mpi/1.3.2	intel/11.0.081 open_mpi/1.3.2
64	5	5.10	5.42	5.81	4.90	4.72
128	10	10.42	9.76	9.73	7.84	5.77	7.72	7.79
256	20	17.79	19.89	19.41		6.39	16.07	16.17
408	31.9		30.25	28.17
512			31.60	29.78	25.86			27.50

Numbers are simulated years per day.

1.9x2.5_gx1v5 Benchmarks on brutus ib

See attached OpenOffice? Calc file: benchmarks_1.9x2.5_gx1v5-B.ods

1.9x2.5_gx1v5 Benchmarks on brutus ib-beta

mach	compiler	mpi	pes	cores	#runs	avg	best
quadrics2c	intel 10.1.018	quadrics_mpi/stable	24/4/4/24/8	64	6	4.88	5.10
quadrics2c	intel 10.1.018	quadrics_mpi/stable	48/8/8/48/16	128	7	10.03	10.42
quadrics2c	intel 10.1.018	quadrics_mpi/stable	96/16/16/96/32	256	6	17.41	17.79
ib-beta	intel 10.1.018	open_mpi/1.2.7	24/4/4/24/8	64	22	4.89	5.42
ib-beta	intel 10.1.018	open_mpi/1.2.7	48/8/8/48/16	128	5	9.22	9.76
ib-beta	intel 10.1.018	open_mpi/1.2.7	96/16/16/96/32	256	16	18.34	19.89
ib-beta	intel 10.1.018	open_mpi/1.2.7	160/24/16/160/48	408	10	28.80	30.25
ib-beta	intel 10.1.018	open_mpi/1.2.7	192/32/32/192/64	512	11	31.21	31.60
ib-beta	intel 10.1.018	open_mpi/1.2.7	192/32/64/192/64	544	2	31.38	31.68
ib-beta	intel 10.1.018	open_mpi/1.3.2	24/4/4/24/8	64	- crash
ib-beta	intel 10.1.018	open_mpi/1.3.2	48/8/8/48/16	128	2 crash 5 ok	6.87	7.72
ib-beta	intel 10.1.018	open_mpi/1.3.2	96/16/16/96/32	256	2 crash 2 ok	15.89	16.07
ib-beta	intel 10.1.018	open_mpi/1.3.2	192/32/32/192/64	512	- crash
ib-beta	intel 11.0.081	open_mpi/1.3.2	24/4/4/24/8	64	- crash
ib-beta	intel 11.0.081	open_mpi/1.3.2	48/8/8/48/16	128	9	6.91	7.79
ib-beta	intel 11.0.081	open_mpi/1.3.2	96/16/16/96/32	256	9 crash 2 ok	16.03	16.17
ib-beta	intel 11.0.081	open_mpi/1.3.2	192/32/32/192/64	512	1 crash 5 ok	26.86	27.50
ib-beta	pgi 7.2-5	open_mpi/1.2.7	24/4/4/24/8	64	7	5.38	5.81
ib-beta	pgi 7.2-5	open_mpi/1.2.7	48/8/8/48/16	128	3	9.45	9.73
ib-beta	pgi 7.2-5	open_mpi/1.2.7	96/16/16/96/32	256	11	18.42	19.41
ib-beta	pgi 7.2-5	open_mpi/1.2.7	160/24/16/160/48	408	7	27.64	28.17
ib-beta	pgi 7.2-5	open_mpi/1.2.7	192/32/32/192/64	512	11	29.41	29.78
ib-beta	pgi 7.2-5	open_mpi/1.3.2	24/4/4/24/8	64	7	4.33	4.90
ib-beta	pgi 7.2-5	open_mpi/1.3.2	48/8/8/48/16	128	2 crash 5 ok	7.01	7.84
ib-beta	pgi 7.2-5	open_mpi/1.3.2	96/16/16/96/32	256	- crash
ib-beta	pgi 7.2-5	open_mpi/1.3.2	192/32/32/192/64	512	7	24.99	25.86
ib-beta	pgi 8.0-6	open_mpi/1.3.2	24/4/4/24/8	64	- crash
ib-beta	pgi 8.0-5	open_mpi/1.3.2	24/4/4/24/8	64	- crash
ib-beta	pgi 9.0-1	open_mpi/1.2.7	24/4/4/24/8	64	- crash
ib-beta	pgi 9.0-1	open_mpi/1.3.2	24/4/4/24/8	64	- crash
ib-beta	pgi 9.0-1	open_mpi/1.3.2	48/8/8/48/16	128	- crash
ib-beta	pgi 9.0-1	open_mpi/1.3.2	96/16/16/96/32	256	- crash
ib-beta	pgi 9.0-1	open_mpi/1.3.2	192/32/32/192/64	512	- crash

Old results (before 3. July)

ib-beta	pgi 7.2-5	open_mpi/1.3.2	24/4/4/24/8	64	1	4.90	4.90
ib-beta	pgi 7.2-5	open_mpi/1.3.2	48/8/8/48/16	128	5	5.00	5.64
ib-beta	pgi 8.0-5	open_mpi/1.3.2	24/4/4/24/8	64	1	4.72	4.72
ib-beta	pgi 8.0-5	open_mpi/1.3.2	48/8/8/48/16	128	6	4.82	5.77
ib-beta	pgi 8.0-5	open_mpi/1.3.2	96/16/16/96/32	256	2	6.38	6.39

mach:
- quadrics2c: 2x Dual-Core AMD Opteron 2220, 16GB memory
- ib-beta: 4 x Quad-Core AMD Opteron 8380, 32GB memory
pes : tasks per component settings: atm/ln/ice/ocn/cpl
#runs: number of benchmark runs
avg: average time for simulated years / day
best: best time for simulated years / day

Problems

CCSM compiled with pgi 9.0 does not run

with pgi 9.0 the model is crashing at the second day (0001-01-02) of the calculation. I see the following in the logs. For example

# cat /cluster/home/uwis/beyerleu/ccsm35.b24-trunk/scripts/1.9x2.5_gx1v5-B-bench-pm-1.4rc2-9.0-1/out.845161
...
(tStamp_write) cpl  model date 0001-01-02 00000s  wall clock 2009-09-09 14:22:41  avg dt    32s  dt    32s
rank 37 in job 1  a6513.hpc-net.ethz.ch_57590   caused collective abort of all ranks
  exit status of rank 37: killed by signal 9
rank 26 in job 1  a6513.hpc-net.ethz.ch_57590   caused collective abort of all ranks
  exit status of rank 26: killed by signal 9
rank 17 in job 1  a6513.hpc-net.ethz.ch_57590   caused collective abort of all ranks
  exit status of rank 17: killed by signal 9

cat /cluster/home/uwis/beyerleu/ccsm35.b24-trunk/scripts/1.9x2.5_gx1v5-B-bench-pm-1.4rc2-9.0-1/err.845161
...
Fatal error in MPI_Abort:
Invalid communicator, error stack:
MPI_Abort(140): MPI_Abort(comm=0x0, errorcode=20342561) failed
MPI_Abort(79).: Invalid communicator
...

The same happens openmpi 1.x. and MVAPICH2

Many Crash open_mpi 1.3.3 and 1.3.2

Simulation cashes the model third day (on 64 cores) using intel 10.1.018 and open_mpi 1.3.3

The file

/cluster/work/uwis/beyerleu/1.9x2.5_gx1v5-B-bench-io-1.3.3_new-debug-10.1.018/ocn/hist/1.9x2.5_gx1v5-B-bench-io-1.3.3_new-debug-10.1.018.pop.dv.0001-01-03-00000

is not found - Why???

(tStamp_write) cpl  model date 0001-01-03 00000s  wall clock 2009-09-10 16:42:07  avg dt    44s  dt
   55s
******** Thu Sep 10 16:43:12 CEST 2009 -- CSM EXECUTION HAS FINISHED
forrtl: No such file or directory
forrtl: severe (29): file not found, unit 7, file /cluster/work/uwis/beyerleu/1.9x2.5_gx1v5-B-bench-io-1.3.3_new-debug-10.1.018/ocn/hist/1.9x2.5_gx1v5-B-bench-io-1.3.3_new-debug-10.1.018.pop.dv.0001-01-03-00000

 Image              PC                Routine            Line        Source
ccsm_se.exe        000000000116E85E  Unknown               Unknown  Unknown
ccsm_se.exe        000000000116D7F0  Unknown               Unknown  Unknown
ccsm_se.exe        00000000011257E6  Unknown               Unknown  Unknown
ccsm_se.exe        00000000010C7D97  Unknown               Unknown  Unknown
ccsm_se.exe        00000000010C7674  Unknown               Unknown  Unknown
ccsm_se.exe        00000000010DAB4D  Unknown               Unknown  Unknown
ccsm_se.exe        0000000000773496  diagnostics_mp_di        2632  diagnostics.F90
ccsm_se.exe        0000000000866469  step_mod_mp_step_         596  step_mod.F90
ccsm_se.exe        00000000007414CE  ccsm_ocn_                  92  POP.F90
ccsm_se.exe        000000000049FFED  MAIN__                     77  se_master.F90
ccsm_se.exe        000000000049F842  Unknown               Unknown  Unknown
libc.so.6          00002AE40B8F6974  Unknown               Unknown  Unknown
ccsm_se.exe        000000000049F769  Unknown               Unknown  Unknown
--------------------------------------------------------------------------
mpirun has exited due to process rank 16 with PID 3785 on
node a6511 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

Simulation crashes with pgi 7.2-5 open_mpi 1.3.3 - same problem with pgi.

(tStamp_write) cpl  model date 0001-01-03 00000s  wall clock 2009-09-11 09:24:00  avg dt    49s  dt    65s
******** Fri Sep 11 09:25:11 CEST 2009 -- CSM EXECUTION HAS FINISHED

 In source file /cluster/work/uwis/beyerleu/1.9x2.5_gx1v5-B-bench-po-1.3.3_new-7.2-5/ocn/obj/source/diagnostics.F90, at line number 2632
--------------------------------------------------------------------------
mpirun has exited due to process rank 16 with PID 4943 on
node a6434 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

ocn/obj/source/diagnostics.F90, at line number 2632

     if ( my_task == master_task ) then

       !*** append velocity output to end of velocity output file

       open (velocity_unit, file=diag_velocity_outfile, status='old', &    <------- line 2632
            position='append')

Simulation runs and crashes with

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 167 in communicator MPI_COMM_WORLD
with errorcode 19845729.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

--------------------------------------------------------------------------
mpirun has exited due to process rank 16 with PID 17581 on
node a6364 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

Some crashes with openmpi

With openmpi 1.2.7 and pgi 7.2-5 on 256cores: (but this is not reproducible)

print_memusage iam   0 post-inidat. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack=  285006   40129    2147    4181       0
print_memusage iam   0 Start aerosol_initialize. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack=  285006   40157    2171    4181       0
print_memusage iam   0 End aerosol_initialize. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack=  285843   41030    2192    4181       0
print_memusage iam   0 After second phase of CAM run. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack=  288690   52535    3219    4181       0
print_memusage iam   0 stepon after physics and dynamics. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack=  288690   52535    3219    4181       0

Poor performance with pgi and open_mpi/1.3.2 at the moment.

Olivier Byrde: Open MPI 1.3.2 has only been tested with GNU compilers so far. The Intel and PGI versions are available but still need to be tested. They may cause unexpected problems or lead to poor performance.

SOLVED - Intel 11.0 not yet running

get compile errors at the very beginning:

/cluster/home/uwis/beyerleu/ccsm35.b24-trunk/models/cpl/cpl6/diag_mod.F90(126): error #5082: Syntax error, found
IDENTIFIER 'CHARACTER' when expecting one of: ( * ) :: , <END-OF-STATEMEN
T> ; + . - (/ [ : ] /) ' ** / // > ...
  &       '      lwup   lat-vap   lat-ice       sen       net W/m2')"   character(*),parameter :: F11="('&
...

Solution: Remove -132 flag from FFLAGS and set FIXEDFLAGS := -132

And remove -FR flag or unset FREEFLAGS for brutus_io

svn diff Machines/Macros.Linux.ia64.brutus_io
...
+FFLAGS     := -c -DLINUX -fp-model precise -O2 -convert big_endian -assume byterecl -ftz
+FREEFLAGS  :=
+FIXEDFLAGS := -132
...

SOLVED - Intel 10.1.018 and open_mpi/1.2.7 on 408 cores crashes

Get a lot of errors at the beginning of calculation:

[a6393.hpc-net.ethz.ch:26768] MPI_ABORT invoked on rank 259 in communicator MPI_COMM_WORLD with errorcode 1
[a6393.hpc-net.ethz.ch:26766] MPI_ABORT invoked on rank 255 in communicator MPI_COMM_WORLD with errorcode 1
[a6393.hpc-net.ethz.ch:26759] MPI_ABORT invoked on rank 257 in communicator MPI_COMM_WORLD with errorcode 1
...

This is solved now (since 3. July) - why? reason unknown

1.9x2.5_gx1v5 Benchmarks on brutus for 64 cores

Run	type	atm	lnd	ocn	ice	cpl	total	pe-hrs/simulated_year	simulated_years/day
1	iq	24	4	20	8	8	64	445	3.6
2	iq	24	4	24	4	8	64	353	4.4
3	iq	24	2	24	4	10	64	406	3.8
4	po	24	2	24	4	10	64	297	5.2
4	po	24	2	24	4	10	64	292	5.3
4	po	24	2	24	4	10	64	314	4.9
4	po	24	2	24	4	10	64	297	5.2
4	po	24	2	24	4	10	64	340	4.5
4	po	24	2	24	4	10	64	314	4.9
4	po	24	2	24	4	10	64	261	5.9
4	po	24	2	24	4	10	64	264	5.8
5	po	24	4	24	4	8	64	354	4.3
5	po	24	4	24	4	8	64	383	4.0
6	io	24	2	24	4	10	64	390	3.9
6	io	24	2	24	4	10	64	380	4.1
6	io	24	2	24	4	10	64	315	4.9
6	io	24	2	24	4	10	64	307	5.0
6	io	24	2	24	4	10	64	313	4.9
7	io	24	4	24	4	8	64	309	5.0
7	io	24	4	24	4	8	64
8	io	12	1	12	2	5	32	331	2.3

io = intel and openmpi (on infiniband)
iq = intel and quadrics2c
po = portland and openmpi (on infiniband)
pq = portland and quadrics2c

Attach

Attachments Attachments

benchmarks_1.9x2.5_gx1v5-B.ods (20.30K)

benchmarks 1.9x2.5_gx1v5-B on brutus (Sept. 2009)

version 1 uploaded by UrsBeyerle on 11 Sep 2009 - 14:03

Run	type	atm	lnd	ocn	ice	cpl	total	pe-hrs/simulated_year	simulated_years/day
1	iq	24	4	20	8	8	64	445	3.6
2	iq	24	4	24	4	8	64	353	4.4
3	iq	24	2	24	4	10	64	406	3.8
4	po	24	2	24	4	10	64	297	5.2
4	po	24	2	24	4	10	64	292	5.3
4	po	24	2	24	4	10	64	314	4.9
4	po	24	2	24	4	10	64	297	5.2
4	po	24	2	24	4	10	64	340	4.5
4	po	24	2	24	4	10	64	314	4.9
4	po	24	2	24	4	10	64	261	5.9
4	po	24	2	24	4	10	64	264	5.8
5	po	24	4	24	4	8	64	354	4.3
5	po	24	4	24	4	8	64	383	4.0
6	io	24	2	24	4	10	64	390	3.9
6	io	24	2	24	4	10	64	380	4.1
6	io	24	2	24	4	10	64	315	4.9
6	io	24	2	24	4	10	64	307	5.0
6	io	24	2	24	4	10	64	313	4.9
7	io	24	4	24	4	8	64	309	5.0
7	io	24	4	24	4	8	64
8	io	12	1	12	2	5	32	331	2.3

Run	type	atm	lnd	ocn	ice	cpl	total	pe-hrs/simulated_year	simulated_years/day
1	iq	24	4	20	8	8	64	445	3.6
2	iq	24	4	24	4	8	64	353	4.4
3	iq	24	2	24	4	10	64	406	3.8
4	po	24	2	24	4	10	64	297	5.2
4	po	24	2	24	4	10	64	292	5.3
4	po	24	2	24	4	10	64	314	4.9
4	po	24	2	24	4	10	64	297	5.2
4	po	24	2	24	4	10	64	340	4.5
4	po	24	2	24	4	10	64	314	4.9
4	po	24	2	24	4	10	64	261	5.9
4	po	24	2	24	4	10	64	264	5.8
5	po	24	4	24	4	8	64	354	4.3
5	po	24	4	24	4	8	64	383	4.0
6	io	24	2	24	4	10	64	390	3.9
6	io	24	2	24	4	10	64	380	4.1
6	io	24	2	24	4	10	64	315	4.9
6	io	24	2	24	4	10	64	307	5.0
6	io	24	2	24	4	10	64	313	4.9
7	io	24	4	24	4	8	64	309	5.0
7	io	24	4	24	4	8	64
8	io	12	1	12	2	5	32	331	2.3

Run	type	atm	lnd	ocn	ice	cpl	total	pe-hrs/simulated_year	simulated_years/day
1	iq	24	4	20	8	8	64	445	3.6
2	iq	24	4	24	4	8	64	353	4.4
3	iq	24	2	24	4	10	64	406	3.8
4	po	24	2	24	4	10	64	297	5.2
4	po	24	2	24	4	10	64	292	5.3
4	po	24	2	24	4	10	64	314	4.9
4	po	24	2	24	4	10	64	297	5.2
4	po	24	2	24	4	10	64	340	4.5
4	po	24	2	24	4	10	64	314	4.9
4	po	24	2	24	4	10	64	261	5.9
4	po	24	2	24	4	10	64	264	5.8
5	po	24	4	24	4	8	64	354	4.3
5	po	24	4	24	4	8	64	383	4.0
6	io	24	2	24	4	10	64	390	3.9
6	io	24	2	24	4	10	64	380	4.1
6	io	24	2	24	4	10	64	315	4.9
6	io	24	2	24	4	10	64	307	5.0
6	io	24	2	24	4	10	64	313	4.9
7	io	24	4	24	4	8	64	309	5.0
7	io	24	4	24	4	8	64
8	io	12	1	12	2	5	32	331	2.3