Repository navigation
Expand file tree
/
Copy pathREADME.IRIX
More file actions
118 lines (71 loc) · 3.28 KB
/
Copy pathREADME.IRIX
File metadata and controls
118 lines (71 loc) · 3.28 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
Running Phred on IRIX
---------------------
Read the MPI and OpenMP man pages! There is lots of important info
there!
http://techpubs.sgi.com/library/tpl/cgi-bin/browse.cgi?coll=linux&db=bks&cmd=toc&pth=/SGI_Developer/MPT_UG
Some interesting man pages:
numa(5), migration(5), mtune(4), /var/sysgen/mtune/numa, refcnt(5),
replication(5), nstats(1), sn(1), mld(3c), mldset(3c), pm(3c),
migration(3c), pminfo(3c), numa_view(1), dplace(1), dprof(1).
Also, see mmci(5).
* Checkpointing
Pass the -cpr or -miser flag to to mpirun in the PBS script. The cpr
command should make it possible to checkpoint an restore the program.
* Debugging
Set MPI_COREDUMP_VERBOSE=1
* Performance testing / profiling
Run an experiment using the ssrun command
ssrun -workshop -usertime my_prog
cvperf my_prog.usertime.m88888
* MPI and OpenMP
Use a CPU list?
Use MPI_DSM_PLACEMENT... firsttouch or threadroundrobin? Firsttouch
caused an error on one run.
MPI_DSM_TOPOLOGY=cube?
MPI_DSM_VERBOSE prints info about numa process placement
MPI_DSM_VERIFY makes some checks...
MPI_OPENMP_INTEROP... see "Using MPI with OpenMP"... set this and
compile with -mp, and link such that the libmpi.so init code runs
before the libmp.so init code: CC -64 -mp compute_mp.C -lmp -lmpi++ -lmpi
MPI_DSM_INTEROP
MPI_DSM_PLACEMENT=threadroundrobin
See the man page for omp and mpi.
- Use a multiple of four omp threads per MPI process
cpusets? MPI_DSM_OFF?
OpenMP options:
_DSM_MIGRATION=ON to migrate pages around...
_DSM_VERBOSE
export OMP_NUM_THREADS=4
export MPI_DSM_TOPOLOGY=cube
export MPI_DSM_VEBOSE=1
export MPI_OPENMP_INTEROP=1
export _DSM_MIGRATION=ON
export _DSM_VERBOSE=1
----------------------------------------------------------------------
-- DEBUGGING USING THE PRODEV WORKSHOP DEBUGGER
----------------------------------------------------------------------
See: http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=bks&fname=/SGI_Developer/books/MPT_MPI_PM/sgi_html/ch04.html&srch=totalview%20mpi%20debug
1) $ cvd /usr/bin/mpirun
2) cvd> set $pendingtraps=true
cvd> stop pgrp all in MPI_SGI_init
3) In command window: /usr/bin/mpirun -np 2 phred problem.py
----------------------------------------------------------------------
-- PROFILING AND TIMING
----------------------------------------------------------------------
Arcturus should get about 4.15 MNPS per CPU:
Number of updated nodes: 1000000, millions of updated nodes: 1
Average wall time: 0.241, avg cpu time: 0.2408
Average millions of nodes per second, w.r.t. wall clock time: 4.14938
Average millions of nodes per second, w.r.t. CPU time: 4.15282
Note: MNPS w.r.t. CPU time may be incorrect.
Execution took 241 wall clock seconds, and 240.98 cpu seconds.
So 32 MNPS with 64 CPUs is rather disappointing. 256 MNPS would be
nicer.
----------------------------------------------------------------------
-- ENABLING SINGLE COPY
----------------------------------------------------------------------
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi/linux/bks/SGI_Developer/books/MPT_UG/sgi_html/ch06.html#Z1028132313jlb
"For MPI_Isend, MPI_Sendrecv, MPI_Alltoall, MPI_Bcast, MPI_Allreduce ,
and MPI_Reduce, this optimization is enabled by default for large
message sizes."
Since MPI_Isend is now being used, this may work fine.