Reading Profile files in profile.* NODE 0;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 7:50.146 7:50.146 1 0 470146225 caflaunch$MAIN$blk 100.0 14 7:50.160 1 5 470160925 .TAU application 0.0 0.013 0.013 1 0 13 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0 0 1 0 0 CI0000$cgca_m3pfem_ NODE 1;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 71.3 5:34.283 5:34.283 1.12021E+07 0 30 MPI_Win_unlock() 99.6 39,983 7:46.949 1 503146 466949368 xx14std 8.2 38,512 38,512 17279 0 2229 MPI_Barrier() 73.3 31,104 5:43.920 511 1.12912E+07 673034 cgca_hxi 50.0 6,721 3:54.250 5 276849 46850085 cgca_clvgp_nocosum 1.2 2,785 5,479 3071 36852 1784 gather 0.5 2,259 2,259 15338 0 147 MPI_Allreduce() 0.4 2,065 2,065 6 0 344199 MPI_Send() 0.3 1,588 1,588 1 0 1588976 MPI_Init() 0.3 1,547 1,547 154726 0 10 MPI_Win_lock() [THROTTLED] 0.9 1,544 4,268 3071 33781 1390 scatter 0.2 868 868 7 0 124138 MPI_File_write_all() 0.2 811 811 12289 0 66 MPI_Recv() 2.4 686 11,487 5 20000 2297568 cgca_pfem_simg 0.1 649 649 135 0 4813 MPI_Bcast() 30.2 576 2:21.740 1 51948 141740535 cgca_sld 0.1 419 419 12288 0 34 MPI_Isend() 0.4 358 1,729 9198 9198 188 dot_product_p 0.1 240 263 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 200 1,071 5 30000 214325 cgca_pfem_intcalc1 0.2 199 1,084 3066 6132 354 checon_par 100.0 174 7:48.902 1 479 468902175 .TAU application 0.0 169 169 12288 0 14 MPI_Probe() 0.0 164 164 100001 0 2 MPI_Get() [THROTTLED] 0.0 160 160 1 0 160120 MPI_Finalize() 0.0 110 110 155 0 711 cgca_dacf 0.0 77 77 100001 0 1 beemat [THROTTLED] 0.7 54 3,511 12284 12284 286 my_barrier 0.0 51 51 100001 0 1 deemat [THROTTLED] 0.0 50 50 7 0 7259 MPI_File_open() 0.0 48 48 6144 0 8 MPI_Waitall() 0.0 45 45 13 0 3538 MPI_Win_create() 0.0 44 44 2 0 22087 cgca_gbs 0.0 42 42 100001 0 0 shape_der [THROTTLED] 0.0 34 34 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 32 32 100001 0 0 invert [THROTTLED] 0.0 26 26 10073 0 3 MPI_Put() 0.0 23 226 155 1726 1463 cgca_gcupdn 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 22 22 80000 0 0 determinant 0.0 21 21 67 0 326 MPI_Reduce() 0.0 21 22 1 1 22469 load 0.0 10 10 7 0 1441 MPI_File_close() 0.0 9 9 12288 0 1 MPI_Get_count() 0.0 9 9 7 0 1301 MPI_File_set_view() 0.0 8 8 1 0 8759 cgca_igb 0.0 7 46 1 160 46431 scatter_nodes 0.0 6 14 1 19 14143 make_ggl 0.0 5 5 1 19 5730 cgca_gcu 0.0 5 5 2000 0 3 find_g3 0.0 4 4 5 0 979 cgca_fv 0.0 4 4 1 0 4548 allocate_gather_scatter 0.0 4 38 1 576 38820 cgca_pfem_cenc 0.1 3 549 1 33 549532 read_g_coord_pp 0.0 3 3 12 0 285 MPI_Win_free() 0.4 2 1,734 7 525 247742 cgca_pswci 0.0 2 2 7081 0 0 cgca_gcr 0.0 0.719 0.719 1 0 719 getname 0.0 0.644 1 1 2 1585 cgca_as 0.0 0.59 0.59 1 0 590 MPI_Comm_dup() 0.0 0.429 7 1 374 7843 cgca_nr 0.0 0.343 0.343 1 0 343 cgca_pfem_salloc 0.0 0.309 0.734 1 9 734 scatter_noadd 0.2 0.299 1,056 1 64 1056029 cgca_ds 0.0 0.233 9 1 64 9163 cgca_drt 0.0 0.222 0.222 1 0 222 rearrange 0.0 0.205 4 1 65 4362 gcupd_alloc 0.0 0.2 5 1 377 5933 cgca_rt 0.0 0.187 0.187 14 0 13 MPI_Type_commit() 0.0 0.176 0.176 1 0 176 calc_nels_pp 0.0 0.122 0.122 5 0 24 cgca_pfem_uym 0.1 0.119 243 1 2 243137 read_g_num_pp 0.0 0.117 0.117 1 0 117 cgca_ins 0.0 0.099 1 5 5 210 sum_p 0.0 0.055 0.055 1 0 55 cgca_gdim 0.0 0.047 0.047 1 0 47 abaqus2sg 0.0 0.046 0.046 14 0 3 MPI_Type_create_subarray() 0.0 0.039 0.039 21 0 2 cgca_gcf 0.4 0.039 2,065 3 6 688411 dismsh_ensi_p 0.0 0.037 0.037 7 0 5 elap_time 0.0 0.036 0.036 1 0 36 cgca_pfem_sdalloc 0.0 0.033 0.033 2 0 16 MPI_Test() 0.0 0.031 0.031 5 0 6 sample 0.0 0.029 0.029 1 0 29 cgca_cadim 0.0 0.019 0.019 12 0 2 MPI_Win_fence() 0.0 0.016 0.016 1 0 16 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.016 0.016 9 0 2 MPI_Comm_size() 0.0 0.015 0.015 14 0 1 MPI_Type_free() 0.0 0.015 0.016 1 2 16 find_pe_procs 0.0 0.014 0.014 1 0 14 calc_neq_pp 0.0 0.013 0.013 1 0 13 cgca_pfem_ctalloc 0.0 0.012 3 1 2 3582 read_loads 0.0 0.009 0.009 10 0 1 cgca_agc 0.0 0.007 0.007 1 0 7 caflaunch$MAIN$blk 0.0 0.006 27 1 2 27996 cgca_art 0.0 0.006 0.006 9 0 1 cgca_dgc 0.0 0.006 3 1 3 3630 read_p121 0.0 0.005 0.005 9 0 1 MPI_Comm_rank() 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.003 0.003 1 0 3 cgca_pfem_integalloc 0.0 0.003 0.214 1 1 214 max_p 0.0 0.003 79 1 1 79609 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 1, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.195 2.665 3.955 0.8066 MPI-IO Write Bandwidth (MB/s) 7 5.195 2.665 3.955 0.8066 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.896E+04 6.592E+04 1.238E+04 Message size received from all nodes 1.229E+04 7.851E+04 4 6.589E+04 1.239E+04 Message size sent to all nodes 6150 7.851E+04 4 6.417E+04 1.437E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 3071 7.851E+04 7.851E+04 7.851E+04 0 Message size sent to node 0 : gather => MPI_Isend() 1 2.495E+04 2.495E+04 2.495E+04 0 Message size sent to node 0 : make_ggl => MPI_Isend() 3071 4.99E+04 4.99E+04 4.99E+04 0 Message size sent to node 0 : scatter => MPI_Isend() 1 4.99E+04 4.99E+04 4.99E+04 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 6144 7.733E+04 3.866E+04 6.762E+04 9710 Message size sent to node 2 3071 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 2 : gather => MPI_Isend() 1 3.866E+04 3.866E+04 3.866E+04 0 Message size sent to node 2 : make_ggl => MPI_Isend() 3071 7.733E+04 7.733E+04 7.733E+04 0 Message size sent to node 2 : scatter => MPI_Isend() 1 7.733E+04 7.733E+04 7.733E+04 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 2 7.733E+04 3.866E+04 5.8E+04 1.933E+04 [GROUP=MAX_MARKER] Message size sent to all nodes 1 3.866E+04 3.866E+04 3.866E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 7.733E+04 7.733E+04 7.733E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 2 7.851E+04 4.99E+04 6.421E+04 1.43E+04 [GROUP=MAX_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 1 7.851E+04 7.851E+04 7.851E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 1 4.99E+04 4.99E+04 4.99E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 1 7.733E+04 7.733E+04 7.733E+04 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 1 7.733E+04 7.733E+04 7.733E+04 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.926E+04 3.926E+04 3.926E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.926E+04 3.926E+04 3.926E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to node 0 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 2;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 58.4 4:33.780 4:33.780 1.12021E+07 0 24 MPI_Win_unlock() 25.7 2:00.519 2:00.519 17279 0 6975 MPI_Barrier() 99.6 39,013 7:46.949 1 503146 466949491 xx14std 57.2 13,726 4:28.136 511 1.12372E+07 524728 cgca_hxi 50.0 6,100 3:54.250 5 269768 46850042 cgca_clvgp_nocosum 1.2 2,554 5,454 3071 36852 1776 gather 0.5 2,418 2,418 15338 0 158 MPI_Allreduce() 0.4 2,097 2,097 6 0 349531 MPI_Send() 0.3 1,601 1,601 1 0 1601306 MPI_Init() 1.1 1,343 5,166 3071 33781 1682 scatter 0.2 922 922 12289 0 75 MPI_Recv() 0.2 869 869 7 0 124170 MPI_File_write_all() 0.1 642 642 135 0 4758 MPI_Bcast() 0.1 488 488 12288 0 40 MPI_Probe() 30.2 461 2:21.750 1 51201 141750250 cgca_sld 0.4 320 1,836 9198 9198 200 dot_product_p 0.1 253 253 12288 0 21 MPI_Isend() 0.1 225 248 100001 100001 2 cgca_clvgn [THROTTLED] 0.2 184 1,082 3066 6132 353 checon_par 100.0 161 7:48.902 1 564 468902204 .TAU application 0.0 160 160 1 0 160123 MPI_Finalize() 0.0 153 153 100001 0 2 MPI_Win_lock() [THROTTLED] 1.5 131 6,942 5 20000 1388456 cgca_pfem_simg 0.0 99 99 155 0 641 cgca_dacf 0.0 67 67 100001 0 1 beemat [THROTTLED] 0.0 54 54 100001 0 1 MPI_Get() [THROTTLED] 0.0 50 50 7 0 7235 MPI_File_open() 0.0 50 50 100001 0 1 deemat [THROTTLED] 0.0 44 44 2 0 22056 cgca_gbs 0.0 40 40 100001 0 0 cgca_clvgsd [THROTTLED] 0.9 38 4,398 12284 12284 358 my_barrier 0.0 37 37 100001 0 0 shape_der [THROTTLED] 0.0 30 30 100001 0 0 invert [THROTTLED] 0.0 26 26 6144 0 4 MPI_Waitall() 0.0 26 26 67 0 392 MPI_Reduce() 0.0 25 25 13 0 1964 MPI_Win_create() 0.0 22 22 100001 0 0 cgca_csym [THROTTLED] 0.1 22 427 5 30000 85421 cgca_pfem_intcalc1 0.0 21 22 1 1 22483 load 0.0 20 20 80000 0 0 determinant 0.0 9 9 12288 0 1 MPI_Get_count() 0.0 8 8 7 0 1256 MPI_File_set_view() 0.0 8 8 1 0 8759 cgca_igb 0.0 8 8 7 0 1240 MPI_File_close() 0.0 7 7 1 37 7987 cgca_gcu 0.0 6 14 1 19 14058 make_ggl 0.0 6 188 155 1720 1219 cgca_gcupdn 0.0 5 5 10107 0 1 MPI_Put() 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4476 allocate_gather_scatter 0.0 3 46 1 160 46482 scatter_nodes 0.1 3 542 1 33 542840 read_g_coord_pp 0.0 3 3 12 0 276 MPI_Win_free() 0.0 2 2 5 0 467 cgca_fv 0.0 2 59 1 576 59179 cgca_pfem_cenc 0.4 0.905 1,732 7 525 247519 cgca_pswci 0.0 0.804 0.804 1 0 804 getname 0.0 0.634 1 1 2 1714 cgca_as 0.0 0.611 0.611 1 0 611 MPI_Comm_dup() 0.0 0.3 0.74 1 9 740 scatter_noadd 0.2 0.277 1,024 1 64 1024271 cgca_ds 0.0 0.22 0.22 1 0 220 rearrange 0.0 0.208 0.208 1 0 208 cgca_pfem_salloc 0.0 0.191 7 1 374 7785 cgca_nr 0.0 0.159 0.159 14 0 11 MPI_Type_commit() 0.0 0.13 0.13 1 0 130 calc_nels_pp 0.0 0.13 5 1 377 5970 cgca_rt 0.1 0.119 249 1 2 249963 read_g_num_pp 0.0 0.104 0.104 5 0 21 cgca_pfem_uym 0.0 0.091 0.091 1 0 91 cgca_ins 0.0 0.09 9 1 64 9197 cgca_drt 0.0 0.086 4 1 65 4292 gcupd_alloc 0.0 0.051 1 5 5 232 sum_p 0.0 0.05 0.05 1 0 50 cgca_cadim 0.0 0.047 0.047 14 0 3 MPI_Type_create_subarray() 0.0 0.044 0.044 1 0 44 abaqus2sg 0.0 0.037 0.037 1 0 37 cgca_pfem_sdalloc 0.0 0.037 0.037 7 0 5 elap_time 0.4 0.034 2,097 3 6 699074 dismsh_ensi_p 0.0 0.022 0.022 2 0 11 MPI_Test() 0.0 0.021 0.022 1 2 22 find_pe_procs 0.0 0.018 0.018 1 0 18 cgca_pfem_ctalloc 0.0 0.017 0.017 5 0 3 sample 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.015 1 0 15 calc_neq_pp 0.0 0.014 0.014 14 0 1 MPI_Type_free() 0.0 0.014 0.014 19 0 1 cgca_agc 0.0 0.014 0.014 15 0 1 cgca_gcf 0.0 0.013 0.013 12 0 1 MPI_Win_fence() 0.0 0.013 0.013 1 0 13 cgca_gdim 0.0 0.011 3 1 3 3694 read_p121 0.0 0.01 0.01 18 0 1 cgca_dgc 0.0 0.008 0.008 9 0 1 MPI_Comm_rank() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.005 0.005 9 0 1 MPI_Comm_size() 0.0 0.005 7 1 2 7770 cgca_art 0.0 0.005 3 1 2 3701 read_loads 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.003 0.201 1 1 201 max_p 0.0 0.003 79 1 1 79626 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc 0.0 0 0 1 0 0 cgca_imco --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 2, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.194 2.665 3.954 0.8069 MPI-IO Write Bandwidth (MB/s) 7 5.194 2.665 3.954 0.8069 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.13E+04 6.567E+04 1.658E+04 Message size received from all nodes 1.229E+04 8.486E+04 4 6.565E+04 1.659E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 6144 7.733E+04 2.896E+04 6.762E+04 9716 Message size sent to node 1 3071 7.733E+04 7.733E+04 7.733E+04 0 Message size sent to node 1 : gather => MPI_Isend() 1 2.896E+04 2.896E+04 2.896E+04 0 Message size sent to node 1 : make_ggl => MPI_Isend() 3071 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 1 : scatter => MPI_Isend() 1 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 6144 8.486E+04 4.243E+04 6.372E+04 2.113E+04 Message size sent to node 3 3071 4.259E+04 4.259E+04 4.259E+04 0 Message size sent to node 3 : gather => MPI_Isend() 1 4.243E+04 4.243E+04 4.243E+04 0 Message size sent to node 3 : make_ggl => MPI_Isend() 3071 8.486E+04 8.486E+04 8.486E+04 0 Message size sent to node 3 : scatter => MPI_Isend() 1 8.486E+04 8.486E+04 8.486E+04 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to node 1 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 1 8.486E+04 8.486E+04 8.486E+04 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 1 8.486E+04 8.486E+04 8.486E+04 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.866E+04 3.866E+04 3.866E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.866E+04 3.866E+04 3.866E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 3;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 54.9 4:17.565 4:17.565 8.01289E+06 0 32 MPI_Win_unlock() 30.0 2:20.654 2:20.654 17279 0 8140 MPI_Barrier() 99.6 39,184 7:46.949 1 503146 466949425 xx14std 46.3 10,754 3:37.036 511 8.04756E+06 424729 cgca_hxi 50.0 5,547 3:54.249 5 269768 46849989 cgca_clvgp_nocosum 1.1 2,506 5,350 3071 36852 1742 gather 0.5 2,453 2,453 15338 0 160 MPI_Allreduce() 0.5 2,129 2,129 6 0 354891 MPI_Send() 0.3 1,602 1,602 1 0 1602175 MPI_Init() 1.1 1,341 5,063 3071 33781 1649 scatter 0.2 869 869 7 0 124269 MPI_File_write_all() 0.2 768 768 12289 0 62 MPI_Recv() 0.1 633 633 135 0 4694 MPI_Bcast() 30.2 410 2:21.750 1 52197 141750221 cgca_sld 0.4 314 1,868 9198 9198 203 dot_product_p 0.1 225 249 100001 100001 2 cgca_clvgn [THROTTLED] 0.0 221 221 12288 0 18 MPI_Isend() 0.2 183 1,078 3066 6132 352 checon_par 0.0 180 180 12288 0 15 MPI_Probe() 100.0 164 7:48.906 1 579 468906133 .TAU application 0.0 160 160 1 0 160239 MPI_Finalize() 1.7 132 8,001 5 20000 1600383 cgca_pfem_simg 0.0 105 105 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 100 100 155 0 648 cgca_dacf 0.0 65 65 100001 0 1 beemat [THROTTLED] 0.0 58 58 100001 0 1 deemat [THROTTLED] 0.0 54 54 100001 0 1 MPI_Get() [THROTTLED] 0.0 50 50 7 0 7248 MPI_File_open() 0.0 45 45 6144 0 7 MPI_Waitall() 0.0 44 44 2 0 22044 cgca_gbs 0.0 41 41 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 38 38 100001 0 0 shape_der [THROTTLED] 1.0 37 4,613 12284 12284 376 my_barrier 0.0 30 30 100001 0 0 invert [THROTTLED] 0.0 26 26 67 0 401 MPI_Reduce() 0.0 25 25 13 0 1973 MPI_Win_create() 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.1 22 621 5 30000 124389 cgca_pfem_intcalc1 0.0 21 22 1 1 22349 load 0.0 19 19 80000 0 0 determinant 0.0 9 9 7 0 1396 MPI_File_close() 0.0 9 9 1 0 9502 cgca_igb 0.0 9 9 12288 0 1 MPI_Get_count() 0.0 8 8 7 0 1201 MPI_File_set_view() 0.0 6 14 1 19 14122 make_ggl 0.0 5 5 1 47 5995 cgca_gcu 0.0 5 5 2000 0 3 find_g3 0.0 5 5 10113 0 1 MPI_Put() 0.0 4 4 1 0 4862 allocate_gather_scatter 0.0 3 46 1 160 46451 scatter_nodes 0.1 3 534 1 33 534891 read_g_coord_pp 0.0 3 53 155 1086 348 cgca_gcupdn 0.0 3 3 12 0 278 MPI_Win_free() 0.0 2 2 5 0 450 cgca_fv 0.0 2 59 1 576 59241 cgca_pfem_cenc 0.0 0.786 0.786 1 0 786 getname 0.4 0.756 1,734 7 525 247772 cgca_pswci 0.0 0.631 1 1 2 1675 cgca_as 0.0 0.437 0.437 1 0 437 MPI_Comm_dup() 0.0 0.306 0.605 1 9 605 scatter_noadd 0.2 0.27 992 1 64 992187 cgca_ds 0.0 0.228 0.228 1 0 228 rearrange 0.0 0.198 0.198 1 0 198 cgca_pfem_salloc 0.0 0.191 7 1 374 7879 cgca_nr 0.0 0.16 0.16 5 0 32 cgca_pfem_uym 0.0 0.152 5 1 377 5950 cgca_rt 0.0 0.149 0.149 1 0 149 calc_nels_pp 0.0 0.143 0.143 14 0 10 MPI_Type_commit() 0.0 0.131 0.131 1 0 131 cgca_ins 0.1 0.118 257 1 2 257843 read_g_num_pp 0.0 0.087 9 1 64 9198 cgca_drt 0.0 0.086 4 1 65 4346 gcupd_alloc 0.0 0.049 0.049 1 0 49 cgca_cadim 0.0 0.046 1 5 5 275 sum_p 0.0 0.043 0.043 1 0 43 abaqus2sg 0.0 0.042 0.042 14 0 3 MPI_Type_create_subarray() 0.0 0.037 0.037 1 0 37 cgca_gdim 0.0 0.037 0.037 1 0 37 cgca_pfem_sdalloc 0.5 0.035 2,129 3 6 709793 dismsh_ensi_p 0.0 0.034 0.034 7 0 5 elap_time 0.0 0.027 0.027 1 0 27 calc_neq_pp 0.0 0.022 0.022 5 0 4 sample 0.0 0.019 0.019 1 0 19 cgca_pfem_ctalloc 0.0 0.018 0.018 23 0 1 cgca_dgc 0.0 0.016 0.016 1 0 16 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.016 1 2 16 find_pe_procs 0.0 0.014 0.014 12 0 1 MPI_Win_fence() 0.0 0.013 0.013 2 0 6 MPI_Test() 0.0 0.013 0.013 14 0 1 MPI_Type_free() 0.0 0.013 0.013 24 0 1 cgca_agc 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.006 0.006 9 0 1 MPI_Comm_rank() 0.0 0.006 3 1 3 3677 read_p121 0.0 0.005 7 1 2 7691 cgca_art 0.0 0.004 0.004 9 0 0 MPI_Comm_size() 0.0 0.004 3 1 2 3700 read_loads 0.0 0.004 79 1 1 79637 read_rest 0.0 0.003 0.003 1 0 3 MPI_Alloc_mem() 0.0 0.003 0.003 1 0 3 cgca_pfem_integalloc 0.0 0.002 0.002 1 0 2 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 cgca_gcf 0.0 0.002 0.259 1 1 259 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ctdalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 3, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.189 2.664 3.95 0.8049 MPI-IO Write Bandwidth (MB/s) 7 5.189 2.664 3.95 0.8049 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.189E+04 6.083E+04 1.824E+04 Message size received from all nodes 1.229E+04 8.486E+04 4 6.08E+04 1.824E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 6144 8.486E+04 2.13E+04 6.371E+04 2.114E+04 Message size sent to node 2 3071 8.486E+04 8.486E+04 8.486E+04 0 Message size sent to node 2 : gather => MPI_Isend() 1 2.13E+04 2.13E+04 2.13E+04 0 Message size sent to node 2 : make_ggl => MPI_Isend() 3071 4.259E+04 4.259E+04 4.259E+04 0 Message size sent to node 2 : scatter => MPI_Isend() 1 4.259E+04 4.259E+04 4.259E+04 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 6144 7.206E+04 3.603E+04 5.792E+04 1.415E+04 Message size sent to node 4 3071 4.378E+04 4.378E+04 4.378E+04 0 Message size sent to node 4 : gather => MPI_Isend() 1 3.603E+04 3.603E+04 3.603E+04 0 Message size sent to node 4 : make_ggl => MPI_Isend() 3071 7.206E+04 7.206E+04 7.206E+04 0 Message size sent to node 4 : scatter => MPI_Isend() 1 7.206E+04 7.206E+04 7.206E+04 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 2 7.206E+04 3.603E+04 5.405E+04 1.802E+04 [GROUP=MAX_MARKER] Message size sent to all nodes 1 3.603E+04 3.603E+04 3.603E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 7.206E+04 7.206E+04 7.206E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 2 8.486E+04 4.259E+04 6.372E+04 2.113E+04 [GROUP=MAX_MARKER] Message size sent to node 2 1 8.486E+04 8.486E+04 8.486E+04 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 1 4.259E+04 4.259E+04 4.259E+04 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 1 7.206E+04 7.206E+04 7.206E+04 0 [GROUP=MAX_MARKER] Message size sent to node 4 1 7.206E+04 7.206E+04 7.206E+04 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 4.243E+04 4.243E+04 4.243E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 4.243E+04 4.243E+04 4.243E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 4;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 56.8 4:26.153 4:26.153 1.12021E+07 0 24 MPI_Win_unlock() 27.8 2:10.384 2:10.384 17279 0 7546 MPI_Barrier() 99.6 39,239 7:46.949 1 503146 466949426 xx14std 55.9 13,968 4:22.076 511 1.12372E+07 512871 cgca_hxi 50.0 3,901 3:54.249 5 369771 46849885 cgca_clvgp_nocosum 1.2 2,574 5,457 3071 36852 1777 gather 0.5 2,490 2,490 15338 0 162 MPI_Allreduce() 0.5 2,160 2,160 6 0 360154 MPI_Send() 0.3 1,584 1,584 1 0 1584020 MPI_Init() 1.0 1,321 4,807 3071 33781 1566 scatter 0.2 869 869 7 0 124195 MPI_File_write_all() 0.2 796 796 12289 0 65 MPI_Recv() 0.1 626 626 135 0 4639 MPI_Bcast() 30.2 367 2:21.750 1 51201 141750199 cgca_sld 0.4 293 1,861 9198 9198 202 dot_product_p 0.1 237 237 12288 0 19 MPI_Isend() 0.1 228 252 100001 100001 3 cgca_clvgn [THROTTLED] 100.0 178 7:48.907 1 561 468907407 .TAU application 0.2 176 1,095 3066 6132 357 checon_par 0.0 165 165 1 0 165882 MPI_Finalize() 0.0 128 128 12288 0 10 MPI_Probe() 0.0 120 120 100001 0 1 invert [THROTTLED] 0.0 114 114 155 0 738 cgca_dacf 0.0 96 96 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 71 71 100001 0 1 beemat [THROTTLED] 0.0 53 53 100001 0 1 MPI_Get() [THROTTLED] 0.0 50 50 7 0 7249 MPI_File_open() 0.0 46 46 6144 0 8 MPI_Waitall() 0.0 46 46 100001 0 0 deemat [THROTTLED] 0.0 44 44 2 0 22032 cgca_gbs 0.6 43 2,935 5 20000 587016 cgca_pfem_simg 0.0 42 42 100001 0 0 cgca_clvgsd [THROTTLED] 0.9 39 4,453 12284 12284 363 my_barrier 0.0 38 38 100001 0 0 shape_der [THROTTLED] 0.0 27 27 67 0 407 MPI_Reduce() 0.0 25 25 13 0 1952 MPI_Win_create() 0.0 24 24 100001 0 0 cgca_gcr [THROTTLED] 0.0 24 24 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22364 load 0.0 20 183 5 30000 36784 cgca_pfem_intcalc1 0.0 20 20 80000 0 0 determinant 0.0 9 9 1 0 9577 cgca_igb 0.0 8 8 7 0 1277 MPI_File_close() 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 8 8 7 0 1190 MPI_File_set_view() 0.0 6 14 1 19 14037 make_ggl 0.0 5 5 10106 0 1 MPI_Put() 0.0 5 5 2000 0 3 find_g3 0.0 5 186 155 1729 1204 cgca_gcupdn 0.0 4 4 1 15 4717 cgca_gcu 0.0 4 4 1 0 4485 allocate_gather_scatter 0.0 3 46 1 160 46345 scatter_nodes 0.1 3 527 1 33 527592 read_g_coord_pp 0.0 3 3 12 0 274 MPI_Win_free() 0.0 2 2 5 0 450 cgca_fv 0.0 1 59 1 576 59019 cgca_pfem_cenc 0.4 0.83 1,733 7 525 247603 cgca_pswci 0.0 0.791 0.791 1 0 791 getname 0.0 0.661 1 1 2 1559 cgca_as 0.0 0.522 0.522 1 0 522 MPI_Comm_dup() 0.0 0.305 0.611 1 9 611 scatter_noadd 0.2 0.267 960 1 64 960575 cgca_ds 0.0 0.227 0.227 14 0 16 MPI_Type_commit() 0.0 0.225 0.225 1 0 225 calc_nels_pp 0.0 0.221 0.221 1 0 221 rearrange 0.0 0.191 0.191 1 0 191 cgca_pfem_salloc 0.0 0.181 7 1 374 7819 cgca_nr 0.0 0.146 5 1 377 5996 cgca_rt 0.1 0.123 265 1 2 265082 read_g_num_pp 0.0 0.11 0.11 5 0 22 cgca_pfem_uym 0.0 0.105 4 1 65 4325 gcupd_alloc 0.0 0.088 0.088 1 0 88 cgca_ins 0.0 0.081 9 1 64 9243 cgca_drt 0.0 0.051 0.051 7 0 7 elap_time 0.0 0.05 1 5 5 237 sum_p 0.0 0.047 0.047 1 0 47 cgca_cadim 0.0 0.046 0.046 14 0 3 MPI_Type_create_subarray() 0.0 0.045 0.045 1 0 45 abaqus2sg 0.0 0.045 0.045 1 0 45 cgca_gdim 0.5 0.033 2,160 3 6 720320 dismsh_ensi_p 0.0 0.032 0.032 1 0 32 cgca_pfem_sdalloc 0.0 0.027 0.027 9 0 3 MPI_Comm_size() 0.0 0.025 0.025 12 0 2 MPI_Win_fence() 0.0 0.022 0.022 2 0 11 MPI_Test() 0.0 0.022 0.022 26 0 1 cgca_gcf 0.0 0.018 0.018 5 0 4 sample 0.0 0.016 0.016 1 0 16 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.016 0.018 1 2 18 find_pe_procs 0.0 0.015 0.015 1 0 15 calc_neq_pp 0.0 0.014 0.014 1 0 14 cgca_pfem_ctalloc 0.0 0.011 0.011 14 0 1 MPI_Type_free() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.007 7 1 2 7859 cgca_art 0.0 0.007 3 1 3 3704 read_p121 0.0 0.006 0.006 9 0 1 MPI_Comm_rank() 0.0 0.005 0.005 8 0 1 cgca_agc 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 0.004 7 0 1 cgca_dgc 0.0 0.004 0.265 1 1 265 max_p 0.0 0.003 0.003 1 0 3 cgca_pfem_integalloc 0.0 0.003 3 1 2 3704 read_loads 0.0 0.003 79 1 1 79653 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc 0.0 0 0 1 0 0 cgca_imco --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 4, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.198 2.665 3.953 0.8077 MPI-IO Write Bandwidth (MB/s) 7 5.198 2.665 3.953 0.8077 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.732E+04 6.107E+04 1.251E+04 Message size received from all nodes 1.229E+04 7.378E+04 4 6.105E+04 1.252E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 6144 7.206E+04 2.189E+04 5.791E+04 1.415E+04 Message size sent to node 3 3071 7.206E+04 7.206E+04 7.206E+04 0 Message size sent to node 3 : gather => MPI_Isend() 1 2.189E+04 2.189E+04 2.189E+04 0 Message size sent to node 3 : make_ggl => MPI_Isend() 3071 4.378E+04 4.378E+04 4.378E+04 0 Message size sent to node 3 : scatter => MPI_Isend() 1 4.378E+04 4.378E+04 4.378E+04 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 6144 7.378E+04 3.689E+04 6.421E+04 9574 Message size sent to node 5 3071 5.464E+04 5.464E+04 5.464E+04 0 Message size sent to node 5 : gather => MPI_Isend() 1 3.689E+04 3.689E+04 3.689E+04 0 Message size sent to node 5 : make_ggl => MPI_Isend() 3071 7.378E+04 7.378E+04 7.378E+04 0 Message size sent to node 5 : scatter => MPI_Isend() 1 7.378E+04 7.378E+04 7.378E+04 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 2 7.378E+04 3.689E+04 5.533E+04 1.844E+04 [GROUP=MAX_MARKER] Message size sent to all nodes 1 3.689E+04 3.689E+04 3.689E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 7.378E+04 7.378E+04 7.378E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 2 7.206E+04 4.378E+04 5.792E+04 1.414E+04 [GROUP=MAX_MARKER] Message size sent to node 3 1 7.206E+04 7.206E+04 7.206E+04 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 1 4.378E+04 4.378E+04 4.378E+04 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 1 7.378E+04 7.378E+04 7.378E+04 0 [GROUP=MAX_MARKER] Message size sent to node 5 1 7.378E+04 7.378E+04 7.378E+04 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.732E+04 2.732E+04 2.732E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.732E+04 2.732E+04 2.732E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 5;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 72.7 5:40.780 5:40.780 1.44714E+07 0 24 MPI_Win_unlock() 9.6 45,136 45,136 17279 0 2612 MPI_Barrier() 99.6 39,267 7:46.949 1 503146 466949387 xx14std 74.7 22,729 5:50.177 511 1.45061E+07 685280 cgca_hxi 50.0 5,622 3:54.249 5 369783 46849985 cgca_clvgp_nocosum 1.2 2,550 5,442 3071 36852 1772 gather 0.5 2,394 2,394 15338 0 156 MPI_Allreduce() 0.5 2,192 2,192 6 0 365450 MPI_Send() 0.3 1,584 1,584 1 0 1584537 MPI_Init() 1.1 1,287 4,926 3071 33781 1604 scatter 0.2 869 869 7 0 124269 MPI_File_write_all() 0.2 782 782 12289 0 64 MPI_Recv() 0.1 619 619 135 0 4588 MPI_Bcast() 30.2 489 2:21.750 1 50703 141750305 cgca_sld 0.4 323 1,820 9198 9198 198 dot_product_p 0.1 256 256 12288 0 21 MPI_Isend() 0.1 233 257 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 195 1,089 3066 6132 355 checon_par 100.0 178 7:48.907 1 564 468907891 .TAU application 0.0 173 173 12288 0 14 MPI_Probe() 0.0 165 165 1 0 165841 MPI_Finalize() 0.0 165 165 100001 0 2 MPI_Win_lock() [THROTTLED] 0.0 100 100 155 0 648 cgca_dacf 0.0 75 75 100001 0 1 beemat [THROTTLED] 0.0 66 66 100001 0 1 MPI_Get() [THROTTLED] 0.7 58 3,408 5 20000 681601 cgca_pfem_simg 0.0 51 51 7 0 7344 MPI_File_open() 0.0 47 47 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 47 47 6144 0 8 MPI_Waitall() 0.0 45 45 100001 0 0 deemat [THROTTLED] 0.0 44 44 2 0 22062 cgca_gbs 1.0 39 4,538 12284 12284 369 my_barrier 0.0 37 37 100001 0 0 shape_der [THROTTLED] 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 27 27 80000 0 0 determinant 0.0 27 27 67 0 407 MPI_Reduce() 0.0 26 26 100001 0 0 cgca_gcr [THROTTLED] 0.0 25 25 13 0 1971 MPI_Win_create() 0.0 24 24 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22361 load 0.0 21 196 5 30000 39249 cgca_pfem_intcalc1 0.0 9 9 7 0 1287 MPI_File_close() 0.0 8 8 1 0 8563 cgca_igb 0.0 8 8 7 0 1176 MPI_File_set_view() 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 7 7 1 69 7886 cgca_gcu 0.1 7 240 155 2648 1549 cgca_gcupdn 0.0 6 14 1 19 14022 make_ggl 0.0 5 5 2000 0 3 find_g3 0.0 4 4 10107 0 0 MPI_Put() 0.0 4 4 1 0 4604 allocate_gather_scatter 0.1 3 520 1 33 520547 read_g_coord_pp 0.0 3 46 1 160 46395 scatter_nodes 0.0 3 3 12 0 265 MPI_Win_free() 0.0 2 2 5 0 454 cgca_fv 0.0 1 59 1 576 59332 cgca_pfem_cenc 0.4 0.809 1,734 7 525 247728 cgca_pswci 0.0 0.755 0.755 1 0 755 getname 0.0 0.631 1 1 2 1658 cgca_as 0.0 0.519 0.519 1 0 519 MPI_Comm_dup() 0.0 0.327 0.603 1 9 603 scatter_noadd 0.2 0.279 928 1 64 928788 cgca_ds 0.0 0.22 0.22 1 0 220 rearrange 0.0 0.189 7 1 374 7875 cgca_nr 0.0 0.175 0.175 14 0 12 MPI_Type_commit() 0.0 0.171 0.171 1 0 171 calc_nels_pp 0.0 0.163 0.163 1 0 163 cgca_pfem_salloc 0.0 0.147 5 1 377 5942 cgca_rt 0.0 0.113 0.113 5 0 23 cgca_pfem_uym 0.1 0.108 272 1 2 272203 read_g_num_pp 0.0 0.1 0.1 1 0 100 cgca_ins 0.0 0.099 9 1 64 9244 cgca_drt 0.0 0.088 4 1 65 4465 gcupd_alloc 0.0 0.05 0.05 14 0 4 MPI_Type_create_subarray() 0.0 0.047 0.047 1 0 47 cgca_cadim 0.0 0.047 0.047 1 0 47 cgca_gdim 0.0 0.046 0.046 1 0 46 abaqus2sg 0.0 0.046 1 5 5 249 sum_p 0.5 0.041 2,192 3 6 730913 dismsh_ensi_p 0.0 0.036 0.036 7 0 5 elap_time 0.0 0.032 0.032 1 0 32 cgca_pfem_sdalloc 0.0 0.028 0.028 27 0 1 cgca_gcf 0.0 0.023 0.023 35 0 1 cgca_agc 0.0 0.02 0.02 2 0 10 MPI_Test() 0.0 0.017 0.017 12 0 1 MPI_Win_fence() 0.0 0.017 0.017 1 0 17 calc_neq_pp 0.0 0.016 0.016 1 0 16 cgca_pfem_ctalloc 0.0 0.016 0.016 5 0 3 sample 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.015 34 0 0 cgca_dgc 0.0 0.015 0.016 1 2 16 find_pe_procs 0.0 0.014 0.014 9 0 2 MPI_Comm_size() 0.0 0.009 0.009 14 0 1 MPI_Type_free() 0.0 0.008 0.008 9 0 1 MPI_Comm_rank() 0.0 0.007 0.007 1 0 7 caflaunch$MAIN$blk 0.0 0.006 7 1 2 7640 cgca_art 0.0 0.006 3 1 3 3722 read_p121 0.0 0.005 3 1 2 3705 read_loads 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.003 0.003 1 0 3 cgca_pfem_integalloc 0.0 0.003 79 1 1 79648 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.3 1 1 300 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ctdalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 5, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.19 2.663 3.95 0.805 MPI-IO Write Bandwidth (MB/s) 7 5.19 2.663 3.95 0.805 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.896E+04 6.473E+04 8593 Message size received from all nodes 1.229E+04 7.378E+04 4 6.471E+04 8609 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 6144 7.378E+04 2.732E+04 6.42E+04 9579 Message size sent to node 4 3071 7.378E+04 7.378E+04 7.378E+04 0 Message size sent to node 4 : gather => MPI_Isend() 1 2.732E+04 2.732E+04 2.732E+04 0 Message size sent to node 4 : make_ggl => MPI_Isend() 3071 5.464E+04 5.464E+04 5.464E+04 0 Message size sent to node 4 : scatter => MPI_Isend() 1 5.464E+04 5.464E+04 5.464E+04 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 6144 7.259E+04 3.63E+04 6.525E+04 7345 Message size sent to node 6 3071 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 6 : gather => MPI_Isend() 1 3.63E+04 3.63E+04 3.63E+04 0 Message size sent to node 6 : make_ggl => MPI_Isend() 3071 7.259E+04 7.259E+04 7.259E+04 0 Message size sent to node 6 : scatter => MPI_Isend() 1 7.259E+04 7.259E+04 7.259E+04 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 5.464E+04 5.464E+04 5.464E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 5.464E+04 5.464E+04 5.464E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 1 5.464E+04 5.464E+04 5.464E+04 0 [GROUP=MAX_MARKER] Message size sent to node 4 1 5.464E+04 5.464E+04 5.464E+04 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 1 7.259E+04 7.259E+04 7.259E+04 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 1 7.259E+04 7.259E+04 7.259E+04 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.689E+04 3.689E+04 3.689E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.689E+04 3.689E+04 3.689E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 6;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 58.9 4:36.111 4:36.111 1.44714E+07 0 19 MPI_Win_unlock() 24.6 1:55.467 1:55.467 17279 0 6683 MPI_Barrier() 99.6 38,904 7:46.949 1 503146 466949517 xx14std 59.3 17,435 4:37.908 511 1.45061E+07 543851 cgca_hxi 50.0 5,685 3:54.249 5 269771 46849922 cgca_clvgp_nocosum 1.2 2,558 5,403 3071 36852 1760 gather 0.5 2,480 2,480 15338 0 162 MPI_Allreduce() 0.5 2,224 2,224 6 0 370730 MPI_Send() 0.3 1,602 1,602 1 0 1602208 MPI_Init() 1.1 1,364 5,276 3071 33781 1718 scatter 0.2 870 870 7 0 124303 MPI_File_write_all() 0.2 806 806 12289 0 66 MPI_Recv() 0.1 612 612 135 0 4534 MPI_Bcast() 30.2 427 2:21.750 1 50703 141750195 cgca_sld 0.4 306 1,866 9198 9198 203 dot_product_p 0.0 232 232 12288 0 19 MPI_Isend() 0.1 227 250 100001 100001 3 cgca_clvgn [THROTTLED] 0.0 181 181 1 0 181403 MPI_Finalize() 0.2 176 1,093 3066 6132 357 checon_par 100.0 165 7:48.928 1 564 468928126 .TAU application 0.0 133 133 12288 0 11 MPI_Probe() 0.0 101 101 155 0 657 cgca_dacf 0.0 97 97 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 67 67 100001 0 1 beemat [THROTTLED] 0.0 54 54 100001 0 1 MPI_Get() [THROTTLED] 0.9 54 3,992 5 20000 798566 cgca_pfem_simg 0.0 50 50 7 0 7283 MPI_File_open() 0.0 47 47 6144 0 8 MPI_Waitall() 0.0 45 45 100001 0 0 deemat [THROTTLED] 0.0 44 44 2 0 22060 cgca_gbs 0.0 40 40 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 39 39 100001 0 0 shape_der [THROTTLED] 1.0 38 4,814 12284 12284 392 my_barrier 0.0 36 36 100001 0 0 invert [THROTTLED] 0.0 27 27 67 0 412 MPI_Reduce() 0.0 25 25 13 0 1937 MPI_Win_create() 0.0 24 24 80000 0 0 determinant 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22390 load 0.0 21 212 5 30000 42439 cgca_pfem_intcalc1 0.0 9 9 1 0 9074 cgca_igb 0.1 8 264 155 2654 1707 cgca_gcupdn 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 7 7 7 0 1136 MPI_File_set_view() 0.0 7 7 7 0 1074 MPI_File_close() 0.0 6 14 1 19 14099 make_ggl 0.0 6 6 1 39 6140 cgca_gcu 0.0 5 5 10107 0 1 MPI_Put() 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4392 allocate_gather_scatter 0.1 4 513 1 33 513415 read_g_coord_pp 0.0 3 46 1 160 46353 scatter_nodes 0.0 2 2 12 0 250 MPI_Win_free() 0.0 2 2 5 0 445 cgca_fv 0.0 2 59 1 576 59506 cgca_pfem_cenc 0.4 0.961 1,732 7 525 247508 cgca_pswci 0.0 0.79 0.79 1 0 790 getname 0.0 0.633 1 1 2 1680 cgca_as 0.0 0.575 0.575 1 0 575 MPI_Comm_dup() 0.0 0.31 0.592 1 9 592 scatter_noadd 0.2 0.249 897 1 64 897104 cgca_ds 0.0 0.211 0.211 1 0 211 rearrange 0.0 0.203 0.203 1 0 203 cgca_pfem_salloc 0.0 0.195 0.195 1 0 195 calc_nels_pp 0.0 0.178 7 1 374 7819 cgca_nr 0.0 0.168 0.168 14 0 12 MPI_Type_commit() 0.0 0.159 5 1 377 5962 cgca_rt 0.1 0.121 279 1 2 279236 read_g_num_pp 0.0 0.098 4 1 65 4434 gcupd_alloc 0.0 0.097 9 1 64 9287 cgca_drt 0.0 0.087 0.087 1 0 87 cgca_ins 0.0 0.071 0.071 5 0 14 cgca_pfem_uym 0.0 0.051 0.051 1 0 51 cgca_cadim 0.0 0.051 1 5 5 250 sum_p 0.0 0.044 0.044 1 0 44 abaqus2sg 0.0 0.042 0.042 14 0 3 MPI_Type_create_subarray() 0.0 0.036 0.036 1 0 36 cgca_pfem_sdalloc 0.0 0.035 0.035 7 0 5 elap_time 0.0 0.034 0.034 1 0 34 cgca_gdim 0.5 0.029 2,224 3 6 741469 dismsh_ensi_p 0.0 0.028 0.028 9 0 3 MPI_Comm_size() 0.0 0.028 0.028 20 0 1 cgca_gcf 0.0 0.028 0.028 5 0 6 sample 0.0 0.023 0.023 1 0 23 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.018 0.018 12 0 2 MPI_Win_fence() 0.0 0.018 0.018 1 0 18 calc_neq_pp 0.0 0.016 0.018 1 2 18 find_pe_procs 0.0 0.014 0.014 1 0 14 cgca_pfem_ctalloc 0.0 0.013 0.013 1 0 13 caflaunch$MAIN$blk 0.0 0.012 0.012 2 0 6 MPI_Test() 0.0 0.012 0.012 14 0 1 MPI_Type_free() 0.0 0.011 3 1 3 3775 read_p121 0.0 0.01 0.01 20 0 0 cgca_agc 0.0 0.008 0.008 9 0 1 MPI_Comm_rank() 0.0 0.008 0.008 19 0 0 cgca_dgc 0.0 0.006 0.006 1 0 6 MPI_Alloc_mem() 0.0 0.005 7 1 2 7336 cgca_art 0.0 0.004 3 1 2 3717 read_loads 0.0 0.004 79 1 1 79642 read_rest 0.0 0.003 0.003 1 0 3 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 2 0 1 cgca_gcr 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.002 0.196 1 1 196 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 6, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.187 2.662 3.949 0.8051 MPI-IO Write Bandwidth (MB/s) 7 5.187 2.662 3.949 0.8051 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.366E+04 6.449E+04 1.276E+04 Message size received from all nodes 1.229E+04 8.012E+04 4 6.447E+04 1.277E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 6144 7.259E+04 2.896E+04 6.525E+04 7350 Message size sent to node 5 3071 7.259E+04 7.259E+04 7.259E+04 0 Message size sent to node 5 : gather => MPI_Isend() 1 2.896E+04 2.896E+04 2.896E+04 0 Message size sent to node 5 : make_ggl => MPI_Isend() 3071 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 5 : scatter => MPI_Isend() 1 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 6144 8.012E+04 4.006E+04 6.372E+04 1.64E+04 Message size sent to node 7 3071 4.733E+04 4.733E+04 4.733E+04 0 Message size sent to node 7 : gather => MPI_Isend() 1 4.006E+04 4.006E+04 4.006E+04 0 Message size sent to node 7 : make_ggl => MPI_Isend() 3071 8.012E+04 8.012E+04 8.012E+04 0 Message size sent to node 7 : scatter => MPI_Isend() 1 8.012E+04 8.012E+04 8.012E+04 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to node 5 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 1 8.012E+04 8.012E+04 8.012E+04 0 [GROUP=MAX_MARKER] Message size sent to node 7 1 8.012E+04 8.012E+04 8.012E+04 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.366E+04 2.366E+04 2.366E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.366E+04 2.366E+04 2.366E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 7;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 56.0 4:22.657 4:22.657 1.12021E+07 0 23 MPI_Win_unlock() 28.3 2:12.707 2:12.707 17279 0 7680 MPI_Barrier() 99.6 38,887 7:46.949 1 503146 466949435 xx14std 54.3 13,800 4:14.789 511 1.12372E+07 498610 cgca_hxi 50.0 5,547 3:54.250 5 269768 46850007 cgca_clvgp_nocosum 1.1 2,519 5,378 3071 36852 1751 gather 0.5 2,420 2,420 15338 0 158 MPI_Allreduce() 0.5 2,256 2,256 6 0 376009 MPI_Send() 0.3 1,589 1,589 1 0 1589125 MPI_Init() 1.1 1,382 5,371 3071 33781 1749 scatter 0.2 870 870 7 0 124345 MPI_File_write_all() 0.2 860 860 12289 0 70 MPI_Recv() 0.1 605 605 135 0 4486 MPI_Bcast() 30.2 368 2:21.750 1 51201 141750231 cgca_sld 0.4 315 1,824 9198 9198 198 dot_product_p 0.1 241 264 100001 100001 3 cgca_clvgn [THROTTLED] 0.0 234 234 12288 0 19 MPI_Isend() 0.0 205 205 12288 0 17 MPI_Probe() 0.2 180 1,088 3066 6132 355 checon_par 100.0 174 7:48.908 1 553 468908449 .TAU application 0.0 165 165 1 0 165558 MPI_Finalize() 0.0 100 100 155 0 652 cgca_dacf 0.0 83 83 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 66 66 100001 0 1 beemat [THROTTLED] 0.0 53 53 100001 0 1 MPI_Get() [THROTTLED] 0.0 53 53 6144 0 9 MPI_Waitall() 0.0 50 50 7 0 7268 MPI_File_open() 0.9 46 3,999 5 20000 799956 cgca_pfem_simg 0.0 45 45 100001 0 0 deemat [THROTTLED] 0.0 44 44 2 0 22038 cgca_gbs 1.0 39 4,833 12284 12284 394 my_barrier 0.0 37 37 100001 0 0 shape_der [THROTTLED] 0.0 33 33 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 27 27 67 0 412 MPI_Reduce() 0.0 27 27 80000 0 0 determinant 0.0 25 25 13 0 1988 MPI_Win_create() 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.1 22 238 5 30000 47742 cgca_pfem_intcalc1 0.0 21 22 1 1 22483 load 0.0 9 9 1 0 9080 cgca_igb 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 8 8 7 0 1145 MPI_File_set_view() 0.0 7 7 7 0 1070 MPI_File_close() 0.0 6 14 1 19 14117 make_ggl 0.0 6 6 1 37 6067 cgca_gcu 0.0 5 5 10103 0 1 MPI_Put() 0.0 5 5 2000 0 3 find_g3 0.0 4 81 155 1706 524 cgca_gcupdn 0.0 4 4 1 0 4402 allocate_gather_scatter 0.0 3 46 1 160 46383 scatter_nodes 0.1 3 506 1 33 506453 read_g_coord_pp 0.0 3 3 12 0 266 MPI_Win_free() 0.0 2 2 5 0 432 cgca_fv 0.0 1 58 1 576 58876 cgca_pfem_cenc 0.4 0.87 1,732 7 525 247534 cgca_pswci 0.0 0.754 0.754 1 0 754 getname 0.0 0.746 1 1 2 1782 cgca_as 0.0 0.567 0.567 1 0 567 MPI_Comm_dup() 0.0 0.318 0.729 1 9 729 scatter_noadd 0.2 0.272 865 1 64 865391 cgca_ds 0.0 0.22 0.22 1 0 220 rearrange 0.0 0.211 0.211 1 0 211 calc_nels_pp 0.0 0.176 7 1 374 7893 cgca_nr 0.0 0.175 0.175 14 0 12 MPI_Type_commit() 0.0 0.158 0.158 1 0 158 cgca_pfem_salloc 0.1 0.133 286 1 2 286220 read_g_num_pp 0.0 0.123 5 1 377 5944 cgca_rt 0.0 0.116 0.116 1 0 116 cgca_ins 0.0 0.086 9 1 64 9272 cgca_drt 0.0 0.077 0.077 5 0 15 cgca_pfem_uym 0.0 0.064 4 1 65 4445 gcupd_alloc 0.0 0.048 0.048 14 0 3 MPI_Type_create_subarray() 0.0 0.046 0.046 1 0 46 cgca_cadim 0.0 0.046 1 5 5 270 sum_p 0.0 0.044 0.044 1 0 44 abaqus2sg 0.0 0.044 0.044 1 0 44 cgca_gdim 0.0 0.038 0.038 1 0 38 cgca_pfem_sdalloc 0.0 0.035 0.035 7 0 5 elap_time 0.5 0.028 2,256 3 6 752027 dismsh_ensi_p 0.0 0.026 0.026 5 0 5 sample 0.0 0.022 0.022 1 0 22 calc_neq_pp 0.0 0.022 0.022 1 0 22 cgca_pfem_ctalloc 0.0 0.019 0.019 1 0 19 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.018 0.018 1 2 18 find_pe_procs 0.0 0.013 0.013 12 0 1 MPI_Win_fence() 0.0 0.011 0.011 2 0 6 MPI_Test() 0.0 0.011 0.011 14 0 1 MPI_Type_free() 0.0 0.01 0.01 9 0 1 MPI_Comm_size() 0.0 0.01 0.01 19 0 1 cgca_agc 0.0 0.009 0.009 18 0 0 cgca_dgc 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.007 0.007 9 0 1 MPI_Comm_rank() 0.0 0.007 3 1 3 3797 read_p121 0.0 0.006 8 1 2 8072 cgca_art 0.0 0.005 3 1 2 3757 read_loads 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 0.27 1 1 270 max_p 0.0 0.003 0.003 1 0 3 MPI_Finalized() 0.0 0.003 0.003 1 0 3 cgca_pfem_integalloc 0.0 0.002 79 1 1 79639 read_rest 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_gcf 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ctdalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 7, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.183 2.663 3.947 0.8038 MPI-IO Write Bandwidth (MB/s) 7 5.183 2.663 3.947 0.8038 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.426E+04 6.083E+04 1.371E+04 Message size received from all nodes 1.229E+04 8.012E+04 4 6.08E+04 1.371E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 6144 8.012E+04 2.366E+04 6.371E+04 1.64E+04 Message size sent to node 6 3071 8.012E+04 8.012E+04 8.012E+04 0 Message size sent to node 6 : gather => MPI_Isend() 1 2.366E+04 2.366E+04 2.366E+04 0 Message size sent to node 6 : make_ggl => MPI_Isend() 3071 4.733E+04 4.733E+04 4.733E+04 0 Message size sent to node 6 : scatter => MPI_Isend() 1 4.733E+04 4.733E+04 4.733E+04 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 6144 6.733E+04 3.366E+04 5.792E+04 9412 Message size sent to node 8 3071 4.851E+04 4.851E+04 4.851E+04 0 Message size sent to node 8 : gather => MPI_Isend() 1 3.366E+04 3.366E+04 3.366E+04 0 Message size sent to node 8 : make_ggl => MPI_Isend() 3071 6.733E+04 6.733E+04 6.733E+04 0 Message size sent to node 8 : scatter => MPI_Isend() 1 6.733E+04 6.733E+04 6.733E+04 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 2 8.012E+04 4.733E+04 6.372E+04 1.64E+04 [GROUP=MAX_MARKER] Message size sent to node 6 1 8.012E+04 8.012E+04 8.012E+04 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 1 4.733E+04 4.733E+04 4.733E+04 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 1 6.733E+04 6.733E+04 6.733E+04 0 [GROUP=MAX_MARKER] Message size sent to node 8 1 6.733E+04 6.733E+04 6.733E+04 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 4.006E+04 4.006E+04 4.006E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 4.006E+04 4.006E+04 4.006E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 8;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 61.8 4:49.812 4:49.812 1.12021E+07 0 26 MPI_Win_unlock() 22.7 1:46.647 1:46.647 17279 0 6172 MPI_Barrier() 99.6 39,314 7:46.949 1 503146 466949440 xx14std 55.2 13,861 4:18.829 511 1.12372E+07 506515 cgca_hxi 50.0 3,642 3:54.249 5 326985 46849933 cgca_clvgp_nocosum 1.2 2,712 5,515 3071 36852 1796 gather 0.5 2,444 2,444 15338 0 159 MPI_Allreduce() 0.5 2,287 2,287 6 0 381275 MPI_Send() 0.3 1,581 1,581 1 0 1581752 MPI_Init() 1.0 1,332 4,828 3071 33781 1572 scatter 0.2 889 889 12289 0 72 MPI_Recv() 0.2 869 869 7 0 124182 MPI_File_write_all() 0.1 598 598 135 0 4432 MPI_Bcast() 30.2 441 2:21.750 1 51201 141750309 cgca_sld 0.4 293 1,807 9198 9198 197 dot_product_p 0.1 250 250 12288 0 20 MPI_Isend() 0.1 225 247 100001 100001 2 cgca_clvgn [THROTTLED] 0.0 189 189 12288 0 15 MPI_Probe() 100.0 180 7:48.904 1 548 468904015 .TAU application 0.2 164 1,091 3066 6132 356 checon_par 0.0 162 162 1 0 162346 MPI_Finalize() 0.0 97 97 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 94 94 155 0 613 cgca_dacf 5.8 93 27,117 5 20000 5423564 cgca_pfem_simg 0.0 66 66 100001 0 1 beemat [THROTTLED] 0.0 52 52 100001 0 1 MPI_Get() [THROTTLED] 0.0 51 51 7 0 7417 MPI_File_open() 0.0 47 47 6144 0 8 MPI_Waitall() 0.0 46 46 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21946 cgca_gbs 0.0 43 43 100001 0 0 cgca_clvgsd [THROTTLED] 0.9 41 4,343 12284 12284 354 my_barrier 0.0 37 37 100001 0 0 shape_der [THROTTLED] 0.2 32 1,086 5 30000 217392 cgca_pfem_intcalc1 0.0 30 30 100001 0 0 invert [THROTTLED] 0.0 26 26 67 0 401 MPI_Reduce() 0.0 23 23 13 0 1796 MPI_Win_create() 0.0 22 22 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22519 load 0.0 19 19 80000 0 0 determinant 0.0 13 13 57216 0 0 cgca_gcr 0.0 10 10 1 0 10128 cgca_igb 0.0 9 9 7 0 1352 MPI_File_close() 0.0 8 8 10101 0 1 MPI_Put() 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 7 7 7 0 1058 MPI_File_set_view() 0.0 6 14 1 19 14090 make_ggl 0.0 5 5 2000 0 3 find_g3 0.1 5 500 1 33 500246 read_g_coord_pp 0.0 5 5 1 0 5177 allocate_gather_scatter 0.0 4 83 155 1726 541 cgca_gcupdn 0.0 4 4 1 7 4303 cgca_gcu 0.0 4 46 1 160 46838 scatter_nodes 0.0 3 3 12 0 279 MPI_Win_free() 0.0 2 2 5 0 422 cgca_fv 0.0 1 61 1 576 61322 cgca_pfem_cenc 0.4 0.859 1,733 7 525 247687 cgca_pswci 0.0 0.783 0.783 1 0 783 getname 0.0 0.714 0.714 1 0 714 cgca_ins 0.0 0.62 1 1 2 1555 cgca_as 0.0 0.549 0.549 1 0 549 MPI_Comm_dup() 0.0 0.363 0.744 1 9 744 scatter_noadd 0.2 0.298 833 1 64 833780 cgca_ds 0.0 0.212 0.212 1 0 212 cgca_pfem_salloc 0.0 0.202 0.202 1 0 202 rearrange 0.0 0.191 7 1 374 7826 cgca_nr 0.0 0.171 0.171 1 0 171 calc_nels_pp 0.0 0.161 0.161 14 0 12 MPI_Type_commit() 0.0 0.133 5 1 377 5987 cgca_rt 0.1 0.125 293 1 2 293146 read_g_num_pp 0.0 0.117 0.117 5 0 23 cgca_pfem_uym 0.0 0.082 9 1 64 9206 cgca_drt 0.0 0.071 4 1 65 4494 gcupd_alloc 0.0 0.055 0.055 14 0 4 MPI_Type_create_subarray() 0.0 0.055 0.055 7 0 8 elap_time 0.0 0.052 0.052 1 0 52 cgca_gdim 0.0 0.047 0.977 5 5 195 sum_p 0.0 0.043 0.043 1 0 43 abaqus2sg 0.0 0.041 0.041 1 0 41 cgca_pfem_sdalloc 0.0 0.033 0.033 9 0 4 MPI_Comm_rank() 0.0 0.033 0.033 1 0 33 cgca_cadim 0.5 0.033 2,287 3 6 762562 dismsh_ensi_p 0.0 0.032 0.032 2 0 16 MPI_Test() 0.0 0.022 0.022 5 0 4 sample 0.0 0.019 0.019 12 0 2 MPI_Win_fence() 0.0 0.016 0.017 1 2 17 find_pe_procs 0.0 0.015 0.015 1 0 15 calc_neq_pp 0.0 0.015 0.015 1 0 15 cgca_pfem_ctalloc 0.0 0.014 0.014 22 0 1 cgca_gcf 0.0 0.013 0.013 1 0 13 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.012 0.012 14 0 1 MPI_Type_free() 0.0 0.008 3 1 3 3906 read_p121 0.0 0.007 0.007 9 0 1 MPI_Comm_size() 0.0 0.007 0.007 1 0 7 caflaunch$MAIN$blk 0.0 0.005 5 1 2 5581 cgca_art 0.0 0.005 3 1 2 3766 read_loads 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 0.004 4 0 1 cgca_agc 0.0 0.003 0.003 3 0 1 cgca_dgc 0.0 0.003 79 1 1 79739 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.002 0.261 1 1 261 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 8, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.196 2.664 3.954 0.8074 MPI-IO Write Bandwidth (MB/s) 7 5.196 2.664 3.954 0.8074 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.969E+04 6.107E+04 8169 Message size received from all nodes 1.229E+04 6.904E+04 4 6.105E+04 8177 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 6144 6.733E+04 2.426E+04 5.791E+04 9417 Message size sent to node 7 3071 6.733E+04 6.733E+04 6.733E+04 0 Message size sent to node 7 : gather => MPI_Isend() 1 2.426E+04 2.426E+04 2.426E+04 0 Message size sent to node 7 : make_ggl => MPI_Isend() 3071 4.851E+04 4.851E+04 4.851E+04 0 Message size sent to node 7 : scatter => MPI_Isend() 1 4.851E+04 4.851E+04 4.851E+04 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 6144 6.904E+04 3.452E+04 6.42E+04 4846 Message size sent to node 9 3071 5.938E+04 5.938E+04 5.938E+04 0 Message size sent to node 9 : gather => MPI_Isend() 1 3.452E+04 3.452E+04 3.452E+04 0 Message size sent to node 9 : make_ggl => MPI_Isend() 3071 6.904E+04 6.904E+04 6.904E+04 0 Message size sent to node 9 : scatter => MPI_Isend() 1 6.904E+04 6.904E+04 6.904E+04 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 1 4.851E+04 4.851E+04 4.851E+04 0 [GROUP=MAX_MARKER] Message size sent to node 7 1 4.851E+04 4.851E+04 4.851E+04 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 1 6.904E+04 6.904E+04 6.904E+04 0 [GROUP=MAX_MARKER] Message size sent to node 9 1 6.904E+04 6.904E+04 6.904E+04 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.969E+04 2.969E+04 2.969E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.969E+04 2.969E+04 2.969E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 9;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 64.5 5:02.408 5:02.408 1.44714E+07 0 21 MPI_Win_unlock() 18.8 1:28.065 1:28.065 17279 0 5097 MPI_Barrier() 99.6 39,346 7:46.949 1 503146 466949342 xx14std 59.9 18,245 4:40.998 511 1.45061E+07 549900 cgca_hxi 50.0 5,265 3:54.250 5 281969 46850028 cgca_clvgp_nocosum 1.2 2,708 5,516 3071 36852 1796 gather 0.5 2,332 2,332 15338 0 152 MPI_Allreduce() 0.5 2,319 2,319 6 0 386549 MPI_Send() 0.3 1,582 1,582 1 0 1582269 MPI_Init() 1.0 1,314 4,864 3071 33781 1584 scatter 0.2 869 869 7 0 124237 MPI_File_write_all() 0.2 802 802 12289 0 65 MPI_Recv() 0.1 590 590 135 0 4373 MPI_Bcast() 30.2 528 2:21.750 1 50703 141750336 cgca_sld 0.4 309 1,728 9198 9198 188 dot_product_p 0.1 250 250 12288 0 20 MPI_Isend() 0.1 230 253 100001 100001 3 cgca_clvgn [THROTTLED] 0.0 190 190 12288 0 16 MPI_Probe() 0.2 181 1,090 3066 6132 356 checon_par 100.0 179 7:48.906 1 534 468906150 .TAU application 0.0 164 164 1 0 164566 MPI_Finalize() 0.0 130 130 100001 0 1 MPI_Win_lock() [THROTTLED] 5.8 94 26,990 5 20000 5398031 cgca_pfem_simg 0.0 91 91 155 0 592 cgca_dacf 0.0 68 68 100001 0 1 beemat [THROTTLED] 0.0 52 52 100001 0 1 MPI_Get() [THROTTLED] 0.0 51 51 7 0 7418 MPI_File_open() 0.0 48 48 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21966 cgca_gbs 0.9 43 4,408 12284 12284 359 my_barrier 0.0 41 41 100001 0 0 shape_der [THROTTLED] 0.0 39 39 6144 0 7 MPI_Waitall() 0.2 32 1,077 5 30000 215476 cgca_pfem_intcalc1 0.0 32 32 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 27 27 67 0 405 MPI_Reduce() 0.0 25 25 80000 0 0 determinant 0.0 24 24 13 0 1907 MPI_Win_create() 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22521 load 0.0 9 9 1 0 9357 cgca_igb 0.0 8 8 7 0 1275 MPI_File_close() 0.0 8 8 10096 0 1 MPI_Put() 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 7 7 7 0 1083 MPI_File_set_view() 0.0 7 136 155 2657 879 cgca_gcupdn 0.0 6 14 1 19 14154 make_ggl 0.1 6 493 1 33 493260 read_g_coord_pp 0.0 5 5 2000 0 3 find_g3 0.0 5 5 1 27 5109 cgca_gcu 0.0 4 4 1 0 4565 allocate_gather_scatter 0.0 3 46 1 160 46843 scatter_nodes 0.0 3 3 12 0 280 MPI_Win_free() 0.0 2 2 12200 0 0 cgca_gcr 0.0 2 2 5 0 427 cgca_fv 0.0 1 60 1 576 60156 cgca_pfem_cenc 0.0 0.77 0.77 1 0 770 getname 0.4 0.735 1,733 7 525 247676 cgca_pswci 0.0 0.618 1 1 2 1627 cgca_as 0.0 0.536 0.536 1 0 536 MPI_Comm_dup() 0.0 0.334 0.691 1 9 691 scatter_noadd 0.2 0.273 802 1 64 802144 cgca_ds 0.0 0.213 0.213 1 0 213 rearrange 0.0 0.2 7 1 374 7879 cgca_nr 0.0 0.161 0.161 1 0 161 cgca_pfem_salloc 0.0 0.159 0.159 1 0 159 calc_nels_pp 0.0 0.151 0.151 14 0 11 MPI_Type_commit() 0.0 0.144 5 1 377 5952 cgca_rt 0.0 0.133 9 1 64 9206 cgca_drt 0.1 0.112 300 1 2 300320 read_g_num_pp 0.0 0.1 0.1 1 0 100 cgca_ins 0.0 0.095 0.095 5 0 19 cgca_pfem_uym 0.0 0.07 4 1 65 4355 gcupd_alloc 0.0 0.062 0.062 14 0 4 MPI_Type_create_subarray() 0.0 0.052 0.052 1 0 52 cgca_gdim 0.0 0.047 1 5 5 217 sum_p 0.0 0.042 0.042 1 0 42 abaqus2sg 0.0 0.042 0.042 7 0 6 elap_time 0.0 0.041 0.041 1 0 41 cgca_pfem_sdalloc 0.0 0.037 0.037 1 0 37 cgca_cadim 0.5 0.031 2,319 3 6 773109 dismsh_ensi_p 0.0 0.019 0.019 23 0 1 cgca_gcf 0.0 0.019 0.019 5 0 4 sample 0.0 0.018 0.018 1 0 18 cgca_pfem_ctalloc 0.0 0.016 0.016 12 0 1 MPI_Win_fence() 0.0 0.016 0.016 1 2 16 find_pe_procs 0.0 0.015 0.015 1 0 15 calc_neq_pp 0.0 0.014 0.014 1 0 14 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.012 0.012 14 0 1 MPI_Type_free() 0.0 0.01 0.01 2 0 5 MPI_Test() 0.0 0.009 0.009 9 0 1 MPI_Comm_rank() 0.0 0.007 0.007 1 0 7 caflaunch$MAIN$blk 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.006 6 1 2 6805 cgca_art 0.0 0.006 3 1 3 3862 read_p121 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 0.004 14 0 0 cgca_agc 0.0 0.004 0.004 13 0 0 cgca_dgc 0.0 0.003 0.003 1 0 3 MPI_Finalized() 0.0 0.003 0.301 1 1 301 max_p 0.0 0.003 3 1 2 3731 read_loads 0.0 0.003 79 1 1 79890 read_rest 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc 0.0 0 0 1 0 0 cgca_pfem_ealloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 9, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.192 2.664 3.951 0.8055 MPI-IO Write Bandwidth (MB/s) 7 5.192 2.664 3.951 0.8055 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.896E+04 6.355E+04 5037 Message size received from all nodes 1.229E+04 6.904E+04 4 6.353E+04 5058 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 6144 6.786E+04 3.393E+04 6.288E+04 4981 Message size sent to node 10 3071 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 10 : gather => MPI_Isend() 1 3.393E+04 3.393E+04 3.393E+04 0 Message size sent to node 10 : make_ggl => MPI_Isend() 3071 6.786E+04 6.786E+04 6.786E+04 0 Message size sent to node 10 : scatter => MPI_Isend() 1 6.786E+04 6.786E+04 6.786E+04 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 6144 6.904E+04 2.969E+04 6.42E+04 4852 Message size sent to node 8 3071 6.904E+04 6.904E+04 6.904E+04 0 Message size sent to node 8 : gather => MPI_Isend() 1 2.969E+04 2.969E+04 2.969E+04 0 Message size sent to node 8 : make_ggl => MPI_Isend() 3071 5.938E+04 5.938E+04 5.938E+04 0 Message size sent to node 8 : scatter => MPI_Isend() 1 5.938E+04 5.938E+04 5.938E+04 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 5.938E+04 5.938E+04 5.938E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 5.938E+04 5.938E+04 5.938E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 1 6.786E+04 6.786E+04 6.786E+04 0 [GROUP=MAX_MARKER] Message size sent to node 10 1 6.786E+04 6.786E+04 6.786E+04 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 1 5.938E+04 5.938E+04 5.938E+04 0 [GROUP=MAX_MARKER] Message size sent to node 8 1 5.938E+04 5.938E+04 5.938E+04 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.452E+04 3.452E+04 3.452E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.452E+04 3.452E+04 3.452E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 10;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 64.3 5:01.589 5:01.589 1.44714E+07 0 21 MPI_Win_unlock() 18.9 1:28.784 1:28.784 17279 0 5138 MPI_Barrier() 99.6 39,246 7:46.949 1 503146 466949405 xx14std 60.2 18,906 4:42.248 511 1.45061E+07 552346 cgca_hxi 50.0 4,847 3:54.249 5 269768 46849990 cgca_clvgp_nocosum 1.2 2,704 5,456 3071 36852 1777 gather 0.5 2,392 2,392 15338 0 156 MPI_Allreduce() 0.5 2,350 2,350 6 0 391816 MPI_Send() 0.3 1,583 1,583 1 0 1583579 MPI_Init() 1.1 1,324 4,964 3071 33781 1617 scatter 0.2 870 870 7 0 124300 MPI_File_write_all() 0.2 831 831 12289 0 68 MPI_Recv() 0.1 585 585 135 0 4337 MPI_Bcast() 30.2 407 2:21.750 1 50703 141750297 cgca_sld 0.4 315 1,796 9198 9198 195 dot_product_p 0.1 244 244 12288 0 20 MPI_Isend() 0.1 220 243 100001 100001 2 cgca_clvgn [THROTTLED] 0.2 180 1,087 3066 6132 355 checon_par 100.0 178 7:48.906 1 543 468906097 .TAU application 0.0 166 166 12288 0 14 MPI_Probe() 0.0 164 164 1 0 164414 MPI_Finalize() 0.0 125 125 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 88 88 155 0 569 cgca_dacf 5.8 87 26,984 5 20000 5396919 cgca_pfem_simg 0.0 67 67 100001 0 1 beemat [THROTTLED] 0.0 54 54 100001 0 1 MPI_Get() [THROTTLED] 0.0 51 51 7 0 7385 MPI_File_open() 0.0 49 49 6144 0 8 MPI_Waitall() 0.0 46 46 100001 0 0 deemat [THROTTLED] 0.0 46 46 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 43 43 2 0 21946 cgca_gbs 0.9 37 4,427 12284 12284 360 my_barrier 0.0 37 37 100001 0 0 shape_der [THROTTLED] 0.0 35 35 100001 0 0 invert [THROTTLED] 0.2 33 1,074 5 30000 214868 cgca_pfem_intcalc1 0.0 27 27 67 0 404 MPI_Reduce() 0.0 23 23 13 0 1796 MPI_Win_create() 0.0 22 22 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22421 load 0.0 20 20 80000 0 0 determinant 0.0 9 9 1 0 9886 cgca_igb 0.0 9 9 12288 0 1 MPI_Get_count() 0.0 9 9 10099 0 1 MPI_Put() 0.0 8 8 7 0 1192 MPI_File_close() 0.0 7 7 7 0 1031 MPI_File_set_view() 0.0 6 14 1 19 14114 make_ggl 0.0 6 111 155 2655 722 cgca_gcupdn 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4610 allocate_gather_scatter 0.0 4 4 1 15 4608 cgca_gcu 0.1 4 486 1 33 486291 read_g_coord_pp 0.0 4 46 1 160 46530 scatter_nodes 0.0 3 3 12 0 268 MPI_Win_free() 0.0 2 2 5 0 420 cgca_fv 0.0 1 61 1 576 61329 cgca_pfem_cenc 0.0 0.772 0.772 1 0 772 getname 0.4 0.736 1,733 7 525 247581 cgca_pswci 0.0 0.607 1 1 2 1659 cgca_as 0.0 0.55 0.55 1 0 550 MPI_Comm_dup() 0.0 0.321 0.69 1 9 690 scatter_noadd 0.2 0.251 770 1 64 770541 cgca_ds 0.0 0.202 0.202 1 0 202 rearrange 0.0 0.199 0.199 1 0 199 cgca_pfem_salloc 0.0 0.191 7 1 374 7838 cgca_nr 0.0 0.167 0.167 1 0 167 calc_nels_pp 0.0 0.135 5 1 377 5983 cgca_rt 0.0 0.125 0.125 14 0 9 MPI_Type_commit() 0.1 0.115 307 1 2 307256 read_g_num_pp 0.0 0.105 0.105 1 0 105 cgca_ins 0.0 0.088 0.088 5 0 18 cgca_pfem_uym 0.0 0.083 9 1 64 9196 cgca_drt 0.0 0.077 4 1 65 4408 gcupd_alloc 0.0 0.056 1 5 5 208 sum_p 0.0 0.052 0.052 1 0 52 cgca_cadim 0.0 0.041 0.041 14 0 3 MPI_Type_create_subarray() 0.0 0.041 0.041 1 0 41 abaqus2sg 0.0 0.041 0.041 1 0 41 cgca_pfem_sdalloc 0.0 0.041 0.041 7 0 6 elap_time 0.5 0.03 2,350 3 6 783642 dismsh_ensi_p 0.0 0.029 0.029 1 0 29 cgca_gdim 0.0 0.026 0.026 5 0 5 sample 0.0 0.022 0.022 14 0 2 MPI_Type_free() 0.0 0.019 0.019 20 0 1 cgca_gcf 0.0 0.018 0.018 1 0 18 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.017 0.018 1 2 18 find_pe_procs 0.0 0.016 0.016 1 0 16 cgca_pfem_ctalloc 0.0 0.015 0.015 9 0 2 MPI_Comm_rank() 0.0 0.015 0.015 1 0 15 calc_neq_pp 0.0 0.015 79 1 1 79962 read_rest 0.0 0.014 0.014 12 0 1 MPI_Win_fence() 0.0 0.012 3 1 3 3961 read_p121 0.0 0.01 0.01 2 0 5 MPI_Test() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.007 0.007 1 0 7 MPI_Alloc_mem() 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.005 0.005 8 0 1 cgca_agc 0.0 0.005 5 1 2 5539 cgca_art 0.0 0.005 0.005 7 0 1 cgca_dgc 0.0 0.005 3 1 2 3776 read_loads 0.0 0.002 0.002 1 0 2 CI0000$cgca_m3pfem_ 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.002 0.229 1 1 229 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ctdalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 10, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.189 2.662 3.949 0.8054 MPI-IO Write Bandwidth (MB/s) 7 5.189 2.662 3.949 0.8054 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.603E+04 6.331E+04 9023 Message size received from all nodes 1.229E+04 7.538E+04 4 6.329E+04 9032 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 6144 7.538E+04 3.769E+04 6.372E+04 1.166E+04 Message size sent to node 11 3071 5.206E+04 5.206E+04 5.206E+04 0 Message size sent to node 11 : gather => MPI_Isend() 1 3.769E+04 3.769E+04 3.769E+04 0 Message size sent to node 11 : make_ggl => MPI_Isend() 3071 7.538E+04 7.538E+04 7.538E+04 0 Message size sent to node 11 : scatter => MPI_Isend() 1 7.538E+04 7.538E+04 7.538E+04 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 6144 6.786E+04 2.896E+04 6.288E+04 4986 Message size sent to node 9 3071 6.786E+04 6.786E+04 6.786E+04 0 Message size sent to node 9 : gather => MPI_Isend() 1 2.896E+04 2.896E+04 2.896E+04 0 Message size sent to node 9 : make_ggl => MPI_Isend() 3071 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 9 : scatter => MPI_Isend() 1 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 1 7.538E+04 7.538E+04 7.538E+04 0 [GROUP=MAX_MARKER] Message size sent to node 11 1 7.538E+04 7.538E+04 7.538E+04 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to node 9 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.603E+04 2.603E+04 2.603E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.603E+04 2.603E+04 2.603E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 11;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 62.3 4:52.342 4:52.342 1.12021E+07 0 26 MPI_Win_unlock() 22.0 1:42.984 1:42.984 17279 0 5960 MPI_Barrier() 99.6 39,068 7:46.949 1 503146 466949346 xx14std 55.9 14,083 4:21.979 511 1.12372E+07 512679 cgca_hxi 50.0 4,892 3:54.249 5 269768 46849943 cgca_clvgp_nocosum 1.2 2,672 5,416 3071 36852 1764 gather 0.5 2,407 2,407 15338 0 157 MPI_Allreduce() 0.5 2,381 2,381 6 0 396981 MPI_Send() 0.3 1,583 1,583 1 0 1583570 MPI_Init() 1.1 1,342 5,179 3071 33781 1687 scatter 0.2 870 870 7 0 124349 MPI_File_write_all() 0.1 611 611 12289 0 50 MPI_Recv() 0.1 578 578 135 0 4283 MPI_Bcast() 30.2 420 2:21.750 1 51201 141750356 cgca_sld 0.1 321 321 12288 0 26 MPI_Probe() 0.4 306 1,802 9198 9198 196 dot_product_p 0.1 274 274 12288 0 22 MPI_Isend() 0.1 250 273 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 183 1,090 3066 6132 356 checon_par 100.0 178 7:48.906 1 450 468906121 .TAU application 0.0 164 164 1 0 164440 MPI_Finalize() 0.0 102 102 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 91 91 155 0 589 cgca_dacf 0.2 81 1,138 5 30000 227754 cgca_pfem_intcalc1 5.8 79 26,968 5 20000 5393729 cgca_pfem_simg 0.0 65 65 100001 0 1 beemat [THROTTLED] 0.0 55 55 100001 0 1 MPI_Get() [THROTTLED] 0.0 51 51 7 0 7355 MPI_File_open() 0.0 46 46 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21963 cgca_gbs 0.0 39 39 100001 0 0 shape_der [THROTTLED] 1.0 38 4,572 12284 12284 372 my_barrier 0.0 32 32 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 27 27 67 0 413 MPI_Reduce() 0.0 25 25 13 0 1956 MPI_Win_create() 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 22 22 6144 0 4 MPI_Waitall() 0.0 21 22 1 1 22405 load 0.0 20 20 80000 0 0 determinant 0.0 9 9 1 0 9227 cgca_igb 0.0 8 8 10064 0 1 MPI_Put() 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 8 8 7 0 1182 MPI_File_close() 0.0 7 7 7 0 1030 MPI_File_set_view() 0.0 6 14 1 19 14225 make_ggl 0.0 5 5 1 27 5655 cgca_gcu 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4977 allocate_gather_scatter 0.0 4 88 155 1706 573 cgca_gcupdn 0.0 4 47 1 160 47285 scatter_nodes 0.1 4 479 1 33 479455 read_g_coord_pp 0.0 3 3 12 0 267 MPI_Win_free() 0.0 2 2 5 0 419 cgca_fv 0.0 1 59 1 576 59376 cgca_pfem_cenc 0.4 0.772 1,733 7 525 247619 cgca_pswci 0.0 0.761 0.761 1 0 761 getname 0.0 0.623 1 1 2 1669 cgca_as 0.0 0.416 0.416 1 0 416 MPI_Comm_dup() 0.0 0.33 0.669 1 9 669 scatter_noadd 0.2 0.268 738 1 64 738828 cgca_ds 0.0 0.235 0.235 1 0 235 calc_nels_pp 0.0 0.203 0.203 1 0 203 rearrange 0.0 0.168 7 1 374 7874 cgca_nr 0.0 0.168 0.168 1 0 168 cgca_pfem_salloc 0.0 0.146 0.146 14 0 10 MPI_Type_commit() 0.0 0.133 5 1 377 5926 cgca_rt 0.0 0.118 0.118 1 0 118 cgca_ins 0.1 0.112 314 1 2 314072 read_g_num_pp 0.0 0.097 4 1 65 4416 gcupd_alloc 0.0 0.093 9 1 64 9198 cgca_drt 0.0 0.074 0.074 5 0 15 cgca_pfem_uym 0.0 0.063 1 5 5 288 sum_p 0.0 0.049 0.049 7 0 7 elap_time 0.0 0.048 0.048 1 0 48 cgca_cadim 0.0 0.042 0.042 1 0 42 cgca_pfem_sdalloc 0.0 0.04 0.04 14 0 3 MPI_Type_create_subarray() 0.0 0.04 0.04 1 0 40 abaqus2sg 0.0 0.037 0.037 1 0 37 cgca_gdim 0.5 0.034 2,381 3 6 793974 dismsh_ensi_p 0.0 0.031 0.031 1 0 31 calc_neq_pp 0.0 0.022 0.022 5 0 4 sample 0.0 0.018 0.018 1 0 18 cgca_pfem_ctalloc 0.0 0.016 0.016 1 2 16 find_pe_procs 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.015 12 0 1 MPI_Win_fence() 0.0 0.012 0.012 2 0 6 MPI_Test() 0.0 0.012 0.012 14 0 1 MPI_Type_free() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.007 0.007 14 0 0 cgca_agc 0.0 0.007 7 1 2 7507 cgca_art 0.0 0.006 0.006 1 0 6 MPI_Alloc_mem() 0.0 0.006 0.006 9 0 1 MPI_Comm_rank() 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.006 3 1 3 3927 read_p121 0.0 0.005 3 1 2 3756 read_loads 0.0 0.003 0.003 13 0 0 cgca_dgc 0.0 0.003 0.003 1 0 3 cgca_pfem_integalloc 0.0 0.003 79 1 1 79531 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.189 1 1 189 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_gcf 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ctdalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 11, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.19 2.659 3.948 0.8067 MPI-IO Write Bandwidth (MB/s) 7 5.19 2.659 3.948 0.8067 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.662E+04 6.083E+04 9394 Message size received from all nodes 1.229E+04 7.538E+04 4 6.08E+04 9398 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 6144 7.538E+04 2.603E+04 6.372E+04 1.167E+04 Message size sent to node 10 3071 7.538E+04 7.538E+04 7.538E+04 0 Message size sent to node 10 : gather => MPI_Isend() 1 2.603E+04 2.603E+04 2.603E+04 0 Message size sent to node 10 : make_ggl => MPI_Isend() 3071 5.206E+04 5.206E+04 5.206E+04 0 Message size sent to node 10 : scatter => MPI_Isend() 1 5.206E+04 5.206E+04 5.206E+04 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 6144 6.259E+04 3.13E+04 5.792E+04 4684 Message size sent to node 12 3071 5.325E+04 5.325E+04 5.325E+04 0 Message size sent to node 12 : gather => MPI_Isend() 1 3.13E+04 3.13E+04 3.13E+04 0 Message size sent to node 12 : make_ggl => MPI_Isend() 3071 6.259E+04 6.259E+04 6.259E+04 0 Message size sent to node 12 : scatter => MPI_Isend() 1 6.259E+04 6.259E+04 6.259E+04 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 5.206E+04 5.206E+04 5.206E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 5.206E+04 5.206E+04 5.206E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 1 5.206E+04 5.206E+04 5.206E+04 0 [GROUP=MAX_MARKER] Message size sent to node 10 1 5.206E+04 5.206E+04 5.206E+04 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 1 6.259E+04 6.259E+04 6.259E+04 0 [GROUP=MAX_MARKER] Message size sent to node 12 1 6.259E+04 6.259E+04 6.259E+04 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.769E+04 3.769E+04 3.769E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.769E+04 3.769E+04 3.769E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 12;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 59.1 4:36.964 4:36.964 8.01279E+06 0 35 MPI_Win_unlock() 26.0 2:02.104 2:02.104 17279 0 7067 MPI_Barrier() 99.6 38,964 7:46.949 1 503146 466949379 xx14std 46.5 10,788 3:37.931 511 8.04756E+06 426481 cgca_hxi 50.0 4,441 3:54.249 5 268037 46849823 cgca_clvgp_nocosum 1.2 2,690 5,466 3071 36852 1780 gather 0.5 2,432 2,432 15338 0 159 MPI_Allreduce() 0.5 2,413 2,413 6 0 402183 MPI_Send() 0.3 1,583 1,583 1 0 1583191 MPI_Init() 1.1 1,367 5,220 3071 33781 1700 scatter 0.2 870 870 7 0 124366 MPI_File_write_all() 30.2 664 2:21.750 1 52197 141750340 cgca_sld 0.1 571 571 135 0 4234 MPI_Bcast() 0.1 418 418 12289 0 34 MPI_Recv() 0.1 319 319 12288 0 26 MPI_Isend() 0.4 296 1,805 9198 9198 196 dot_product_p 0.1 274 274 12288 0 22 MPI_Probe() 0.1 233 256 98270 100001 3 cgca_clvgn 100.0 178 7:48.905 1 416 468905905 .TAU application 0.2 168 1,088 3066 6132 355 checon_par 0.0 164 164 1 0 164484 MPI_Finalize() 0.0 155 155 100001 0 2 MPI_Win_lock() [THROTTLED] 0.0 93 93 155 0 603 cgca_dacf 5.6 75 26,426 5 20000 5285276 cgca_pfem_simg 0.0 74 74 100001 0 1 beemat [THROTTLED] 0.0 55 55 100001 0 1 MPI_Get() [THROTTLED] 0.0 51 51 7 0 7413 MPI_File_open() 0.0 47 47 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21938 cgca_gbs 1.0 42 4,808 12284 12284 391 my_barrier 0.0 36 36 100001 0 0 shape_der [THROTTLED] 0.0 33 33 100001 0 0 cgca_clvgsd [THROTTLED] 0.2 30 753 5 30000 150644 cgca_pfem_intcalc1 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 28 28 67 0 419 MPI_Reduce() 0.0 26 26 80000 0 0 determinant 0.0 24 24 13 0 1876 MPI_Win_create() 0.0 22 22 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22313 load 0.0 19 19 6144 0 3 MPI_Waitall() 0.0 10 10 1 0 10173 cgca_igb 0.0 7 7 7 0 1086 MPI_File_close() 0.0 7 7 12288 0 1 MPI_Get_count() 0.0 7 7 10052 0 1 MPI_Put() 0.0 6 6 7 0 995 MPI_File_set_view() 0.0 6 14 1 19 14061 make_ggl 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4708 allocate_gather_scatter 0.0 4 47 1 160 47695 scatter_nodes 0.0 4 4 1 3 4194 cgca_gcu 0.1 4 472 1 33 472464 read_g_coord_pp 0.0 3 62 155 1091 406 cgca_gcupdn 0.0 3 3 12 0 261 MPI_Win_free() 0.0 2 2 5 0 416 cgca_fv 0.0 1 59 1 576 59979 cgca_pfem_cenc 0.4 0.828 1,733 7 525 247587 cgca_pswci 0.0 0.795 0.795 1 0 795 getname 0.0 0.622 1 1 2 1582 cgca_as 0.0 0.54 0.54 1 0 540 MPI_Comm_dup() 0.0 0.341 0.491 1 9 491 scatter_noadd 0.2 0.267 707 1 64 707118 cgca_ds 0.0 0.203 0.203 1 0 203 rearrange 0.0 0.175 7 1 374 7846 cgca_nr 0.0 0.175 0.175 1 0 175 cgca_pfem_salloc 0.0 0.133 0.133 1 0 133 calc_nels_pp 0.0 0.13 0.13 14 0 9 MPI_Type_commit() 0.0 0.128 0.128 1 0 128 cgca_ins 0.0 0.127 5 1 377 5974 cgca_rt 0.1 0.119 321 1 2 321191 read_g_num_pp 0.0 0.091 4 1 65 4267 gcupd_alloc 0.0 0.08 0.08 5 0 16 cgca_pfem_uym 0.0 0.078 9 1 64 9196 cgca_drt 0.0 0.054 0.054 1 0 54 cgca_gdim 0.0 0.051 1 5 5 246 sum_p 0.0 0.041 0.041 14 0 3 MPI_Type_create_subarray() 0.0 0.041 0.041 1 0 41 abaqus2sg 0.0 0.04 0.04 1 0 40 cgca_pfem_sdalloc 0.0 0.034 0.034 14 0 2 MPI_Type_free() 0.5 0.034 2,413 3 6 804378 dismsh_ensi_p 0.0 0.032 0.032 7 0 5 elap_time 0.0 0.03 0.03 1 0 30 cgca_cadim 0.0 0.029 0.029 5 0 6 sample 0.0 0.021 0.022 1 2 22 find_pe_procs 0.0 0.018 0.018 9 0 2 MPI_Comm_rank() 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.015 12 0 1 MPI_Win_fence() 0.0 0.015 0.015 1 0 15 calc_neq_pp 0.0 0.014 0.014 1 0 14 cgca_pfem_ctalloc 0.0 0.012 0.012 2 0 6 MPI_Test() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.006 3 1 3 3961 read_p121 0.0 0.005 0.005 9 0 1 MPI_Comm_size() 0.0 0.005 6 1 2 6934 cgca_art 0.0 0.005 3 1 2 3802 read_loads 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 0.004 6 0 1 cgca_gcf 0.0 0.004 0.164 1 1 164 max_p 0.0 0.003 0.003 1 0 3 MPI_Finalized() 0.0 0.003 79 1 1 79986 read_rest 0.0 0.002 0.002 1 0 2 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 2 0 0 cgca_agc 0.0 0.001 0.001 1 0 1 cgca_dgc 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 12, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.195 2.662 3.947 0.8069 MPI-IO Write Bandwidth (MB/s) 7 5.195 2.662 3.947 0.8069 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 3.13E+04 6.107E+04 4662 Message size received from all nodes 1.229E+04 6.766E+04 4 6.105E+04 4677 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 6144 6.259E+04 2.662E+04 5.791E+04 4689 Message size sent to node 11 3071 6.259E+04 6.259E+04 6.259E+04 0 Message size sent to node 11 : gather => MPI_Isend() 1 2.662E+04 2.662E+04 2.662E+04 0 Message size sent to node 11 : make_ggl => MPI_Isend() 3071 5.325E+04 5.325E+04 5.325E+04 0 Message size sent to node 11 : scatter => MPI_Isend() 1 5.325E+04 5.325E+04 5.325E+04 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 6144 6.43E+04 3.215E+04 6.42E+04 420 Message size sent to node 13 3071 6.411E+04 6.411E+04 6.411E+04 0 Message size sent to node 13 : gather => MPI_Isend() 1 3.215E+04 3.215E+04 3.215E+04 0 Message size sent to node 13 : make_ggl => MPI_Isend() 3071 6.43E+04 6.43E+04 6.43E+04 0 Message size sent to node 13 : scatter => MPI_Isend() 1 6.43E+04 6.43E+04 6.43E+04 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 5.325E+04 5.325E+04 5.325E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 5.325E+04 5.325E+04 5.325E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 1 5.325E+04 5.325E+04 5.325E+04 0 [GROUP=MAX_MARKER] Message size sent to node 11 1 5.325E+04 5.325E+04 5.325E+04 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 1 6.43E+04 6.43E+04 6.43E+04 0 [GROUP=MAX_MARKER] Message size sent to node 13 1 6.43E+04 6.43E+04 6.43E+04 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.13E+04 3.13E+04 3.13E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.13E+04 3.13E+04 3.13E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 13;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 59.9 4:41.038 4:41.038 1.12022E+07 0 25 MPI_Win_unlock() 24.1 1:53.167 1:53.167 17279 0 6549 MPI_Barrier() 99.6 38,995 7:46.949 1 503146 466949347 xx14std 53.0 14,292 4:08.463 511 1.12372E+07 486231 cgca_hxi 50.0 5,730 3:54.249 5 269768 46849922 cgca_clvgp_nocosum 1.2 2,641 5,457 3071 36852 1777 gather 0.5 2,444 2,444 6 0 407459 MPI_Send() 0.5 2,365 2,365 15338 0 154 MPI_Allreduce() 0.3 1,583 1,583 1 0 1583639 MPI_Init() 1.1 1,337 5,253 3071 33781 1711 scatter 0.2 870 870 7 0 124402 MPI_File_write_all() 30.2 712 2:21.750 1 51201 141750410 cgca_sld 0.1 564 564 135 0 4183 MPI_Bcast() 0.1 429 429 12289 0 35 MPI_Recv() 0.1 346 346 12288 0 28 MPI_Probe() 0.1 326 326 12288 0 27 MPI_Isend() 0.4 306 1,758 9198 9198 191 dot_product_p 0.1 234 257 100001 100001 3 cgca_clvgn [THROTTLED] 100.0 178 7:48.899 1 540 468899117 .TAU application 0.2 178 1,088 3066 6132 355 checon_par 0.0 157 157 1 0 157435 MPI_Finalize() 0.0 157 157 100001 0 2 MPI_Win_lock() [THROTTLED] 0.0 89 89 155 0 579 cgca_dacf 0.0 71 71 100001 0 1 beemat [THROTTLED] 5.6 70 26,396 5 20000 5279233 cgca_pfem_simg 0.0 52 52 7 0 7483 MPI_File_open() 0.0 51 51 100001 0 1 MPI_Get() [THROTTLED] 1.0 44 4,863 12284 12284 396 my_barrier 0.0 44 44 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21940 cgca_gbs 0.0 38 38 100001 0 0 shape_der [THROTTLED] 0.2 33 766 5 30000 153268 cgca_pfem_intcalc1 0.0 32 32 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 27 27 67 0 415 MPI_Reduce() 0.0 24 24 13 0 1857 MPI_Win_create() 0.0 22 22 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22302 load 0.0 20 20 80000 0 0 determinant 0.0 20 20 10074 0 2 MPI_Put() 0.0 18 18 6144 0 3 MPI_Waitall() 0.0 9 9 1 0 9600 cgca_igb 0.0 7 7 7 0 1014 MPI_File_close() 0.0 6 6 12288 0 1 MPI_Get_count() 0.0 6 6 7 0 985 MPI_File_set_view() 0.0 6 14 1 19 14213 make_ggl 0.0 5 5 2000 0 3 find_g3 0.0 5 99 155 1711 640 cgca_gcupdn 0.0 4 47 1 160 47752 scatter_nodes 0.0 4 4 1 15 4777 cgca_gcu 0.0 4 4 1 0 4553 allocate_gather_scatter 0.1 4 465 1 33 465499 read_g_coord_pp 0.0 3 3 12 0 263 MPI_Win_free() 0.0 2 2 5 0 414 cgca_fv 0.0 1 60 1 576 60805 cgca_pfem_cenc 0.4 0.862 1,732 7 525 247558 cgca_pswci 0.0 0.772 0.772 1 0 772 getname 0.0 0.629 1 1 2 1568 cgca_as 0.0 0.537 0.537 1 0 537 MPI_Comm_dup() 0.0 0.33 0.468 1 9 468 scatter_noadd 0.1 0.263 675 1 64 675463 cgca_ds 0.0 0.215 0.215 1 0 215 cgca_pfem_salloc 0.0 0.202 0.202 1 0 202 rearrange 0.0 0.191 7 1 374 7900 cgca_nr 0.0 0.147 0.147 1 0 147 calc_nels_pp 0.0 0.14 0.14 14 0 10 MPI_Type_commit() 0.0 0.127 5 1 377 5928 cgca_rt 0.0 0.116 0.116 1 0 116 cgca_ins 0.1 0.107 328 1 2 328161 read_g_num_pp 0.0 0.094 0.094 5 0 19 cgca_pfem_uym 0.0 0.079 9 1 64 9196 cgca_drt 0.0 0.074 4 1 65 4472 gcupd_alloc 0.0 0.047 1 5 5 290 sum_p 0.0 0.046 0.046 1 0 46 cgca_gdim 0.0 0.042 0.042 7 0 6 elap_time 0.0 0.04 0.04 14 0 3 MPI_Type_create_subarray() 0.0 0.04 0.04 1 0 40 abaqus2sg 0.0 0.035 0.035 1 0 35 cgca_pfem_sdalloc 0.5 0.034 2,444 3 6 814930 dismsh_ensi_p 0.0 0.02 0.02 5 0 4 sample 0.0 0.016 0.016 1 0 16 calc_neq_pp 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.015 1 0 15 cgca_pfem_ctalloc 0.0 0.013 0.013 2 0 6 MPI_Test() 0.0 0.013 0.014 1 2 14 find_pe_procs 0.0 0.011 0.011 12 0 1 MPI_Win_fence() 0.0 0.01 0.01 14 0 1 MPI_Type_free() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.008 0.008 1 0 8 cgca_cadim 0.0 0.007 0.007 9 0 1 MPI_Comm_rank() 0.0 0.007 6 1 2 6175 cgca_art 0.0 0.007 0.007 6 0 1 cgca_gcf 0.0 0.007 3 1 3 3991 read_p121 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.005 3 1 2 3741 read_loads 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.003 0.003 8 0 0 cgca_agc 0.0 0.003 0.003 7 0 0 cgca_dgc 0.0 0.003 0.215 1 1 215 max_p 0.0 0.003 79 1 1 79982 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 13, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.186 2.659 3.946 0.8057 MPI-IO Write Bandwidth (MB/s) 7 5.186 2.659 3.946 0.8057 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.896E+04 6.237E+04 2779 Message size received from all nodes 1.229E+04 6.766E+04 4 6.235E+04 2808 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 6144 6.43E+04 3.206E+04 6.42E+04 421.2 Message size sent to node 12 3071 6.43E+04 6.43E+04 6.43E+04 0 Message size sent to node 12 : gather => MPI_Isend() 1 3.206E+04 3.206E+04 3.206E+04 0 Message size sent to node 12 : make_ggl => MPI_Isend() 3071 6.411E+04 6.411E+04 6.411E+04 0 Message size sent to node 12 : scatter => MPI_Isend() 1 6.411E+04 6.411E+04 6.411E+04 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 6144 6.312E+04 3.156E+04 6.052E+04 2626 Message size sent to node 14 3071 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 14 : gather => MPI_Isend() 1 3.156E+04 3.156E+04 3.156E+04 0 Message size sent to node 14 : make_ggl => MPI_Isend() 3071 6.312E+04 6.312E+04 6.312E+04 0 Message size sent to node 14 : scatter => MPI_Isend() 1 6.312E+04 6.312E+04 6.312E+04 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 6.411E+04 6.411E+04 6.411E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 6.411E+04 6.411E+04 6.411E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 1 6.411E+04 6.411E+04 6.411E+04 0 [GROUP=MAX_MARKER] Message size sent to node 12 1 6.411E+04 6.411E+04 6.411E+04 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 1 6.312E+04 6.312E+04 6.312E+04 0 [GROUP=MAX_MARKER] Message size sent to node 14 1 6.312E+04 6.312E+04 6.312E+04 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.215E+04 3.215E+04 3.215E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.215E+04 3.215E+04 3.215E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 14;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 60.7 4:44.612 4:44.612 1.12021E+07 0 25 MPI_Win_unlock() 23.5 1:50.378 1:50.378 17279 0 6388 MPI_Barrier() 99.6 38,969 7:46.949 1 503146 466949399 xx14std 54.1 14,095 4:13.787 511 1.12372E+07 496649 cgca_hxi 50.0 4,574 3:54.249 5 269768 46849839 cgca_clvgp_nocosum 1.2 2,711 5,451 3071 36852 1775 gather 0.5 2,476 2,476 6 0 412725 MPI_Send() 0.5 2,410 2,410 15338 0 157 MPI_Allreduce() 0.3 1,583 1,583 1 0 1583683 MPI_Init() 1.1 1,366 5,244 3071 33781 1708 scatter 0.2 872 872 7 0 124675 MPI_File_write_all() 0.2 735 735 12289 0 60 MPI_Recv() 30.2 688 2:21.750 1 51201 141750438 cgca_sld 0.1 558 558 135 0 4134 MPI_Bcast() 0.1 333 333 12288 0 27 MPI_Probe() 0.4 307 1,802 9198 9198 196 dot_product_p 0.1 282 282 12288 0 23 MPI_Isend() 0.1 247 270 100001 100001 3 cgca_clvgn [THROTTLED] 0.0 208 208 6144 0 34 MPI_Waitall() 100.0 178 7:48.899 1 472 468899970 .TAU application 0.2 174 1,086 3066 6132 354 checon_par 0.0 158 158 1 0 158283 MPI_Finalize() 0.0 155 155 100001 0 2 MPI_Win_lock() [THROTTLED] 0.0 91 91 155 0 592 cgca_dacf 5.7 70 26,564 5 20000 5312892 cgca_pfem_simg 0.0 67 67 100001 0 1 beemat [THROTTLED] 0.2 54 780 5 30000 156020 cgca_pfem_intcalc1 0.0 54 54 100001 0 1 MPI_Get() [THROTTLED] 0.0 52 52 7 0 7444 MPI_File_open() 0.0 46 46 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21972 cgca_gbs 1.0 38 4,619 12284 12284 376 my_barrier 0.0 38 38 100001 0 0 shape_der [THROTTLED] 0.0 32 32 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 30 30 100001 0 0 invert [THROTTLED] 0.0 28 28 67 0 422 MPI_Reduce() 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 21 21 13 0 1687 MPI_Win_create() 0.0 21 22 1 1 22435 load 0.0 20 20 80000 0 0 determinant 0.0 9 9 1 0 9494 cgca_igb 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 7 7 10072 0 1 MPI_Put() 0.0 6 14 1 19 14122 make_ggl 0.0 6 6 7 0 938 MPI_File_set_view() 0.0 5 5 1 19 5510 cgca_gcu 0.0 5 5 2000 0 3 find_g3 0.0 4 87 155 1710 563 cgca_gcupdn 0.0 4 4 7 0 664 MPI_File_close() 0.0 4 4 1 0 4625 allocate_gather_scatter 0.1 4 458 1 33 458444 read_g_coord_pp 0.0 3 47 1 160 47720 scatter_nodes 0.0 2 2 12 0 240 MPI_Win_free() 0.0 2 2 5 0 410 cgca_fv 0.0 1 62 1 576 62278 cgca_pfem_cenc 0.0 0.781 0.781 1 0 781 getname 0.4 0.78 1,732 7 525 247447 cgca_pswci 0.0 0.617 1 1 2 1587 cgca_as 0.0 0.558 0.558 1 0 558 MPI_Comm_dup() 0.0 0.347 0.601 1 9 601 scatter_noadd 0.1 0.289 643 1 64 643823 cgca_ds 0.0 0.218 0.218 1 0 218 cgca_pfem_salloc 0.0 0.203 0.203 1 0 203 rearrange 0.0 0.165 7 1 374 7963 cgca_nr 0.0 0.149 0.149 1 0 149 calc_nels_pp 0.0 0.142 0.142 1 0 142 cgca_ins 0.0 0.134 0.134 14 0 10 MPI_Type_commit() 0.0 0.134 5 1 377 5823 cgca_rt 0.1 0.115 335 1 2 335199 read_g_num_pp 0.0 0.101 9 1 64 9198 cgca_drt 0.0 0.098 0.098 5 0 20 cgca_pfem_uym 0.0 0.083 4 1 65 4168 gcupd_alloc 0.0 0.055 1 5 5 281 sum_p 0.0 0.053 0.053 1 0 53 cgca_gdim 0.0 0.047 0.047 14 0 3 MPI_Type_create_subarray() 0.0 0.041 0.041 1 0 41 abaqus2sg 0.0 0.041 0.041 1 0 41 cgca_pfem_sdalloc 0.0 0.036 0.036 7 0 5 elap_time 0.5 0.035 2,476 3 6 825461 dismsh_ensi_p 0.0 0.032 0.032 1 0 32 cgca_cadim 0.0 0.026 0.026 12 0 2 MPI_Win_fence() 0.0 0.025 0.025 1 0 25 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.023 0.023 1 0 23 calc_neq_pp 0.0 0.02 0.02 14 0 1 MPI_Type_free() 0.0 0.019 0.019 5 0 4 sample 0.0 0.017 0.017 1 2 17 find_pe_procs 0.0 0.015 0.015 1 0 15 cgca_pfem_ctalloc 0.0 0.012 0.012 5 0 2 cgca_gcf 0.0 0.011 0.011 2 0 6 MPI_Test() 0.0 0.01 0.01 1 0 10 caflaunch$MAIN$blk 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.005 0.005 9 0 1 MPI_Comm_rank() 0.0 0.005 0.005 10 0 0 cgca_agc 0.0 0.005 4 1 2 4644 cgca_art 0.0 0.005 3 1 2 3824 read_loads 0.0 0.005 4 1 3 4051 read_p121 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.003 0.158 1 1 158 max_p 0.0 0.003 79 1 1 79998 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 9 0 0 cgca_dgc 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_edalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc 0.0 0 0 1 0 0 cgca_imco 0.0 0 0 1 0 0 cgca_pfem_ealloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 14, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.185 2.655 3.937 0.8048 MPI-IO Write Bandwidth (MB/s) 7 5.185 2.655 3.937 0.8048 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.84E+04 6.212E+04 5555 Message size received from all nodes 1.229E+04 7.065E+04 4 6.21E+04 5567 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 6144 6.312E+04 2.896E+04 6.051E+04 2631 Message size sent to node 13 3071 6.312E+04 6.312E+04 6.312E+04 0 Message size sent to node 13 : gather => MPI_Isend() 1 2.896E+04 2.896E+04 2.896E+04 0 Message size sent to node 13 : make_ggl => MPI_Isend() 3071 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 13 : scatter => MPI_Isend() 1 5.792E+04 5.792E+04 5.792E+04 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 6144 7.065E+04 3.532E+04 6.372E+04 6933 Message size sent to node 15 3071 5.68E+04 5.68E+04 5.68E+04 0 Message size sent to node 15 : gather => MPI_Isend() 1 3.532E+04 3.532E+04 3.532E+04 0 Message size sent to node 15 : make_ggl => MPI_Isend() 3071 7.065E+04 7.065E+04 7.065E+04 0 Message size sent to node 15 : scatter => MPI_Isend() 1 7.065E+04 7.065E+04 7.065E+04 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to node 13 1 5.792E+04 5.792E+04 5.792E+04 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 1 7.065E+04 7.065E+04 7.065E+04 0 [GROUP=MAX_MARKER] Message size sent to node 15 1 7.065E+04 7.065E+04 7.065E+04 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.84E+04 2.84E+04 2.84E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.84E+04 2.84E+04 2.84E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 15;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 60.3 4:42.558 4:42.558 8.01283E+06 0 35 MPI_Win_unlock() 24.6 1:55.285 1:55.285 17279 0 6672 MPI_Barrier() 99.6 38,884 7:46.949 1 503146 466949380 xx14std 50.0 10,734 3:54.231 511 8.09085E+06 458378 cgca_hxi 50.0 3,661 3:54.249 5 269768 46849951 cgca_clvgp_nocosum 1.2 2,634 5,411 3071 36852 1762 gather 0.5 2,507 2,507 6 0 417906 MPI_Send() 0.5 2,395 2,395 15338 0 156 MPI_Allreduce() 0.3 1,582 1,582 1 0 1582101 MPI_Init() 0.3 1,440 1,440 144037 0 10 MPI_Win_lock() [THROTTLED] 1.1 1,372 5,365 3071 33781 1747 scatter 0.2 871 871 7 0 124542 MPI_File_write_all() 0.2 798 798 12288 0 65 MPI_Isend() 30.2 664 2:21.750 1 52944 141750480 cgca_sld 0.1 606 606 12289 0 49 MPI_Recv() 0.1 551 551 135 0 4082 MPI_Bcast() 0.1 456 456 12288 0 37 MPI_Probe() 0.4 306 1,795 9198 9198 195 dot_product_p 0.1 228 257 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 184 1,087 3066 6132 355 checon_par 100.0 179 7:48.896 1 475 468896950 .TAU application 0.0 155 155 1 0 155568 MPI_Finalize() 0.0 90 90 155 0 587 cgca_dacf 5.7 67 26,578 5 20000 5315770 cgca_pfem_simg 0.0 67 67 100001 0 1 beemat [THROTTLED] 0.0 55 55 100001 0 1 MPI_Get() [THROTTLED] 0.0 52 52 7 0 7465 MPI_File_open() 0.0 46 46 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21941 cgca_gbs 0.0 38 38 100001 0 0 invert [THROTTLED] 0.0 38 38 100001 0 0 shape_der [THROTTLED] 1.0 37 4,492 12284 12284 366 my_barrier 0.2 33 763 5 30000 152796 cgca_pfem_intcalc1 0.0 32 32 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 29 29 100001 0 0 cgca_csym [THROTTLED] 0.0 28 28 67 0 432 MPI_Reduce() 0.0 24 24 6144 0 4 MPI_Waitall() 0.0 22 22 13 0 1741 MPI_Win_create() 0.0 21 22 1 1 22456 load 0.0 20 20 80000 0 0 determinant 0.0 10 10 1 0 10103 cgca_igb 0.0 7 7 10073 0 1 MPI_Put() 0.0 7 7 12288 0 1 MPI_Get_count() 0.0 6 6 7 0 941 MPI_File_set_view() 0.0 6 14 1 19 14193 make_ggl 0.0 6 6 7 0 893 MPI_File_close() 0.0 5 5 2000 0 3 find_g3 0.1 4 451 1 33 451804 read_g_coord_pp 0.0 4 4 1 0 4535 allocate_gather_scatter 0.0 4 4 1 7 4529 cgca_gcu 0.0 4 48 1 160 48361 scatter_nodes 0.0 3 70 155 1085 452 cgca_gcupdn 0.0 3 3 12 0 261 MPI_Win_free() 0.0 2 2 5 0 408 cgca_fv 0.0 1 62 1 576 62128 cgca_pfem_cenc 0.0 0.794 0.794 1 0 794 getname 0.4 0.792 1,732 7 525 247479 cgca_pswci 0.0 0.621 1 1 2 1641 cgca_as 0.0 0.438 0.438 1 0 438 MPI_Comm_dup() 0.0 0.336 0.712 1 9 712 scatter_noadd 0.1 0.239 612 1 64 612123 cgca_ds 0.0 0.208 0.208 1 0 208 rearrange 0.0 0.205 0.205 1 0 205 calc_nels_pp 0.0 0.198 8 1 374 8037 cgca_nr 0.0 0.184 0.184 1 0 184 cgca_pfem_salloc 0.0 0.152 5 1 377 5809 cgca_rt 0.0 0.129 0.129 14 0 9 MPI_Type_commit() 0.1 0.111 342 1 2 342145 read_g_num_pp 0.0 0.085 4 1 65 4425 gcupd_alloc 0.0 0.083 0.083 1 0 83 cgca_ins 0.0 0.073 0.073 5 0 15 cgca_pfem_uym 0.0 0.061 9 1 64 9199 cgca_drt 0.0 0.053 0.053 1 0 53 cgca_gdim 0.0 0.046 1 5 5 319 sum_p 0.0 0.04 0.04 1 0 40 abaqus2sg 0.0 0.039 0.039 14 0 3 MPI_Type_create_subarray() 0.0 0.038 0.038 1 0 38 cgca_pfem_sdalloc 0.0 0.038 0.038 7 0 5 elap_time 0.0 0.031 0.031 1 0 31 cgca_cadim 0.0 0.027 0.027 1 0 27 calc_neq_pp 0.5 0.027 2,507 3 6 835820 dismsh_ensi_p 0.0 0.021 0.022 1 2 22 find_pe_procs 0.0 0.02 0.02 5 0 4 sample 0.0 0.014 0.014 1 0 14 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.014 0.014 14 0 1 MPI_Type_free() 0.0 0.014 0.014 1 0 14 cgca_pfem_ctalloc 0.0 0.013 0.013 12 0 1 MPI_Win_fence() 0.0 0.012 0.012 2 0 6 MPI_Test() 0.0 0.01 4 1 3 4058 read_p121 0.0 0.009 0.009 9 0 1 MPI_Comm_rank() 0.0 0.007 0.007 1 0 7 caflaunch$MAIN$blk 0.0 0.007 4 1 2 4823 cgca_art 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.005 3 1 2 3791 read_loads 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.003 0.003 1 0 3 cgca_pfem_integalloc 0.0 0.003 80 1 1 80269 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 4 0 0 cgca_agc 0.0 0.002 0.002 3 0 1 cgca_dgc 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.111 1 1 111 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 15, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.183 2.659 3.941 0.8044 MPI-IO Write Bandwidth (MB/s) 7 5.183 2.659 3.941 0.8044 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.899E+04 6.083E+04 5773 Message size received from all nodes 1.229E+04 7.065E+04 4 6.08E+04 5784 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 6144 7.065E+04 2.84E+04 6.372E+04 6938 Message size sent to node 14 3071 7.065E+04 7.065E+04 7.065E+04 0 Message size sent to node 14 : gather => MPI_Isend() 1 2.84E+04 2.84E+04 2.84E+04 0 Message size sent to node 14 : make_ggl => MPI_Isend() 3071 5.68E+04 5.68E+04 5.68E+04 0 Message size sent to node 14 : scatter => MPI_Isend() 1 5.68E+04 5.68E+04 5.68E+04 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 6144 5.798E+04 2.893E+04 5.792E+04 375.3 Message size sent to node 16 3071 5.798E+04 5.798E+04 5.798E+04 0 Message size sent to node 16 : gather => MPI_Isend() 1 2.893E+04 2.893E+04 2.893E+04 0 Message size sent to node 16 : make_ggl => MPI_Isend() 3071 5.786E+04 5.786E+04 5.786E+04 0 Message size sent to node 16 : scatter => MPI_Isend() 1 5.786E+04 5.786E+04 5.786E+04 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 5.68E+04 5.68E+04 5.68E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 5.68E+04 5.68E+04 5.68E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 1 5.68E+04 5.68E+04 5.68E+04 0 [GROUP=MAX_MARKER] Message size sent to node 14 1 5.68E+04 5.68E+04 5.68E+04 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 1 5.786E+04 5.786E+04 5.786E+04 0 [GROUP=MAX_MARKER] Message size sent to node 16 1 5.786E+04 5.786E+04 5.786E+04 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.899E+04 2.899E+04 2.899E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.899E+04 2.899E+04 2.899E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 16;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 61.2 4:46.889 4:46.889 8.01286E+06 0 36 MPI_Win_unlock() 23.3 1:49.262 1:49.262 17279 0 6323 MPI_Barrier() 99.7 39,154 7:46.947 1 503146 466947682 xx14std 55.2 13,844 4:18.746 511 8.04756E+06 506354 cgca_hxi 50.0 5,705 3:54.250 5 269768 46850054 cgca_clvgp_nocosum 1.2 2,641 5,428 3071 36852 1768 gather 0.5 2,435 2,435 15338 0 159 MPI_Allreduce() 1.1 1,368 5,051 3071 33781 1645 scatter 0.2 959 959 1 0 959685 MPI_Init() 0.2 868 868 7 0 124077 MPI_File_write_all() 0.2 855 855 12288 0 70 MPI_Isend() 0.2 788 788 12289 0 64 MPI_Recv() 0.1 540 540 135 0 4005 MPI_Bcast() 30.3 417 2:21.740 1 52197 141740504 cgca_sld 0.1 328 328 12288 0 27 MPI_Probe() 0.4 312 1,827 9198 9198 199 dot_product_p 0.1 294 294 1 0 294194 MPI_Finalize() 0.1 225 247 100001 100001 2 cgca_clvgn [THROTTLED] 0.2 171 1,089 3066 6132 355 checon_par 100.0 165 7:48.398 1 483 468398435 .TAU application 0.0 130 130 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 102 102 155 0 659 cgca_dacf 0.3 82 1,243 5 30000 248601 cgca_pfem_intcalc1 5.9 75 27,846 5 20000 5569248 cgca_pfem_simg 0.0 75 75 100001 0 1 beemat [THROTTLED] 0.0 63 63 100001 0 1 MPI_Get() [THROTTLED] 0.0 54 54 7 0 7793 MPI_File_open() 0.0 52 52 13 0 4014 MPI_Win_create() 0.9 48 4,089 12284 12284 333 my_barrier 0.0 45 45 100001 0 0 deemat [THROTTLED] 0.0 44 44 2 0 22066 cgca_gbs 0.0 41 41 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 38 38 100001 0 0 shape_der [THROTTLED] 0.0 36 36 6 0 6140 MPI_Send() 0.0 28 28 100001 0 0 invert [THROTTLED] 0.0 26 26 67 0 399 MPI_Reduce() 0.0 25 25 6144 0 4 MPI_Waitall() 0.0 21 21 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22382 load 0.0 20 20 10053 0 2 MPI_Put() 0.0 20 20 80000 0 0 determinant 0.0 10 10 7 0 1566 MPI_File_close() 0.0 10 10 1 0 10101 MPI_Comm_dup() 0.0 10 48 1 160 48596 scatter_nodes 0.0 9 9 14 0 652 MPI_Type_create_subarray() 0.0 8 8 1 0 8583 cgca_igb 0.0 7 168 155 1105 1085 cgca_gcupdn 0.0 7 7 12288 0 1 MPI_Get_count() 0.0 6 6 1 41 6183 cgca_gcu 0.0 6 14 1 19 14060 make_ggl 0.0 5 5 2000 0 3 find_g3 0.0 5 5 7 0 739 MPI_File_set_view() 0.1 4 442 1 33 442119 read_g_coord_pp 0.0 3 3 12 0 272 MPI_Win_free() 0.0 3 24 1 576 24781 cgca_pfem_cenc 0.1 2 351 1 2 351957 read_g_num_pp 0.0 2 2 5 0 464 cgca_fv 0.0 1 1 1 0 1934 allocate_gather_scatter 0.4 1 1,734 7 525 247814 cgca_pswci 0.0 0.6 1 1 2 1504 cgca_as 0.0 0.391 5 1 377 5980 cgca_rt 0.7 0.359 3,081 1 64 3081941 cgca_ds 0.0 0.305 0.675 1 9 675 scatter_noadd 0.0 0.275 7 1 374 7814 cgca_nr 0.0 0.225 0.225 1 0 225 getname 0.0 0.224 0.224 1 0 224 calc_nels_pp 0.0 0.212 0.212 1 0 212 rearrange 0.0 0.21 0.21 14 0 15 MPI_Type_commit() 0.0 0.125 0.125 1 0 125 cgca_ins 0.0 0.122 4 1 65 4380 gcupd_alloc 0.0 0.114 36 3 6 12319 dismsh_ensi_p 0.0 0.113 9 1 64 9259 cgca_drt 0.0 0.106 0.106 1 0 106 cgca_pfem_salloc 0.0 0.105 0.105 5 0 21 cgca_pfem_uym 0.0 0.05 0.05 20 0 2 cgca_gcf 0.0 0.05 0.05 1 0 50 cgca_gdim 0.0 0.048 0.048 1 0 48 cgca_pfem_sdalloc 0.0 0.047 1 5 5 220 sum_p 0.0 0.043 0.043 1 0 43 abaqus2sg 0.0 0.042 0.042 1 0 42 cgca_pfem_integalloc 0.0 0.041 0.041 7 0 6 elap_time 0.0 0.03 0.03 9 0 3 MPI_Comm_size() 0.0 0.027 0.027 1 0 27 cgca_pfem_ctalloc 0.0 0.026 0.026 1 0 26 cgca_cadim 0.0 0.02 0.02 5 0 4 sample 0.0 0.018 0.018 21 0 1 cgca_agc 0.0 0.018 0.019 1 2 19 find_pe_procs 0.0 0.017 0.017 14 0 1 MPI_Type_free() 0.0 0.016 0.016 1 0 16 calc_neq_pp 0.0 0.014 0.014 1 0 14 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.014 0.014 12 0 1 MPI_Win_fence() 0.0 0.013 0.013 2 0 6 MPI_Test() 0.0 0.013 0.013 1 0 13 cgca_pfem_ealloc 0.0 0.011 41 1 2 41762 cgca_art 0.0 0.011 0.011 20 0 1 cgca_dgc 0.0 0.01 0.01 1 0 10 MPI_Alloc_mem() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.005 0.005 9 0 1 MPI_Comm_rank() 0.0 0.005 2 1 3 2938 read_p121 0.0 0.004 3 1 2 3849 read_loads 0.0 0.003 0.003 1 0 3 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.302 1 1 302 max_p 0.0 0.002 80 1 1 80204 read_rest 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc 0.0 0 0 1 0 0 cgca_imco --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 16, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.194 2.666 3.957 0.8069 MPI-IO Write Bandwidth (MB/s) 7 5.194 2.666 3.957 0.8069 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.893E+04 6.107E+04 4642 Message size received from all nodes 1.229E+04 6.884E+04 4 6.105E+04 4656 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 6144 5.798E+04 2.899E+04 5.792E+04 374.5 Message size sent to node 15 3071 5.786E+04 5.786E+04 5.786E+04 0 Message size sent to node 15 : gather => MPI_Isend() 1 2.899E+04 2.899E+04 2.899E+04 0 Message size sent to node 15 : make_ggl => MPI_Isend() 3071 5.798E+04 5.798E+04 5.798E+04 0 Message size sent to node 15 : scatter => MPI_Isend() 1 5.798E+04 5.798E+04 5.798E+04 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 6144 6.884E+04 2.979E+04 6.42E+04 4652 Message size sent to node 17 3071 6.884E+04 6.884E+04 6.884E+04 0 Message size sent to node 17 : gather => MPI_Isend() 1 2.979E+04 2.979E+04 2.979E+04 0 Message size sent to node 17 : make_ggl => MPI_Isend() 3071 5.958E+04 5.958E+04 5.958E+04 0 Message size sent to node 17 : scatter => MPI_Isend() 1 5.958E+04 5.958E+04 5.958E+04 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 5.798E+04 5.798E+04 5.798E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 5.798E+04 5.798E+04 5.798E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 1 5.798E+04 5.798E+04 5.798E+04 0 [GROUP=MAX_MARKER] Message size sent to node 15 1 5.798E+04 5.798E+04 5.798E+04 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 1 5.958E+04 5.958E+04 5.958E+04 0 [GROUP=MAX_MARKER] Message size sent to node 17 1 5.958E+04 5.958E+04 5.958E+04 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.442E+04 3.442E+04 3.442E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.442E+04 3.442E+04 3.442E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 17;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 74.9 5:50.858 5:50.858 1.1202E+07 0 31 MPI_Win_unlock() 99.7 39,553 7:46.949 1 503146 466949740 xx14std 72.4 28,799 5:39.026 511 1.12372E+07 663457 cgca_hxi 6.1 28,376 28,376 17279 0 1642 MPI_Barrier() 50.0 7,138 3:54.250 5 269768 46850117 cgca_clvgp_nocosum 1.2 2,814 5,503 3071 36852 1792 gather 0.5 2,264 2,264 15338 0 148 MPI_Allreduce() 1.0 1,641 4,683 3071 33781 1525 scatter 0.2 960 960 1 0 960462 MPI_Init() 0.2 868 868 7 0 124050 MPI_File_write_all() 0.1 683 683 12289 0 56 MPI_Recv() 0.1 538 538 135 0 3990 MPI_Bcast() 0.1 456 456 12288 0 37 MPI_Isend() 30.3 448 2:21.740 1 51201 141740528 cgca_sld 0.1 364 364 12288 0 30 MPI_Probe() 0.4 360 1,732 9198 9198 188 dot_product_p 0.1 301 301 1 0 301223 MPI_Finalize() 0.1 243 266 100001 100001 3 cgca_clvgn [THROTTLED] 6.1 233 28,444 5 20000 5688956 cgca_pfem_simg 0.2 195 1,084 3066 6132 354 checon_par 100.0 164 7:48.405 1 386 468405467 .TAU application 0.0 116 116 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 107 107 155 0 692 cgca_dacf 0.3 79 1,502 5 30000 300561 cgca_pfem_intcalc1 0.0 77 77 100001 0 1 MPI_Get() [THROTTLED] 0.0 67 67 100001 0 1 beemat [THROTTLED] 0.0 65 65 6144 0 11 MPI_Waitall() 0.8 56 3,893 12284 12284 317 my_barrier 0.0 54 54 7 0 7828 MPI_File_open() 0.0 44 44 100001 0 0 deemat [THROTTLED] 0.0 44 44 2 0 22134 cgca_gbs 0.0 43 43 13 0 3308 MPI_Win_create() 0.0 39 39 100001 0 0 shape_der [THROTTLED] 0.0 36 36 6 0 6005 MPI_Send() 0.0 33 33 100001 0 0 invert [THROTTLED] 0.0 32 32 100001 0 0 cgca_clvgsd [THROTTLED] 0.1 28 238 155 1726 1541 cgca_gcupdn 0.0 24 24 67 0 367 MPI_Reduce() 0.0 22 22 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22346 load 0.0 20 20 80000 0 0 determinant 0.0 12 12 10041 0 1 MPI_Put() 0.0 10 10 7 0 1550 MPI_File_close() 0.0 10 10 1 0 10064 MPI_Comm_dup() 0.0 9 9 12288 0 1 MPI_Get_count() 0.0 9 9 1 83 9351 cgca_gcu 0.0 9 9 14 0 655 MPI_Type_create_subarray() 0.0 8 48 1 160 48658 scatter_nodes 0.0 7 7 1 0 7917 cgca_igb 0.0 6 14 1 19 14158 make_ggl 0.0 5 5 2000 0 3 find_g3 0.0 5 5 7 0 738 MPI_File_set_view() 0.0 4 4 1 0 4861 allocate_gather_scatter 0.0 4 32 1 576 32332 cgca_pfem_cenc 0.1 4 437 1 33 437780 read_g_coord_pp 0.0 3 3 12 0 272 MPI_Win_free() 0.0 2 2 5 0 546 cgca_fv 0.4 1 1,734 7 525 247801 cgca_pswci 0.0 0.607 0.607 1 0 607 getname 0.0 0.604 1 1 2 1498 cgca_as 0.0 0.422 7 1 374 7889 cgca_nr 0.7 0.325 3,083 1 64 3083156 cgca_ds 0.0 0.294 0.294 1 0 294 cgca_pfem_salloc 0.0 0.289 0.645 1 9 645 scatter_noadd 0.0 0.27 5 1 377 5889 cgca_rt 0.0 0.233 0.233 14 0 17 MPI_Type_commit() 0.0 0.233 0.233 1 0 233 cgca_ins 0.0 0.216 0.216 1 0 216 rearrange 0.0 0.156 4 1 65 4389 gcupd_alloc 0.0 0.139 0.139 1 0 139 calc_nels_pp 0.0 0.12 9 1 64 9263 cgca_drt 0.1 0.104 356 1 2 356333 read_g_num_pp 0.0 0.096 0.096 5 0 19 cgca_pfem_uym 0.0 0.057 0.057 21 0 3 cgca_gcf 0.0 0.057 1 5 5 207 sum_p 0.0 0.054 0.054 5 0 11 sample 0.0 0.053 36 3 6 12027 dismsh_ensi_p 0.0 0.049 0.049 1 0 49 cgca_pfem_sdalloc 0.0 0.041 0.041 1 0 41 abaqus2sg 0.0 0.039 0.039 1 0 39 cgca_gdim 0.0 0.038 0.038 7 0 5 elap_time 0.0 0.027 0.027 12 0 2 MPI_Win_fence() 0.0 0.026 0.026 1 0 26 cgca_pfem_ctalloc 0.0 0.025 0.025 42 0 1 cgca_agc 0.0 0.023 0.023 41 0 1 cgca_dgc 0.0 0.022 0.022 2 0 11 MPI_Test() 0.0 0.017 34 1 2 34470 cgca_art 0.0 0.016 0.016 1 0 16 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.015 1 0 15 calc_neq_pp 0.0 0.015 0.016 1 2 16 find_pe_procs 0.0 0.014 0.014 1 0 14 cgca_cadim 0.0 0.013 0.013 14 0 1 MPI_Type_free() 0.0 0.01 0.01 9 0 1 MPI_Comm_rank() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.008 4 1 3 4660 read_p121 0.0 0.005 0.005 9 0 1 MPI_Comm_size() 0.0 0.005 0.005 1 0 5 cgca_pfem_integalloc 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.003 0.003 1 0 3 cgca_pfem_ctdalloc 0.0 0.003 0.444 1 1 444 max_p 0.0 0.003 3 1 2 3811 read_loads 0.0 0.003 80 1 1 80203 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 17, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.189 2.667 3.957 0.8056 MPI-IO Write Bandwidth (MB/s) 7 5.189 2.667 3.957 0.8056 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.979E+04 6.215E+04 3992 Message size received from all nodes 1.229E+04 6.884E+04 4 6.213E+04 4010 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 6144 6.884E+04 3.442E+04 6.42E+04 4647 Message size sent to node 16 3071 5.958E+04 5.958E+04 5.958E+04 0 Message size sent to node 16 : gather => MPI_Isend() 1 3.442E+04 3.442E+04 3.442E+04 0 Message size sent to node 16 : make_ggl => MPI_Isend() 3071 6.884E+04 6.884E+04 6.884E+04 0 Message size sent to node 16 : scatter => MPI_Isend() 1 6.884E+04 6.884E+04 6.884E+04 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 6144 6.034E+04 2.992E+04 6.009E+04 455.7 Message size sent to node 18 3071 6.034E+04 6.034E+04 6.034E+04 0 Message size sent to node 18 : gather => MPI_Isend() 1 2.992E+04 2.992E+04 2.992E+04 0 Message size sent to node 18 : make_ggl => MPI_Isend() 3071 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 18 : scatter => MPI_Isend() 1 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 6.884E+04 6.884E+04 6.884E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 6.884E+04 6.884E+04 6.884E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 1 6.884E+04 6.884E+04 6.884E+04 0 [GROUP=MAX_MARKER] Message size sent to node 16 1 6.884E+04 6.884E+04 6.884E+04 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 1 5.985E+04 5.985E+04 5.985E+04 0 [GROUP=MAX_MARKER] Message size sent to node 18 1 5.985E+04 5.985E+04 5.985E+04 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.979E+04 2.979E+04 2.979E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.979E+04 2.979E+04 2.979E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 18;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 62.9 4:54.466 4:54.466 1.12022E+07 0 26 MPI_Win_unlock() 21.7 1:41.447 1:41.447 17279 0 5871 MPI_Barrier() 99.7 38,853 7:46.947 1 503146 466947645 xx14std 57.5 14,478 4:29.400 511 1.12372E+07 527202 cgca_hxi 50.0 5,680 3:54.250 5 269768 46850154 cgca_clvgp_nocosum 1.2 2,601 5,417 3071 36852 1764 gather 0.5 2,419 2,419 15338 0 158 MPI_Allreduce() 1.1 1,322 5,349 3071 33781 1742 scatter 0.2 959 959 1 0 959434 MPI_Init() 0.2 868 868 7 0 124112 MPI_File_write_all() 0.2 707 707 100001 0 7 MPI_Win_lock() [THROTTLED] 0.1 670 670 12289 0 55 MPI_Recv() 0.1 615 615 12288 0 50 MPI_Probe() 0.1 530 530 135 0 3927 MPI_Bcast() 30.3 350 2:21.740 1 51201 141740534 cgca_sld 0.4 334 1,851 9198 9198 201 dot_product_p 0.1 305 305 1 0 305238 MPI_Finalize() 0.1 293 293 12288 0 24 MPI_Isend() 0.1 234 257 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 183 1,082 3066 6132 353 checon_par 100.0 142 7:48.386 1 626 468386692 .TAU application 0.0 97 97 155 0 630 cgca_dacf 0.0 71 71 100001 0 1 beemat [THROTTLED] 5.9 62 27,479 5 20000 5495804 cgca_pfem_simg 0.0 54 54 7 0 7807 MPI_File_open() 0.0 50 50 100001 0 1 MPI_Get() [THROTTLED] 0.0 44 44 6144 0 7 MPI_Waitall() 0.0 44 44 100001 0 0 deemat [THROTTLED] 0.0 44 44 2 0 22034 cgca_gbs 0.3 42 1,198 5 30000 239737 cgca_pfem_intcalc1 1.0 38 4,719 12284 12284 384 my_barrier 0.0 38 38 100001 0 0 shape_der [THROTTLED] 0.0 36 36 6 0 6072 MPI_Send() 0.0 33 33 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 28 28 67 0 426 MPI_Reduce() 0.0 22 22 10105 0 2 MPI_Put() 0.0 22 22 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22420 load 0.0 20 20 80000 0 0 determinant 0.0 15 15 13 0 1228 MPI_Win_create() 0.0 10 10 1 0 10094 MPI_Comm_dup() 0.0 9 9 7 0 1355 MPI_File_close() 0.0 9 9 12288 0 1 MPI_Get_count() 0.0 9 9 14 0 650 MPI_Type_create_subarray() 0.0 8 8 1 0 8642 cgca_igb 0.0 6 6 1 55 6817 cgca_gcu 0.0 6 14 1 19 14049 make_ggl 0.0 6 203 155 1720 1314 cgca_gcupdn 0.0 5 5 2000 0 3 find_g3 0.0 4 4 7 0 703 MPI_File_set_view() 0.0 4 4 1 0 4812 allocate_gather_scatter 0.0 4 48 1 160 48676 scatter_nodes 0.1 4 430 1 33 430774 read_g_coord_pp 0.0 3 3 12 0 256 MPI_Win_free() 0.0 2 2 5 0 444 cgca_fv 0.0 1 61 1 576 61374 cgca_pfem_cenc 0.4 0.885 1,733 7 525 247620 cgca_pswci 0.0 0.591 1 1 2 1485 cgca_as 0.0 0.324 0.324 1 0 324 getname 0.0 0.32 0.661 1 9 661 scatter_noadd 0.7 0.3 3,082 1 64 3082931 cgca_ds 0.0 0.215 0.215 1 0 215 rearrange 0.0 0.192 0.192 1 0 192 cgca_pfem_salloc 0.0 0.185 7 1 374 7844 cgca_nr 0.0 0.162 0.162 14 0 12 MPI_Type_commit() 0.0 0.16 5 1 377 5948 cgca_rt 0.0 0.133 0.133 1 0 133 calc_nels_pp 0.0 0.123 0.123 1 0 123 cgca_ins 0.1 0.113 363 1 2 363375 read_g_num_pp 0.0 0.108 0.108 5 0 22 cgca_pfem_uym 0.0 0.084 9 1 64 9266 cgca_drt 0.0 0.064 4 1 65 4410 gcupd_alloc 0.0 0.049 0.049 1 0 49 cgca_cadim 0.0 0.045 0.045 1 0 45 cgca_pfem_sdalloc 0.0 0.044 1 5 5 226 sum_p 0.0 0.041 0.041 1 0 41 abaqus2sg 0.0 0.04 0.04 7 0 6 elap_time 0.0 0.036 36 3 6 12155 dismsh_ensi_p 0.0 0.023 0.024 1 2 24 find_pe_procs 0.0 0.02 0.02 1 0 20 cgca_gdim 0.0 0.019 0.019 1 0 19 cgca_pfem_ctalloc 0.0 0.018 0.018 15 0 1 cgca_gcf 0.0 0.017 0.017 5 0 3 sample 0.0 0.014 0.014 1 0 14 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.014 0.014 12 0 1 MPI_Win_fence() 0.0 0.014 0.014 1 0 14 calc_neq_pp 0.0 0.012 0.012 14 0 1 MPI_Type_free() 0.0 0.012 0.012 28 0 0 cgca_agc 0.0 0.01 0.01 2 0 5 MPI_Test() 0.0 0.01 0.01 27 0 0 cgca_dgc 0.0 0.009 2 1 3 2843 read_p121 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.006 0.006 9 0 1 MPI_Comm_rank() 0.0 0.006 5 1 2 5496 cgca_art 0.0 0.006 3 1 2 3909 read_loads 0.0 0.005 0.005 1 0 5 MPI_Alloc_mem() 0.0 0.005 0.005 9 0 1 MPI_Comm_size() 0.0 0.003 0.003 1 0 3 cgca_pfem_integalloc 0.0 0.003 0.269 1 1 269 max_p 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 80 1 1 80203 read_rest 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 18, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.188 2.666 3.955 0.8056 MPI-IO Write Bandwidth (MB/s) 7 5.188 2.666 3.955 0.8056 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.992E+04 6.191E+04 2592 Message size received from all nodes 1.229E+04 6.766E+04 4 6.189E+04 2620 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 6144 6.034E+04 3.017E+04 6.009E+04 453 Message size sent to node 17 3071 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 17 : gather => MPI_Isend() 1 3.017E+04 3.017E+04 3.017E+04 0 Message size sent to node 17 : make_ggl => MPI_Isend() 3071 6.034E+04 6.034E+04 6.034E+04 0 Message size sent to node 17 : scatter => MPI_Isend() 1 6.034E+04 6.034E+04 6.034E+04 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 6144 6.594E+04 3.297E+04 6.372E+04 2246 Message size sent to node 19 3071 6.151E+04 6.151E+04 6.151E+04 0 Message size sent to node 19 : gather => MPI_Isend() 1 3.297E+04 3.297E+04 3.297E+04 0 Message size sent to node 19 : make_ggl => MPI_Isend() 3071 6.594E+04 6.594E+04 6.594E+04 0 Message size sent to node 19 : scatter => MPI_Isend() 1 6.594E+04 6.594E+04 6.594E+04 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 6.034E+04 6.034E+04 6.034E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 6.034E+04 6.034E+04 6.034E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 1 6.034E+04 6.034E+04 6.034E+04 0 [GROUP=MAX_MARKER] Message size sent to node 17 1 6.034E+04 6.034E+04 6.034E+04 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 1 6.594E+04 6.594E+04 6.594E+04 0 [GROUP=MAX_MARKER] Message size sent to node 19 1 6.594E+04 6.594E+04 6.594E+04 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.076E+04 3.076E+04 3.076E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.076E+04 3.076E+04 3.076E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 19;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 59.1 4:36.918 4:36.918 8.01294E+06 0 35 MPI_Win_unlock() 26.6 2:04.370 2:04.370 17279 0 7198 MPI_Barrier() 99.7 38,721 7:46.949 1 503146 466949757 xx14std 45.8 11,316 3:34.625 511 8.04756E+06 420011 cgca_hxi 50.0 4,425 3:54.250 5 269768 46850126 cgca_clvgp_nocosum 1.2 2,639 5,407 3071 36852 1761 gather 0.5 2,448 2,448 15338 0 160 MPI_Allreduce() 1.2 1,357 5,479 3071 33781 1784 scatter 0.2 960 960 1 0 960719 MPI_Init() 0.2 869 869 7 0 124154 MPI_File_write_all() 0.1 663 663 12289 0 54 MPI_Recv() 0.1 524 524 135 0 3886 MPI_Bcast() 30.3 392 2:21.750 1 52197 141750416 cgca_sld 0.1 321 321 1 0 321152 MPI_Finalize() 0.4 315 1,865 9198 9198 203 dot_product_p 0.1 295 295 12288 0 24 MPI_Probe() 0.1 289 289 12288 0 24 MPI_Isend() 0.1 234 258 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 182 1,078 3066 6132 352 checon_par 100.0 147 7:48.408 1 621 468408932 .TAU application 0.0 108 108 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 98 98 155 0 636 cgca_dacf 5.9 68 27,451 5 20000 5490236 cgca_pfem_simg 0.0 66 66 100001 0 1 beemat [THROTTLED] 0.0 54 54 100001 0 1 MPI_Get() [THROTTLED] 0.0 54 54 7 0 7779 MPI_File_open() 0.0 49 49 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21996 cgca_gbs 0.3 41 1,204 5 30000 240982 cgca_pfem_intcalc1 0.0 38 38 100001 0 0 shape_der [THROTTLED] 1.0 37 4,878 12284 12284 397 my_barrier 0.0 36 36 6 0 6073 MPI_Send() 0.0 33 33 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 31 31 100001 0 0 invert [THROTTLED] 0.0 29 29 67 0 433 MPI_Reduce() 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 23 23 6144 0 4 MPI_Waitall() 0.0 21 22 1 1 22443 load 0.0 21 21 10103 0 2 MPI_Put() 0.0 19 19 80000 0 0 determinant 0.0 14 14 13 0 1082 MPI_Win_create() 0.0 10 10 1 0 10028 MPI_Comm_dup() 0.0 9 9 1 0 9559 cgca_igb 0.0 9 9 7 0 1325 MPI_File_close() 0.0 9 9 14 0 653 MPI_Type_create_subarray() 0.0 7 7 12288 0 1 MPI_Get_count() 0.0 6 14 1 19 14115 make_ggl 0.0 6 6 1 35 6224 cgca_gcu 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4842 allocate_gather_scatter 0.0 4 4 7 0 664 MPI_File_set_view() 0.0 4 48 1 160 48723 scatter_nodes 0.1 4 423 1 33 423695 read_g_coord_pp 0.0 3 52 155 1086 341 cgca_gcupdn 0.0 3 3 12 0 262 MPI_Win_free() 0.0 2 2 5 0 563 cgca_fv 0.0 1 61 1 576 61392 cgca_pfem_cenc 0.4 0.939 1,733 7 525 247625 cgca_pswci 0.0 0.643 0.643 1 0 643 getname 0.0 0.597 1 1 2 1477 cgca_as 0.0 0.298 0.745 1 9 745 scatter_noadd 0.7 0.24 3,082 1 64 3082927 cgca_ds 0.0 0.218 0.218 1 0 218 rearrange 0.0 0.186 0.186 1 0 186 cgca_pfem_salloc 0.0 0.173 7 1 374 7908 cgca_nr 0.0 0.161 0.161 14 0 12 MPI_Type_commit() 0.0 0.15 5 1 377 5909 cgca_rt 0.0 0.133 0.133 1 0 133 calc_nels_pp 0.0 0.126 0.126 1 0 126 cgca_ins 0.1 0.106 370 1 2 370336 read_g_num_pp 0.0 0.083 0.083 5 0 17 cgca_pfem_uym 0.0 0.08 4 1 65 4428 gcupd_alloc 0.0 0.079 9 1 64 9263 cgca_drt 0.0 0.074 1 5 5 238 sum_p 0.0 0.048 0.048 1 0 48 cgca_pfem_sdalloc 0.0 0.043 0.043 1 0 43 cgca_cadim 0.0 0.042 0.042 1 0 42 abaqus2sg 0.0 0.039 0.039 7 0 6 elap_time 0.0 0.033 0.033 1 0 33 calc_neq_pp 0.0 0.03 36 3 6 12155 dismsh_ensi_p 0.0 0.027 0.027 1 0 27 cgca_gdim 0.0 0.027 0.027 5 0 5 sample 0.0 0.018 0.018 2 0 9 MPI_Test() 0.0 0.017 0.017 1 0 17 cgca_pfem_ctalloc 0.0 0.017 0.018 1 2 18 find_pe_procs 0.0 0.016 0.016 1 0 16 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.016 0.016 14 0 1 MPI_Type_free() 0.0 0.012 0.012 12 0 1 MPI_Win_fence() 0.0 0.01 0.01 17 0 1 cgca_dgc 0.0 0.009 0.009 18 0 0 cgca_agc 0.0 0.009 4 1 3 4709 read_p121 0.0 0.008 0.008 9 0 1 MPI_Comm_rank() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.008 5 1 2 5592 cgca_art 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 0.004 9 0 0 MPI_Comm_size() 0.0 0.004 3 1 2 3899 read_loads 0.0 0.003 0.003 1 0 3 cgca_pfem_integalloc 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 calc_npes_pp 0.0 0.002 0.002 1 0 2 cgca_gcf 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.239 1 1 239 max_p 0.0 0.002 80 1 1 80206 read_rest 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc 0.0 0 0 1 0 0 cgca_imco --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 19, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.188 2.665 3.953 0.805 MPI-IO Write Bandwidth (MB/s) 7 5.188 2.665 3.953 0.805 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 3.134E+04 6.083E+04 4813 Message size received from all nodes 1.229E+04 6.766E+04 4 6.08E+04 4827 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 6144 6.594E+04 3.076E+04 6.372E+04 2251 Message size sent to node 18 3071 6.594E+04 6.594E+04 6.594E+04 0 Message size sent to node 18 : gather => MPI_Isend() 1 3.076E+04 3.076E+04 3.076E+04 0 Message size sent to node 18 : make_ggl => MPI_Isend() 3071 6.151E+04 6.151E+04 6.151E+04 0 Message size sent to node 18 : scatter => MPI_Isend() 1 6.151E+04 6.151E+04 6.151E+04 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 6144 6.269E+04 2.658E+04 5.791E+04 4784 Message size sent to node 20 3071 6.269E+04 6.269E+04 6.269E+04 0 Message size sent to node 20 : gather => MPI_Isend() 1 2.658E+04 2.658E+04 2.658E+04 0 Message size sent to node 20 : make_ggl => MPI_Isend() 3071 5.315E+04 5.315E+04 5.315E+04 0 Message size sent to node 20 : scatter => MPI_Isend() 1 5.315E+04 5.315E+04 5.315E+04 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 6.151E+04 6.151E+04 6.151E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 6.151E+04 6.151E+04 6.151E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 1 6.151E+04 6.151E+04 6.151E+04 0 [GROUP=MAX_MARKER] Message size sent to node 18 1 6.151E+04 6.151E+04 6.151E+04 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 1 5.315E+04 5.315E+04 5.315E+04 0 [GROUP=MAX_MARKER] Message size sent to node 20 1 5.315E+04 5.315E+04 5.315E+04 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.297E+04 3.297E+04 3.297E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.297E+04 3.297E+04 3.297E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 20;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 60.9 4:45.049 4:45.049 1.12021E+07 0 25 MPI_Win_unlock() 24.1 1:52.705 1:52.705 17279 0 6523 MPI_Barrier() 99.7 38,685 7:46.947 1 503146 466947696 xx14std 55.7 14,483 4:20.840 511 1.12372E+07 510450 cgca_hxi 50.0 4,958 3:54.250 5 269788 46850040 cgca_clvgp_nocosum 1.2 2,663 5,445 3071 36852 1773 gather 0.5 2,464 2,464 15338 0 161 MPI_Allreduce() 1.2 1,349 5,489 3071 33781 1788 scatter 0.2 961 961 1 0 961020 MPI_Init() 0.2 868 868 7 0 124111 MPI_File_write_all() 0.2 720 720 12289 0 59 MPI_Recv() 0.1 515 515 135 0 3820 MPI_Bcast() 30.3 335 2:21.750 1 51201 141750329 cgca_sld 0.1 328 328 1 0 328367 MPI_Finalize() 0.4 302 1,846 9198 9198 201 dot_product_p 0.1 276 276 12288 0 22 MPI_Isend() 0.1 226 249 100001 100001 2 cgca_clvgn [THROTTLED] 0.0 218 218 12288 0 18 MPI_Probe() 0.2 172 1,089 3066 6132 355 checon_par 100.0 163 7:48.432 1 526 468432612 .TAU application 0.0 101 101 155 0 655 cgca_dacf 0.0 81 81 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 68 68 100001 0 1 beemat [THROTTLED] 5.3 65 24,840 5 20000 4968130 cgca_pfem_simg 0.0 54 54 7 0 7855 MPI_File_open() 0.0 51 51 100001 0 1 MPI_Get() [THROTTLED] 0.0 45 45 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21982 cgca_gbs 0.0 43 43 100001 0 0 cgca_clvgsd [THROTTLED] 1.1 40 4,924 12284 12284 401 my_barrier 0.1 39 654 5 30000 130803 cgca_pfem_intcalc1 0.0 36 36 100001 0 0 shape_der [THROTTLED] 0.0 36 36 6 0 6072 MPI_Send() 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 29 29 67 0 433 MPI_Reduce() 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22417 load 0.0 21 21 6144 0 4 MPI_Waitall() 0.0 21 21 80000 0 0 determinant 0.0 15 15 13 0 1214 MPI_Win_create() 0.0 12 12 10092 0 1 MPI_Put() 0.0 10 10 1 0 10034 MPI_Comm_dup() 0.0 9 9 1 0 9905 cgca_igb 0.0 9 9 7 0 1356 MPI_File_close() 0.0 9 9 14 0 650 MPI_Type_create_subarray() 0.0 7 7 12288 0 1 MPI_Get_count() 0.0 6 14 1 19 14007 make_ggl 0.0 5 190 155 1731 1229 cgca_gcupdn 0.0 5 5 2000 0 3 find_g3 0.0 5 5 1 45 5221 cgca_gcu 0.0 4 4 1 0 4872 allocate_gather_scatter 0.0 4 4 7 0 640 MPI_File_set_view() 0.0 4 48 1 160 48724 scatter_nodes 0.1 4 416 1 33 416570 read_g_coord_pp 0.0 3 3 12 0 261 MPI_Win_free() 0.0 2 2 5 0 444 cgca_fv 0.0 1 61 1 576 61516 cgca_pfem_cenc 0.4 0.947 1,733 7 525 247632 cgca_pswci 0.0 0.588 1 1 2 1469 cgca_as 0.0 0.34 0.34 1 0 340 getname 0.0 0.326 0.689 1 9 689 scatter_noadd 0.7 0.294 3,082 1 64 3082921 cgca_ds 0.0 0.213 0.213 1 0 213 rearrange 0.0 0.178 0.178 14 0 13 MPI_Type_commit() 0.0 0.175 7 1 374 7849 cgca_nr 0.0 0.167 0.167 1 0 167 cgca_pfem_salloc 0.0 0.145 5 1 377 5974 cgca_rt 0.0 0.142 0.142 1 0 142 calc_nels_pp 0.0 0.123 0.123 1 0 123 cgca_ins 0.1 0.109 377 1 2 377437 read_g_num_pp 0.0 0.101 4 1 65 4423 gcupd_alloc 0.0 0.093 0.093 5 0 19 cgca_pfem_uym 0.0 0.087 9 1 64 9338 cgca_drt 0.0 0.051 0.051 7 0 7 elap_time 0.0 0.049 0.049 1 0 49 cgca_gdim 0.0 0.047 0.968 5 5 194 sum_p 0.0 0.041 0.041 1 0 41 abaqus2sg 0.0 0.032 0.032 1 0 32 cgca_pfem_sdalloc 0.0 0.026 0.026 1 0 26 cgca_cadim 0.0 0.026 0.026 26 0 1 cgca_gcf 0.0 0.024 36 3 6 12153 dismsh_ensi_p 0.0 0.021 0.021 1 0 21 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.018 0.018 2 0 9 MPI_Test() 0.0 0.017 0.017 5 0 3 sample 0.0 0.016 0.016 1 0 16 calc_neq_pp 0.0 0.016 0.016 1 0 16 cgca_pfem_ctalloc 0.0 0.014 0.014 23 0 1 cgca_agc 0.0 0.013 0.014 1 2 14 find_pe_procs 0.0 0.012 0.012 12 0 1 MPI_Win_fence() 0.0 0.011 0.011 14 0 1 MPI_Type_free() 0.0 0.011 0.011 1 0 11 caflaunch$MAIN$blk 0.0 0.01 0.01 20 0 0 cgca_gcr 0.0 0.008 0.008 22 0 0 cgca_dgc 0.0 0.007 0.007 9 0 1 MPI_Comm_size() 0.0 0.006 0.006 1 0 6 MPI_Alloc_mem() 0.0 0.006 3 1 2 3950 read_loads 0.0 0.005 5 1 2 5422 cgca_art 0.0 0.005 2 1 3 2923 read_p121 0.0 0.004 0.004 9 0 0 MPI_Comm_rank() 0.0 0.003 80 1 1 80209 read_rest 0.0 0.002 0.002 1 0 2 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_edalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.002 0.353 1 1 353 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc 0.0 0 0 1 0 0 cgca_imco --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 20, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.191 2.665 3.955 0.8064 MPI-IO Write Bandwidth (MB/s) 7 5.191 2.665 3.955 0.8064 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.658E+04 6.107E+04 8110 Message size received from all nodes 1.229E+04 7.354E+04 4 6.105E+04 8116 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 6144 6.269E+04 3.134E+04 5.792E+04 4780 Message size sent to node 19 3071 5.315E+04 5.315E+04 5.315E+04 0 Message size sent to node 19 : gather => MPI_Isend() 1 3.134E+04 3.134E+04 3.134E+04 0 Message size sent to node 19 : make_ggl => MPI_Isend() 3071 6.269E+04 6.269E+04 6.269E+04 0 Message size sent to node 19 : scatter => MPI_Isend() 1 6.269E+04 6.269E+04 6.269E+04 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 6144 7.354E+04 2.744E+04 6.42E+04 9347 Message size sent to node 21 3071 7.354E+04 7.354E+04 7.354E+04 0 Message size sent to node 21 : gather => MPI_Isend() 1 2.744E+04 2.744E+04 2.744E+04 0 Message size sent to node 21 : make_ggl => MPI_Isend() 3071 5.487E+04 5.487E+04 5.487E+04 0 Message size sent to node 21 : scatter => MPI_Isend() 1 5.487E+04 5.487E+04 5.487E+04 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 6.269E+04 6.269E+04 6.269E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 6.269E+04 6.269E+04 6.269E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 1 6.269E+04 6.269E+04 6.269E+04 0 [GROUP=MAX_MARKER] Message size sent to node 19 1 6.269E+04 6.269E+04 6.269E+04 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 1 5.487E+04 5.487E+04 5.487E+04 0 [GROUP=MAX_MARKER] Message size sent to node 21 1 5.487E+04 5.487E+04 5.487E+04 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.677E+04 3.677E+04 3.677E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.677E+04 3.677E+04 3.677E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 21;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 75.9 5:55.779 5:55.779 1.44715E+07 0 25 MPI_Win_unlock() 99.7 38,818 7:46.947 1 503146 466947769 xx14std 6.9 32,131 32,131 17279 0 1860 MPI_Barrier() 73.6 22,714 5:44.805 511 1.45061E+07 674767 cgca_hxi 50.0 6,284 3:54.250 5 280852 46850081 cgca_clvgp_nocosum 1.2 2,601 5,410 3071 36852 1762 gather 0.5 2,402 2,402 15338 0 157 MPI_Allreduce() 1.2 1,320 5,403 3071 33781 1760 scatter 0.2 960 960 1 0 960585 MPI_Init() 0.2 869 869 7 0 124160 MPI_File_write_all() 0.2 735 735 12289 0 60 MPI_Recv() 0.1 508 508 135 0 3768 MPI_Bcast() 0.1 344 344 1 0 344319 MPI_Finalize() 30.3 326 2:21.740 1 50703 141740515 cgca_sld 0.4 323 1,825 9198 9198 198 dot_product_p 0.1 323 323 12288 0 26 MPI_Probe() 0.1 285 285 12288 0 23 MPI_Isend() 0.1 228 251 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 190 1,088 3066 6132 355 checon_par 100.0 163 7:48.448 1 600 468448378 .TAU application 0.0 162 162 100001 0 2 MPI_Win_lock() [THROTTLED] 0.0 94 94 155 0 609 cgca_dacf 5.3 74 24,678 5 20000 4935723 cgca_pfem_simg 0.0 71 71 100001 0 1 beemat [THROTTLED] 0.0 68 68 6144 0 11 MPI_Waitall() 0.0 55 55 7 0 7910 MPI_File_open() 0.0 53 53 100001 0 1 MPI_Get() [THROTTLED] 0.0 48 48 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 45 45 100001 0 0 deemat [THROTTLED] 0.0 44 44 2 0 22057 cgca_gbs 1.1 39 4,960 12284 12284 404 my_barrier 0.1 38 645 5 30000 129153 cgca_pfem_intcalc1 0.0 37 37 100001 0 0 shape_der [THROTTLED] 0.0 36 36 6 0 6100 MPI_Send() 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 29 29 67 0 437 MPI_Reduce() 0.0 27 27 80000 0 0 determinant 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22327 load 0.0 15 15 13 0 1228 MPI_Win_create() 0.0 11 11 10095 0 1 MPI_Put() 0.0 10 10 1 0 10078 MPI_Comm_dup() 0.0 9 9 7 0 1307 MPI_File_close() 0.0 9 9 14 0 651 MPI_Type_create_subarray() 0.0 8 8 1 69 8898 cgca_gcu 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 7 7 1 0 7987 cgca_igb 0.1 7 240 155 2662 1549 cgca_gcupdn 0.0 6 14 1 19 14108 make_ggl 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4780 allocate_gather_scatter 0.0 4 4 7 0 630 MPI_File_set_view() 0.0 4 48 1 160 48725 scatter_nodes 0.1 4 409 1 33 409432 read_g_coord_pp 0.0 3 3 12 0 261 MPI_Win_free() 0.0 2 2 11084 0 0 cgca_gcr 0.0 2 2 5 0 445 cgca_fv 0.0 1 61 1 576 61577 cgca_pfem_cenc 0.4 0.798 1,733 7 525 247654 cgca_pswci 0.0 0.602 1 1 2 1465 cgca_as 0.0 0.316 0.59 1 9 590 scatter_noadd 0.7 0.269 3,082 1 64 3082789 cgca_ds 0.0 0.256 0.256 1 0 256 getname 0.0 0.242 0.242 1 0 242 cgca_pfem_salloc 0.0 0.214 0.214 1 0 214 rearrange 0.0 0.198 4 1 65 4503 gcupd_alloc 0.0 0.182 7 1 374 7897 cgca_nr 0.0 0.157 0.157 14 0 11 MPI_Type_commit() 0.0 0.146 5 1 377 5925 cgca_rt 0.0 0.136 0.136 1 0 136 calc_nels_pp 0.0 0.123 0.123 1 0 123 cgca_ins 0.1 0.102 384 1 2 384521 read_g_num_pp 0.0 0.098 0.098 27 0 4 cgca_gcf 0.0 0.097 9 1 64 9345 cgca_drt 0.0 0.086 0.086 5 0 17 cgca_pfem_uym 0.0 0.05 0.05 1 0 50 cgca_gdim 0.0 0.047 0.047 1 0 47 cgca_pfem_sdalloc 0.0 0.046 1 5 5 223 sum_p 0.0 0.041 0.041 1 0 41 abaqus2sg 0.0 0.034 0.034 7 0 5 elap_time 0.0 0.028 0.028 2 0 14 MPI_Test() 0.0 0.027 0.027 1 0 27 cgca_cadim 0.0 0.025 36 3 6 12207 dismsh_ensi_p 0.0 0.023 0.023 5 0 5 sample 0.0 0.022 0.022 34 0 1 cgca_dgc 0.0 0.02 0.02 1 0 20 cgca_pfem_ctalloc 0.0 0.019 0.019 35 0 1 cgca_agc 0.0 0.018 0.019 1 2 19 find_pe_procs 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.015 12 0 1 MPI_Win_fence() 0.0 0.014 0.014 1 0 14 calc_neq_pp 0.0 0.012 0.012 14 0 1 MPI_Type_free() 0.0 0.01 3 1 3 3061 read_p121 0.0 0.009 5 1 2 5432 cgca_art 0.0 0.008 0.008 9 0 1 MPI_Comm_rank() 0.0 0.007 0.007 1 0 7 caflaunch$MAIN$blk 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 3 1 2 3937 read_loads 0.0 0.003 0.003 1 0 3 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_edalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.002 0.266 1 1 266 max_p 0.0 0.002 80 1 1 80213 read_rest 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 21, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.186 2.665 3.953 0.8042 MPI-IO Write Bandwidth (MB/s) 7 5.186 2.665 3.953 0.8042 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.744E+04 6.333E+04 6975 Message size received from all nodes 1.229E+04 7.354E+04 4 6.331E+04 6988 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 6144 7.354E+04 3.677E+04 6.421E+04 9342 Message size sent to node 20 3071 5.487E+04 5.487E+04 5.487E+04 0 Message size sent to node 20 : gather => MPI_Isend() 1 3.677E+04 3.677E+04 3.677E+04 0 Message size sent to node 20 : make_ggl => MPI_Isend() 3071 7.354E+04 7.354E+04 7.354E+04 0 Message size sent to node 20 : scatter => MPI_Isend() 1 7.354E+04 7.354E+04 7.354E+04 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 6144 6.504E+04 2.992E+04 6.244E+04 2629 Message size sent to node 22 3071 6.504E+04 6.504E+04 6.504E+04 0 Message size sent to node 22 : gather => MPI_Isend() 1 2.992E+04 2.992E+04 2.992E+04 0 Message size sent to node 22 : make_ggl => MPI_Isend() 3071 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 22 : scatter => MPI_Isend() 1 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 7.354E+04 7.354E+04 7.354E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 7.354E+04 7.354E+04 7.354E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 1 7.354E+04 7.354E+04 7.354E+04 0 [GROUP=MAX_MARKER] Message size sent to node 20 1 7.354E+04 7.354E+04 7.354E+04 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 1 5.985E+04 5.985E+04 5.985E+04 0 [GROUP=MAX_MARKER] Message size sent to node 22 1 5.985E+04 5.985E+04 5.985E+04 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.744E+04 2.744E+04 2.744E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.744E+04 2.744E+04 2.744E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 22;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 62.7 4:53.586 4:53.586 1.44715E+07 0 20 MPI_Win_unlock() 21.3 1:39.956 1:39.956 17279 0 5785 MPI_Barrier() 99.7 38,638 7:46.947 1 503146 466947644 xx14std 59.1 17,803 4:36.730 511 1.45061E+07 541548 cgca_hxi 50.0 5,735 3:54.250 5 269768 46850076 cgca_clvgp_nocosum 1.1 2,621 5,345 3071 36852 1741 gather 0.5 2,471 2,471 15338 0 161 MPI_Allreduce() 1.2 1,362 5,596 3071 33781 1822 scatter 0.2 959 959 1 0 959828 MPI_Init() 0.2 869 869 7 0 124188 MPI_File_write_all() 0.1 695 695 12289 0 57 MPI_Recv() 0.1 501 501 135 0 3715 MPI_Bcast() 30.3 473 2:21.750 1 50703 141750379 cgca_sld 0.1 348 348 1 0 348216 MPI_Finalize() 0.4 313 1,883 9198 9198 205 dot_product_p 0.1 301 301 12288 0 25 MPI_Isend() 0.1 240 240 12288 0 20 MPI_Probe() 0.1 233 257 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 182 1,082 3066 6132 353 checon_par 100.0 164 7:48.452 1 588 468452455 .TAU application 0.0 98 98 155 0 634 cgca_dacf 0.0 88 88 100001 0 1 MPI_Win_lock() [THROTTLED] 5.3 73 24,662 5 20000 4932575 cgca_pfem_simg 0.0 70 70 100001 0 1 beemat [THROTTLED] 0.0 55 55 7 0 7873 MPI_File_open() 0.0 53 53 100001 0 1 deemat [THROTTLED] 0.0 50 50 100001 0 1 MPI_Get() [THROTTLED] 0.0 44 44 2 0 22028 cgca_gbs 1.1 37 4,963 12284 12284 404 my_barrier 0.0 36 36 100001 0 0 shape_der [THROTTLED] 0.0 36 36 6 0 6081 MPI_Send() 0.1 35 634 5 30000 126991 cgca_pfem_intcalc1 0.0 34 34 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 31 31 100001 0 0 invert [THROTTLED] 0.0 29 29 67 0 440 MPI_Reduce() 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 23 23 6144 0 4 MPI_Waitall() 0.0 21 22 1 1 22377 load 0.0 19 19 80000 0 0 determinant 0.0 15 15 13 0 1187 MPI_Win_create() 0.0 11 11 10090 0 1 MPI_Put() 0.0 10 10 1 89 10180 cgca_gcu 0.0 10 10 1 0 10092 MPI_Comm_dup() 0.1 9 265 155 2655 1715 cgca_gcupdn 0.0 9 9 14 0 650 MPI_Type_create_subarray() 0.0 8 8 7 0 1222 MPI_File_close() 0.0 8 8 1 0 8523 cgca_igb 0.0 7 7 12288 0 1 MPI_Get_count() 0.0 6 14 1 19 14101 make_ggl 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4842 allocate_gather_scatter 0.0 4 4 7 0 605 MPI_File_set_view() 0.0 4 48 1 160 48734 scatter_nodes 0.1 4 402 1 33 402436 read_g_coord_pp 0.0 2 2 12 0 230 MPI_Win_free() 0.0 2 2 5 0 443 cgca_fv 0.0 1 61 1 576 61842 cgca_pfem_cenc 0.4 0.785 1,732 7 525 247562 cgca_pswci 0.0 0.599 1 1 2 1474 cgca_as 0.0 0.34 0.34 1 0 340 getname 0.0 0.301 0.587 1 9 587 scatter_noadd 0.7 0.291 3,082 1 64 3082761 cgca_ds 0.0 0.217 0.217 1 0 217 cgca_pfem_salloc 0.0 0.208 0.208 1 0 208 rearrange 0.0 0.187 0.187 1 0 187 calc_nels_pp 0.0 0.185 7 1 374 7844 cgca_nr 0.0 0.149 0.149 14 0 11 MPI_Type_commit() 0.0 0.139 5 1 377 5923 cgca_rt 0.0 0.123 0.123 1 0 123 cgca_ins 0.1 0.114 391 1 2 391488 read_g_num_pp 0.0 0.094 9 1 64 9273 cgca_drt 0.0 0.086 4 1 65 4520 gcupd_alloc 0.0 0.082 0.082 5 0 16 cgca_pfem_uym 0.0 0.047 0.047 1 0 47 cgca_gdim 0.0 0.045 0.99 5 5 198 sum_p 0.0 0.041 0.041 1 0 41 abaqus2sg 0.0 0.039 0.039 7 0 6 elap_time 0.0 0.039 0.039 5 0 8 sample 0.0 0.035 0.035 1 0 35 calc_neq_pp 0.0 0.033 0.033 1 0 33 cgca_cadim 0.0 0.032 0.032 1 0 32 cgca_pfem_sdalloc 0.0 0.027 0.027 20 0 1 cgca_gcf 0.0 0.026 0.026 45 0 1 cgca_agc 0.0 0.025 36 3 6 12171 dismsh_ensi_p 0.0 0.021 0.021 44 0 0 cgca_dgc 0.0 0.017 0.017 1 0 17 cgca_pfem_ctalloc 0.0 0.015 0.015 12 0 1 MPI_Win_fence() 0.0 0.015 0.016 1 2 16 find_pe_procs 0.0 0.014 0.014 1 0 14 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.012 0.012 14 0 1 MPI_Type_free() 0.0 0.011 0.011 2 0 6 MPI_Test() 0.0 0.008 0.008 9 0 1 MPI_Comm_rank() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.006 5 1 2 5109 cgca_art 0.0 0.005 4 1 2 4002 read_loads 0.0 0.005 2 1 3 2969 read_p121 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.002 0.093 1 1 93 max_p 0.0 0.002 80 1 1 80200 read_rest 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 22, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.184 2.665 3.952 0.8043 MPI-IO Write Bandwidth (MB/s) 7 5.184 2.665 3.952 0.8043 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.992E+04 6.309E+04 2795 Message size received from all nodes 1.229E+04 6.766E+04 4 6.306E+04 2831 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 6144 6.504E+04 3.252E+04 6.244E+04 2624 Message size sent to node 21 3071 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 21 : gather => MPI_Isend() 1 3.252E+04 3.252E+04 3.252E+04 0 Message size sent to node 21 : make_ggl => MPI_Isend() 3071 6.504E+04 6.504E+04 6.504E+04 0 Message size sent to node 21 : scatter => MPI_Isend() 1 6.504E+04 6.504E+04 6.504E+04 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 6144 6.622E+04 3.062E+04 6.372E+04 2527 Message size sent to node 23 3071 6.622E+04 6.622E+04 6.622E+04 0 Message size sent to node 23 : gather => MPI_Isend() 1 3.062E+04 3.062E+04 3.062E+04 0 Message size sent to node 23 : make_ggl => MPI_Isend() 3071 6.123E+04 6.123E+04 6.123E+04 0 Message size sent to node 23 : scatter => MPI_Isend() 1 6.123E+04 6.123E+04 6.123E+04 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 6.504E+04 6.504E+04 6.504E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 6.504E+04 6.504E+04 6.504E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 1 6.504E+04 6.504E+04 6.504E+04 0 [GROUP=MAX_MARKER] Message size sent to node 21 1 6.504E+04 6.504E+04 6.504E+04 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 1 6.123E+04 6.123E+04 6.123E+04 0 [GROUP=MAX_MARKER] Message size sent to node 23 1 6.123E+04 6.123E+04 6.123E+04 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.992E+04 2.992E+04 2.992E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.992E+04 2.992E+04 2.992E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 23;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 60.4 4:42.760 4:42.760 1.12021E+07 0 25 MPI_Win_unlock() 24.8 1:56.174 1:56.174 17279 0 6723 MPI_Barrier() 99.7 38,753 7:46.949 1 503146 466949773 xx14std 53.8 14,145 4:12.021 511 1.12372E+07 493193 cgca_hxi 50.0 3,712 3:54.250 5 269768 46850077 cgca_clvgp_nocosum 1.1 2,640 5,370 3071 36852 1749 gather 0.5 2,433 2,433 15338 0 159 MPI_Allreduce() 1.2 1,349 5,503 3071 33781 1792 scatter 0.2 1,007 1,007 12289 0 82 MPI_Recv() 0.2 959 959 1 0 959493 MPI_Init() 0.2 869 869 7 0 124260 MPI_File_write_all() 0.1 496 496 135 0 3678 MPI_Bcast() 30.3 393 2:21.750 1 51201 141750393 cgca_sld 0.1 356 356 1 0 356305 MPI_Finalize() 0.4 315 1,848 9198 9198 201 dot_product_p 0.1 246 269 100001 100001 3 cgca_clvgn [THROTTLED] 0.1 240 240 12288 0 20 MPI_Isend() 0.0 194 194 12288 0 16 MPI_Probe() 0.2 183 1,081 3066 6132 353 checon_par 100.0 146 7:48.442 1 524 468442088 .TAU application 0.0 143 143 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 99 99 155 0 643 cgca_dacf 0.0 69 69 100001 0 1 beemat [THROTTLED] 5.3 64 24,830 5 20000 4966032 cgca_pfem_simg 0.0 54 54 7 0 7816 MPI_File_open() 0.0 51 51 100001 0 1 MPI_Get() [THROTTLED] 0.0 46 46 100001 0 0 deemat [THROTTLED] 0.0 44 44 2 0 22037 cgca_gbs 0.1 42 651 5 30000 130264 cgca_pfem_intcalc1 1.0 42 4,879 12284 12284 397 my_barrier 0.0 40 40 6144 0 7 MPI_Waitall() 0.0 39 39 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 37 37 100001 0 0 shape_der [THROTTLED] 0.0 36 36 6 0 6076 MPI_Send() 0.0 29 29 67 0 439 MPI_Reduce() 0.0 28 28 100001 0 0 invert [THROTTLED] 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22372 load 0.0 19 19 80000 0 0 determinant 0.0 13 13 13 0 1008 MPI_Win_create() 0.0 12 12 10092 0 1 MPI_Put() 0.0 10 10 1 0 10029 MPI_Comm_dup() 0.0 9 9 12288 0 1 MPI_Get_count() 0.0 9 9 14 0 652 MPI_Type_create_subarray() 0.0 9 9 1 0 9061 cgca_igb 0.0 8 8 7 0 1163 MPI_File_close() 0.0 7 7 1 43 7599 cgca_gcu 0.0 6 14 1 19 14109 make_ggl 0.0 5 5 2000 0 3 find_g3 0.0 5 81 155 1706 524 cgca_gcupdn 0.0 4 4 1 0 4731 allocate_gather_scatter 0.0 4 4 7 0 601 MPI_File_set_view() 0.1 4 395 1 33 395396 read_g_coord_pp 0.0 4 48 1 160 48749 scatter_nodes 0.0 3 3 12 0 251 MPI_Win_free() 0.0 2 2 5 0 436 cgca_fv 0.0 1 62 1 576 62179 cgca_pfem_cenc 0.4 0.791 1,732 7 525 247540 cgca_pswci 0.0 0.617 0.617 1 0 617 getname 0.0 0.599 1 1 2 1469 cgca_as 0.7 0.304 3,082 1 64 3082761 cgca_ds 0.0 0.304 0.651 1 9 651 scatter_noadd 0.0 0.22 0.22 1 0 220 rearrange 0.0 0.217 0.217 1 0 217 cgca_pfem_salloc 0.0 0.205 0.205 1 0 205 calc_nels_pp 0.0 0.2 7 1 374 7923 cgca_nr 0.0 0.16 0.16 14 0 11 MPI_Type_commit() 0.0 0.147 5 1 377 5916 cgca_rt 0.0 0.123 0.123 1 0 123 cgca_ins 0.1 0.104 398 1 2 398364 read_g_num_pp 0.0 0.095 4 1 65 4474 gcupd_alloc 0.0 0.084 9 1 64 9284 cgca_drt 0.0 0.075 0.075 5 0 15 cgca_pfem_uym 0.0 0.057 1 5 5 220 sum_p 0.0 0.042 0.042 1 0 42 cgca_cadim 0.0 0.041 0.041 1 0 41 abaqus2sg 0.0 0.04 0.04 7 0 6 elap_time 0.0 0.031 0.031 1 0 31 cgca_pfem_sdalloc 0.0 0.029 0.029 1 0 29 cgca_gdim 0.0 0.029 0.029 5 0 6 sample 0.0 0.025 36 3 6 12159 dismsh_ensi_p 0.0 0.023 0.023 1 0 23 calc_neq_pp 0.0 0.02 0.021 1 2 21 find_pe_procs 0.0 0.018 0.018 1 0 18 cgca_pfem_ctalloc 0.0 0.016 0.016 12 0 1 MPI_Win_fence() 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.012 0.012 2 0 6 MPI_Test() 0.0 0.01 0.01 9 0 1 MPI_Comm_rank() 0.0 0.009 0.009 14 0 1 MPI_Type_free() 0.0 0.009 0.009 22 0 0 cgca_agc 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.008 0.008 21 0 0 cgca_dgc 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.006 4 1 2 4711 cgca_art 0.0 0.006 4 1 3 4848 read_p121 0.0 0.005 4 1 2 4031 read_loads 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.003 0.23 1 1 230 max_p 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 cgca_gcf 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.002 80 1 1 80216 read_rest 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ctdalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 23, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.18 2.665 3.949 0.8031 MPI-IO Write Bandwidth (MB/s) 7 5.18 2.665 3.949 0.8031 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 3.062E+04 6.083E+04 7570 Message size received from all nodes 1.229E+04 6.766E+04 4 6.08E+04 7579 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 6144 6.622E+04 3.311E+04 6.372E+04 2522 Message size sent to node 22 3071 6.123E+04 6.123E+04 6.123E+04 0 Message size sent to node 22 : gather => MPI_Isend() 1 3.311E+04 3.311E+04 3.311E+04 0 Message size sent to node 22 : make_ggl => MPI_Isend() 3071 6.622E+04 6.622E+04 6.622E+04 0 Message size sent to node 22 : scatter => MPI_Isend() 1 6.622E+04 6.622E+04 6.622E+04 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 6144 6.739E+04 2.422E+04 5.791E+04 9481 Message size sent to node 24 3071 6.739E+04 6.739E+04 6.739E+04 0 Message size sent to node 24 : gather => MPI_Isend() 1 2.422E+04 2.422E+04 2.422E+04 0 Message size sent to node 24 : make_ggl => MPI_Isend() 3071 4.845E+04 4.845E+04 4.845E+04 0 Message size sent to node 24 : scatter => MPI_Isend() 1 4.845E+04 4.845E+04 4.845E+04 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 6.622E+04 6.622E+04 6.622E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 6.622E+04 6.622E+04 6.622E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 1 6.622E+04 6.622E+04 6.622E+04 0 [GROUP=MAX_MARKER] Message size sent to node 22 1 6.622E+04 6.622E+04 6.622E+04 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 1 4.845E+04 4.845E+04 4.845E+04 0 [GROUP=MAX_MARKER] Message size sent to node 24 1 4.845E+04 4.845E+04 4.845E+04 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.062E+04 3.062E+04 3.062E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.062E+04 3.062E+04 3.062E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 24;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 55.7 4:20.724 4:20.724 1.12021E+07 0 23 MPI_Win_unlock() 28.9 2:15.550 2:15.550 17279 0 7845 MPI_Barrier() 99.7 39,455 7:46.947 1 503146 466947624 xx14std 54.9 15,245 4:17.082 511 1.12372E+07 503096 cgca_hxi 50.0 4,281 3:54.250 5 269768 46850028 cgca_clvgp_nocosum 1.2 2,627 5,515 3071 36852 1796 gather 0.5 2,474 2,474 15338 0 161 MPI_Allreduce() 1.0 1,279 4,650 3071 33781 1514 scatter 0.2 971 971 1 0 971361 MPI_Init() 0.2 965 965 12289 0 79 MPI_Recv() 0.2 868 868 7 0 124119 MPI_File_write_all() 0.1 682 682 100001 0 7 MPI_Win_lock() [THROTTLED] 0.1 489 489 135 0 3624 MPI_Bcast() 0.1 371 371 1 0 371450 MPI_Finalize() 30.3 367 2:21.740 1 51201 141740517 cgca_sld 0.4 292 1,831 9198 9198 199 dot_product_p 0.1 270 270 12288 0 22 MPI_Isend() 0.1 213 235 100001 100001 2 cgca_clvgn [THROTTLED] 0.2 166 1,099 3066 6132 359 checon_par 100.0 152 7:48.475 1 496 468475534 .TAU application 0.0 142 142 12288 0 12 MPI_Probe() 0.0 92 92 155 0 598 cgca_dacf 0.0 67 67 100001 0 1 beemat [THROTTLED] 0.0 55 55 7 0 7961 MPI_File_open() 0.0 52 52 100001 0 1 MPI_Get() [THROTTLED] 0.0 47 47 100001 0 0 cgca_clvgsd [THROTTLED] 0.8 47 3,573 5 20000 714789 cgca_pfem_simg 0.0 44 44 100001 0 0 deemat [THROTTLED] 0.0 44 44 2 0 22064 cgca_gbs 0.0 41 41 6144 0 7 MPI_Waitall() 0.9 40 4,331 12284 12284 353 my_barrier 0.0 37 37 6 0 6332 MPI_Send() 0.0 36 36 100001 0 0 shape_der [THROTTLED] 0.0 35 35 100001 0 0 invert [THROTTLED] 0.0 29 29 67 0 434 MPI_Reduce() 0.1 28 535 5 30000 107050 cgca_pfem_intcalc1 0.0 21 21 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22291 load 0.0 20 20 80000 0 0 determinant 0.0 11 11 13 0 888 MPI_Win_create() 0.0 10 10 1 0 10079 MPI_Comm_dup() 0.0 9 9 7 0 1348 MPI_File_close() 0.0 9 9 14 0 652 MPI_Type_create_subarray() 0.0 8 8 1 0 8485 cgca_igb 0.0 7 7 12288 0 1 MPI_Get_count() 0.0 6 6 1 43 6690 cgca_gcu 0.0 6 14 1 19 14066 make_ggl 0.0 6 6 10081 0 1 MPI_Put() 0.0 5 82 155 1727 530 cgca_gcupdn 0.0 5 5 1 0 5273 allocate_gather_scatter 0.0 5 5 2000 0 3 find_g3 0.1 4 389 1 33 389117 read_g_coord_pp 0.0 4 49 1 160 49334 scatter_nodes 0.0 3 3 7 0 542 MPI_File_set_view() 0.0 3 3 12 0 266 MPI_Win_free() 0.0 2 2 5 0 429 cgca_fv 0.0 1 66 1 576 66545 cgca_pfem_cenc 0.4 0.873 1,733 7 525 247660 cgca_pswci 0.0 0.765 2 1 2 2329 cgca_as 0.0 0.336 0.336 1 0 336 getname 0.0 0.274 0.664 1 9 664 scatter_noadd 0.7 0.237 3,081 1 64 3081206 cgca_ds 0.0 0.21 0.21 1 0 210 rearrange 0.0 0.201 0.201 14 0 14 MPI_Type_commit() 0.0 0.197 0.197 1 0 197 calc_nels_pp 0.0 0.184 7 1 374 7855 cgca_nr 0.0 0.178 0.178 1 0 178 cgca_pfem_salloc 0.0 0.173 0.173 1 0 173 cgca_ins 0.0 0.142 9 1 64 9260 cgca_drt 0.0 0.14 5 1 377 5952 cgca_rt 0.1 0.119 405 1 2 405407 read_g_num_pp 0.0 0.099 0.099 5 0 20 cgca_pfem_uym 0.0 0.087 4 1 65 4537 gcupd_alloc 0.0 0.054 0.054 7 0 8 elap_time 0.0 0.053 0.053 1 0 53 abaqus2sg 0.0 0.051 0.051 1 0 51 cgca_gdim 0.0 0.046 1 5 5 219 sum_p 0.0 0.038 0.038 1 0 38 cgca_pfem_sdalloc 0.0 0.033 0.033 5 0 7 sample 0.0 0.03 0.03 2 0 15 MPI_Test() 0.0 0.03 38 3 6 12675 dismsh_ensi_p 0.0 0.027 0.027 1 0 27 cgca_cadim 0.0 0.027 0.027 22 0 1 cgca_gcf 0.0 0.024 0.024 1 0 24 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.017 1 2 17 find_pe_procs 0.0 0.013 0.013 14 0 1 MPI_Type_free() 0.0 0.013 0.013 1 0 13 calc_neq_pp 0.0 0.013 0.013 1 0 13 cgca_pfem_ctalloc 0.0 0.012 0.012 22 0 1 cgca_agc 0.0 0.012 0.012 21 0 1 cgca_dgc 0.0 0.011 0.011 12 0 1 MPI_Win_fence() 0.0 0.01 0.01 1 0 10 caflaunch$MAIN$blk 0.0 0.009 0.009 9 0 1 MPI_Comm_rank() 0.0 0.009 3 1 3 3034 read_p121 0.0 0.007 4 1 2 4109 read_loads 0.0 0.005 0.005 1 0 5 MPI_Alloc_mem() 0.0 0.005 0.005 9 0 1 MPI_Comm_size() 0.0 0.005 0.48 1 2 480 cgca_art 0.0 0.003 0.003 1 0 3 MPI_Finalized() 0.0 0.003 0.296 1 1 296 max_p 0.0 0.003 80 1 1 80491 read_rest 0.0 0.002 0.002 1 0 2 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc 0.0 0 0 1 0 0 cgca_imco --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 24, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.189 2.666 3.955 0.8058 MPI-IO Write Bandwidth (MB/s) 7 5.189 2.666 3.955 0.8058 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.422E+04 6.107E+04 1.242E+04 Message size received from all nodes 1.229E+04 7.825E+04 4 6.105E+04 1.242E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 6144 6.739E+04 3.37E+04 5.792E+04 9476 Message size sent to node 23 3071 4.845E+04 4.845E+04 4.845E+04 0 Message size sent to node 23 : gather => MPI_Isend() 1 3.37E+04 3.37E+04 3.37E+04 0 Message size sent to node 23 : make_ggl => MPI_Isend() 3071 6.739E+04 6.739E+04 6.739E+04 0 Message size sent to node 23 : scatter => MPI_Isend() 1 6.739E+04 6.739E+04 6.739E+04 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 6144 7.825E+04 2.508E+04 6.42E+04 1.405E+04 Message size sent to node 25 3071 7.825E+04 7.825E+04 7.825E+04 0 Message size sent to node 25 : gather => MPI_Isend() 1 2.508E+04 2.508E+04 2.508E+04 0 Message size sent to node 25 : make_ggl => MPI_Isend() 3071 5.017E+04 5.017E+04 5.017E+04 0 Message size sent to node 25 : scatter => MPI_Isend() 1 5.017E+04 5.017E+04 5.017E+04 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 6.739E+04 6.739E+04 6.739E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 6.739E+04 6.739E+04 6.739E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 1 6.739E+04 6.739E+04 6.739E+04 0 [GROUP=MAX_MARKER] Message size sent to node 23 1 6.739E+04 6.739E+04 6.739E+04 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 2 7.825E+04 5.017E+04 6.421E+04 1.404E+04 [GROUP=MAX_MARKER] Message size sent to node 25 1 7.825E+04 7.825E+04 7.825E+04 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 1 5.017E+04 5.017E+04 5.017E+04 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.912E+04 3.912E+04 3.912E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.912E+04 3.912E+04 3.912E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 25;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 58.7 4:35.068 4:35.068 1.44714E+07 0 19 MPI_Win_unlock() 25.1 1:57.366 1:57.366 17279 0 6792 MPI_Barrier() 99.7 39,341 7:46.947 1 503146 466947650 xx14std 59.5 18,351 4:38.626 511 1.45061E+07 545258 cgca_hxi 50.0 5,304 3:54.250 5 369773 46850122 cgca_clvgp_nocosum 1.2 2,516 5,498 3071 36852 1790 gather 0.5 2,346 2,346 15338 0 153 MPI_Allreduce() 1.0 1,252 4,868 3071 33781 1585 scatter 0.2 960 960 1 0 960477 MPI_Init() 0.2 958 958 12289 0 78 MPI_Recv() 0.2 868 868 7 0 124101 MPI_File_write_all() 0.1 697 697 100001 0 7 MPI_Win_lock() [THROTTLED] 0.1 481 481 135 0 3568 MPI_Bcast() 0.1 391 391 1 0 391105 MPI_Finalize() 30.3 388 2:21.740 1 50703 141740533 cgca_sld 0.4 310 1,756 9198 9198 191 dot_product_p 0.1 261 261 12288 0 21 MPI_Isend() 0.1 230 253 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 181 1,079 3066 6132 352 checon_par 100.0 164 7:48.495 1 491 468495444 .TAU application 0.0 148 148 12288 0 12 MPI_Probe() 0.0 90 90 155 0 584 cgca_dacf 0.0 66 66 100001 0 1 beemat [THROTTLED] 0.9 61 3,998 5 20000 799644 cgca_pfem_simg 0.0 55 55 7 0 7982 MPI_File_open() 0.0 52 52 100001 0 1 MPI_Get() [THROTTLED] 0.0 51 51 100001 0 1 deemat [THROTTLED] 0.0 44 44 100001 0 0 shape_der [THROTTLED] 0.0 43 43 2 0 21985 cgca_gbs 1.0 37 4,639 12284 12284 378 my_barrier 0.0 36 36 6 0 6073 MPI_Send() 0.0 34 34 6144 0 6 MPI_Waitall() 0.0 32 32 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 29 29 67 0 442 MPI_Reduce() 0.1 27 607 5 30000 121568 cgca_pfem_intcalc1 0.0 25 25 100001 0 0 cgca_gcr [THROTTLED] 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22286 load 0.0 19 19 80000 0 0 determinant 0.0 10 10 13 0 844 MPI_Win_create() 0.0 10 10 1 0 10050 MPI_Comm_dup() 0.0 9 9 7 0 1322 MPI_File_close() 0.0 9 9 14 0 652 MPI_Type_create_subarray() 0.0 8 8 1 0 8182 cgca_igb 0.0 7 7 1 53 7848 cgca_gcu 0.0 7 129 155 2654 832 cgca_gcupdn 0.0 7 7 12288 0 1 MPI_Get_count() 0.0 6 14 1 19 14121 make_ggl 0.0 5 5 10079 0 1 MPI_Put() 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4805 allocate_gather_scatter 0.1 4 382 1 33 382073 read_g_coord_pp 0.0 4 49 1 160 49348 scatter_nodes 0.0 4 4 7 0 582 MPI_File_set_view() 0.0 3 3 12 0 271 MPI_Win_free() 0.0 2 2 5 0 410 cgca_fv 0.0 1 66 1 576 66442 cgca_pfem_cenc 0.4 0.779 1,733 7 525 247626 cgca_pswci 0.0 0.607 1 1 2 1447 cgca_as 0.0 0.341 0.341 1 0 341 getname 0.0 0.329 0.651 1 9 651 scatter_noadd 0.7 0.211 3,082 1 64 3082769 cgca_ds 0.0 0.206 0.206 1 0 206 rearrange 0.0 0.182 7 1 374 7908 cgca_nr 0.0 0.181 0.181 14 0 13 MPI_Type_commit() 0.0 0.173 0.173 1 0 173 cgca_pfem_salloc 0.0 0.172 0.172 1 0 172 calc_nels_pp 0.0 0.121 5 1 377 5920 cgca_rt 0.0 0.107 0.107 1 0 107 cgca_ins 0.1 0.107 412 1 2 412520 read_g_num_pp 0.0 0.094 0.094 5 0 19 cgca_pfem_uym 0.0 0.08 9 1 64 9249 cgca_drt 0.0 0.073 4 1 65 4406 gcupd_alloc 0.0 0.051 0.051 1 0 51 abaqus2sg 0.0 0.047 1 5 5 249 sum_p 0.0 0.043 0.043 1 0 43 cgca_pfem_sdalloc 0.0 0.042 0.042 7 0 6 elap_time 0.0 0.03 36 3 6 12155 dismsh_ensi_p 0.0 0.024 0.024 5 0 5 sample 0.0 0.023 0.023 12 0 2 MPI_Win_fence() 0.0 0.023 0.023 1 0 23 cgca_pfem_ctalloc 0.0 0.019 0.019 23 0 1 cgca_gcf 0.0 0.017 0.017 27 0 1 cgca_agc 0.0 0.016 0.017 1 2 17 find_pe_procs 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.015 1 0 15 calc_neq_pp 0.0 0.012 0.012 26 0 0 cgca_dgc 0.0 0.01 0.01 2 0 5 MPI_Test() 0.0 0.01 0.01 14 0 1 MPI_Type_free() 0.0 0.01 0.01 1 0 10 cgca_gdim 0.0 0.009 0.009 1 0 9 cgca_cadim 0.0 0.007 0.007 1 0 7 caflaunch$MAIN$blk 0.0 0.006 0.006 9 0 1 MPI_Comm_rank() 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.006 3 1 3 3072 read_p121 0.0 0.005 0.515 1 2 515 cgca_art 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.003 4 1 2 4065 read_loads 0.0 0.003 80 1 1 80491 read_rest 0.0 0.002 0.002 1 0 2 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.002 0.467 1 1 467 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 25, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.19 2.667 3.955 0.8055 MPI-IO Write Bandwidth (MB/s) 7 5.19 2.667 3.955 0.8055 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.508E+04 6.45E+04 1.057E+04 Message size received from all nodes 1.229E+04 7.825E+04 4 6.448E+04 1.058E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 6144 7.825E+04 3.912E+04 6.421E+04 1.404E+04 Message size sent to node 24 3071 5.017E+04 5.017E+04 5.017E+04 0 Message size sent to node 24 : gather => MPI_Isend() 1 3.912E+04 3.912E+04 3.912E+04 0 Message size sent to node 24 : make_ggl => MPI_Isend() 3071 7.825E+04 7.825E+04 7.825E+04 0 Message size sent to node 24 : scatter => MPI_Isend() 1 7.825E+04 7.825E+04 7.825E+04 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 6144 6.974E+04 2.992E+04 6.479E+04 4968 Message size sent to node 26 3071 6.974E+04 6.974E+04 6.974E+04 0 Message size sent to node 26 : gather => MPI_Isend() 1 2.992E+04 2.992E+04 2.992E+04 0 Message size sent to node 26 : make_ggl => MPI_Isend() 3071 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 26 : scatter => MPI_Isend() 1 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 7.825E+04 7.825E+04 7.825E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 7.825E+04 7.825E+04 7.825E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 1 7.825E+04 7.825E+04 7.825E+04 0 [GROUP=MAX_MARKER] Message size sent to node 24 1 7.825E+04 7.825E+04 7.825E+04 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 1 5.985E+04 5.985E+04 5.985E+04 0 [GROUP=MAX_MARKER] Message size sent to node 26 1 5.985E+04 5.985E+04 5.985E+04 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.508E+04 2.508E+04 2.508E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.508E+04 2.508E+04 2.508E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 26;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 59.1 4:37.068 4:37.068 1.44714E+07 0 19 MPI_Win_unlock() 24.9 1:56.831 1:56.831 17279 0 6761 MPI_Barrier() 99.7 39,336 7:46.947 1 503146 466947623 xx14std 59.9 18,127 4:40.546 511 1.45061E+07 549015 cgca_hxi 50.0 4,546 3:54.250 5 269768 46850124 cgca_clvgp_nocosum 1.2 2,522 5,400 3071 36852 1759 gather 0.5 2,445 2,445 15338 0 159 MPI_Allreduce() 1.0 1,297 4,891 3071 33781 1593 scatter 0.2 961 961 12289 0 78 MPI_Recv() 0.2 961 961 1 0 961587 MPI_Init() 0.2 869 869 7 0 124153 MPI_File_write_all() 0.1 474 474 135 0 3511 MPI_Bcast() 0.1 403 403 1 0 403904 MPI_Finalize() 30.3 378 2:21.750 1 50703 141750292 cgca_sld 0.4 303 1,849 9198 9198 201 dot_product_p 0.1 255 255 12288 0 21 MPI_Isend() 0.1 235 258 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 178 1,074 3066 6132 350 checon_par 100.0 162 7:48.508 1 495 468508240 .TAU application 0.0 151 151 12288 0 12 MPI_Probe() 0.0 128 128 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 87 87 155 0 566 cgca_dacf 0.0 66 66 100001 0 1 beemat [THROTTLED] 1.0 59 4,521 5 20000 904211 cgca_pfem_simg 0.0 55 55 7 0 7944 MPI_File_open() 0.0 53 53 100001 0 1 MPI_Get() [THROTTLED] 0.0 45 45 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21956 cgca_gbs 0.0 38 38 6144 0 6 MPI_Waitall() 0.0 36 36 100001 0 0 shape_der [THROTTLED] 1.0 36 4,554 12284 12284 371 my_barrier 0.0 36 36 6 0 6024 MPI_Send() 0.0 34 34 100001 0 0 invert [THROTTLED] 0.0 31 31 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 29 29 67 0 444 MPI_Reduce() 0.2 28 722 5 30000 144510 cgca_pfem_intcalc1 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22213 load 0.0 20 20 80000 0 0 determinant 0.0 10 10 13 0 843 MPI_Win_create() 0.0 10 10 1 0 10094 MPI_Comm_dup() 0.0 9 9 14 0 650 MPI_Type_create_subarray() 0.0 8 8 7 0 1270 MPI_File_close() 0.0 8 8 1 0 8751 cgca_igb 0.0 8 8 1 65 8483 cgca_gcu 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 7 117 155 2655 758 cgca_gcupdn 0.0 6 14 1 19 14114 make_ggl 0.0 6 6 10081 0 1 MPI_Put() 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4852 allocate_gather_scatter 0.1 4 374 1 33 374960 read_g_coord_pp 0.0 4 48 1 160 48402 scatter_nodes 0.0 3 3 7 0 517 MPI_File_set_view() 0.0 3 3 12 0 253 MPI_Win_free() 0.0 2 2 5 0 414 cgca_fv 0.0 1 66 1 576 66417 cgca_pfem_cenc 0.4 0.803 1,732 7 525 247561 cgca_pswci 0.0 0.604 1 1 2 1471 cgca_as 0.0 0.341 0.341 1 0 341 getname 0.0 0.277 0.564 1 9 564 scatter_noadd 0.7 0.254 3,083 1 64 3083038 cgca_ds 0.0 0.211 0.211 1 0 211 rearrange 0.0 0.177 7 1 374 7860 cgca_nr 0.0 0.169 0.169 1 0 169 calc_nels_pp 0.0 0.166 0.166 1 0 166 cgca_pfem_salloc 0.0 0.165 0.165 14 0 12 MPI_Type_commit() 0.0 0.131 0.131 1 0 131 cgca_ins 0.1 0.122 419 1 2 419595 read_g_num_pp 0.0 0.115 5 1 377 5941 cgca_rt 0.0 0.103 9 1 64 9255 cgca_drt 0.0 0.096 0.096 5 0 19 cgca_pfem_uym 0.0 0.064 4 1 65 4461 gcupd_alloc 0.0 0.053 0.053 1 0 53 abaqus2sg 0.0 0.045 1 5 5 229 sum_p 0.0 0.04 0.04 1 0 40 cgca_gdim 0.0 0.039 0.039 1 0 39 cgca_cadim 0.0 0.038 0.038 1 0 38 cgca_pfem_sdalloc 0.0 0.036 0.036 20 0 2 cgca_gcf 0.0 0.036 0.036 7 0 5 elap_time 0.0 0.033 0.033 5 0 7 sample 0.0 0.022 36 3 6 12056 dismsh_ensi_p 0.0 0.019 0.019 33 0 1 cgca_agc 0.0 0.019 0.019 1 0 19 cgca_pfem_ctalloc 0.0 0.017 0.017 14 0 1 MPI_Type_free() 0.0 0.017 0.017 12 0 1 MPI_Win_fence() 0.0 0.016 0.016 1 2 16 find_pe_procs 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.015 1 0 15 calc_neq_pp 0.0 0.012 0.012 2 0 6 MPI_Test() 0.0 0.009 0.009 32 0 0 cgca_dgc 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.007 3 1 3 3090 read_p121 0.0 0.005 0.005 9 0 1 MPI_Comm_rank() 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 0.004 9 0 0 MPI_Comm_size() 0.0 0.004 0.534 1 2 534 cgca_art 0.0 0.003 0.419 1 1 419 max_p 0.0 0.003 4 1 2 4142 read_loads 0.0 0.003 80 1 1 80811 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 26, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.183 2.665 3.953 0.8045 MPI-IO Write Bandwidth (MB/s) 7 5.183 2.665 3.953 0.8045 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.992E+04 6.426E+04 6270 Message size received from all nodes 1.229E+04 7.092E+04 4 6.424E+04 6291 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 6144 6.974E+04 3.487E+04 6.479E+04 4962 Message size sent to node 25 3071 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 25 : gather => MPI_Isend() 1 3.487E+04 3.487E+04 3.487E+04 0 Message size sent to node 25 : make_ggl => MPI_Isend() 3071 6.974E+04 6.974E+04 6.974E+04 0 Message size sent to node 25 : scatter => MPI_Isend() 1 6.974E+04 6.974E+04 6.974E+04 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 6144 7.092E+04 2.826E+04 6.372E+04 7210 Message size sent to node 27 3071 7.092E+04 7.092E+04 7.092E+04 0 Message size sent to node 27 : gather => MPI_Isend() 1 2.826E+04 2.826E+04 2.826E+04 0 Message size sent to node 27 : make_ggl => MPI_Isend() 3071 5.653E+04 5.653E+04 5.653E+04 0 Message size sent to node 27 : scatter => MPI_Isend() 1 5.653E+04 5.653E+04 5.653E+04 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 6.974E+04 6.974E+04 6.974E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 6.974E+04 6.974E+04 6.974E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 1 6.974E+04 6.974E+04 6.974E+04 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 1 6.974E+04 6.974E+04 6.974E+04 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 1 5.653E+04 5.653E+04 5.653E+04 0 [GROUP=MAX_MARKER] Message size sent to node 27 1 5.653E+04 5.653E+04 5.653E+04 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.546E+04 3.546E+04 3.546E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.546E+04 3.546E+04 3.546E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 27;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 58.0 4:31.654 4:31.654 1.12021E+07 0 24 MPI_Win_unlock() 27.8 2:10.106 2:10.106 17279 0 7530 MPI_Barrier() 99.7 39,140 7:46.947 1 503146 466947635 xx14std 55.8 14,207 4:21.621 511 1.12372E+07 511980 cgca_hxi 1.2 2,543 5,431 3071 36852 1769 gather 0.5 2,430 2,430 15338 0 158 MPI_Allreduce() 1.1 1,309 5,060 3071 33781 1648 scatter 0.2 962 962 12289 0 78 MPI_Recv() 0.2 962 962 1 0 962003 MPI_Init() 50.0 901 3:54.250 5 169767 46850122 cgca_clvgp_nocosum 0.2 869 869 7 0 124275 MPI_File_write_all() 0.1 467 467 135 0 3463 MPI_Bcast() 0.1 418 418 1 0 418902 MPI_Finalize() 30.3 344 2:21.750 1 51201 141750387 cgca_sld 0.4 307 1,847 9198 9198 201 dot_product_p 0.1 251 251 12288 0 20 MPI_Isend() 0.1 233 263 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 183 1,070 3066 6132 349 checon_par 100.0 162 7:48.522 1 506 468522946 .TAU application 0.0 140 140 12288 0 11 MPI_Probe() 0.0 91 91 155 0 590 cgca_dacf 0.0 85 85 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 66 66 100001 0 1 beemat [THROTTLED] 0.0 55 55 7 0 7904 MPI_File_open() 0.0 51 51 100001 0 1 MPI_Get() [THROTTLED] 1.0 49 4,611 5 20000 922206 cgca_pfem_simg 0.0 45 45 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21921 cgca_gbs 0.0 41 41 6144 0 7 MPI_Waitall() 1.0 39 4,672 12284 12284 380 my_barrier 0.0 38 38 100001 0 0 shape_der [THROTTLED] 0.0 36 36 6 0 6067 MPI_Send() 0.2 30 882 5 30000 176509 cgca_pfem_intcalc1 0.0 29 29 100001 0 0 cgca_csym [THROTTLED] 0.0 28 28 100001 0 0 invert [THROTTLED] 0.0 28 28 67 0 422 MPI_Reduce() 0.0 26 26 80000 0 0 determinant 0.0 21 22 1 1 22394 load 0.0 11 11 13 0 877 MPI_Win_create() 0.0 10 10 1 0 10056 MPI_Comm_dup() 0.0 9 9 14 0 652 MPI_Type_create_subarray() 0.0 9 9 1 0 9069 cgca_igb 0.0 8 8 7 0 1165 MPI_File_close() 0.0 7 7 12288 0 1 MPI_Get_count() 0.0 7 7 10085 0 1 MPI_Put() 0.0 6 14 1 19 14144 make_ggl 0.0 5 5 1 23 5818 cgca_gcu 0.0 5 48 1 160 48392 scatter_nodes 0.0 5 5 2000 0 3 find_g3 0.0 5 5 1 0 5037 allocate_gather_scatter 0.0 4 88 155 1706 572 cgca_gcupdn 0.1 4 367 1 33 367997 read_g_coord_pp 0.0 3 3 7 0 506 MPI_File_set_view() 0.0 3 3 12 0 256 MPI_Win_free() 0.0 2 2 5 0 415 cgca_fv 0.0 1 65 1 576 65912 cgca_pfem_cenc 0.4 0.795 1,732 7 525 247534 cgca_pswci 0.0 0.607 1 1 2 1477 cgca_as 0.0 0.347 0.347 1 0 347 getname 0.0 0.298 0.728 1 9 728 scatter_noadd 0.0 0.214 0.214 1 0 214 rearrange 0.7 0.201 3,082 1 64 3082753 cgca_ds 0.0 0.185 7 1 374 7929 cgca_nr 0.0 0.171 0.171 1 0 171 cgca_pfem_salloc 0.0 0.149 0.149 1 0 149 calc_nels_pp 0.0 0.14 0.14 14 0 10 MPI_Type_commit() 0.0 0.139 5 1 377 5900 cgca_rt 0.1 0.107 426 1 2 426601 read_g_num_pp 0.0 0.089 0.089 1 0 89 cgca_ins 0.0 0.077 9 1 64 9257 cgca_drt 0.0 0.075 0.075 5 0 15 cgca_pfem_uym 0.0 0.073 4 1 65 4409 gcupd_alloc 0.0 0.05 0.05 1 0 50 abaqus2sg 0.0 0.05 1 5 5 289 sum_p 0.0 0.042 0.042 1 0 42 cgca_cadim 0.0 0.039 0.039 1 0 39 cgca_pfem_sdalloc 0.0 0.038 0.038 1 0 38 cgca_gdim 0.0 0.038 0.038 7 0 5 elap_time 0.0 0.028 36 3 6 12143 dismsh_ensi_p 0.0 0.026 0.026 14 0 2 MPI_Type_free() 0.0 0.023 0.023 1 0 23 calc_neq_pp 0.0 0.017 0.017 12 0 1 MPI_Win_fence() 0.0 0.017 0.017 5 0 3 sample 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.017 1 2 17 find_pe_procs 0.0 0.014 0.014 1 0 14 cgca_pfem_ctalloc 0.0 0.012 0.012 2 0 6 MPI_Test() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.008 0.008 12 0 1 cgca_agc 0.0 0.008 3 1 3 3129 read_p121 0.0 0.007 0.007 9 0 1 MPI_Comm_rank() 0.0 0.007 0.007 9 0 1 MPI_Comm_size() 0.0 0.006 1 1 2 1104 cgca_art 0.0 0.006 0.006 11 0 1 cgca_dgc 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 4 1 2 4184 read_loads 0.0 0.003 0.003 1 0 3 MPI_Finalized() 0.0 0.003 0.254 1 1 254 max_p 0.0 0.003 80 1 1 80902 read_rest 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_gcf 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 27, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.185 2.661 3.95 0.8058 MPI-IO Write Bandwidth (MB/s) 7 5.185 2.661 3.95 0.8058 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.826E+04 6.083E+04 1.165E+04 Message size received from all nodes 1.229E+04 7.21E+04 4 6.08E+04 1.165E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 6144 7.092E+04 3.546E+04 6.372E+04 7204 Message size sent to node 26 3071 5.653E+04 5.653E+04 5.653E+04 0 Message size sent to node 26 : gather => MPI_Isend() 1 3.546E+04 3.546E+04 3.546E+04 0 Message size sent to node 26 : make_ggl => MPI_Isend() 3071 7.092E+04 7.092E+04 7.092E+04 0 Message size sent to node 26 : scatter => MPI_Isend() 1 7.092E+04 7.092E+04 7.092E+04 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 6144 7.21E+04 2.187E+04 5.791E+04 1.418E+04 Message size sent to node 28 3071 7.21E+04 7.21E+04 7.21E+04 0 Message size sent to node 28 : gather => MPI_Isend() 1 2.187E+04 2.187E+04 2.187E+04 0 Message size sent to node 28 : make_ggl => MPI_Isend() 3071 4.374E+04 4.374E+04 4.374E+04 0 Message size sent to node 28 : scatter => MPI_Isend() 1 4.374E+04 4.374E+04 4.374E+04 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 7.092E+04 7.092E+04 7.092E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 7.092E+04 7.092E+04 7.092E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 1 7.092E+04 7.092E+04 7.092E+04 0 [GROUP=MAX_MARKER] Message size sent to node 26 1 7.092E+04 7.092E+04 7.092E+04 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 2 7.21E+04 4.374E+04 5.792E+04 1.418E+04 [GROUP=MAX_MARKER] Message size sent to node 28 1 7.21E+04 7.21E+04 7.21E+04 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 1 4.374E+04 4.374E+04 4.374E+04 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.605E+04 3.605E+04 3.605E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.605E+04 3.605E+04 3.605E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 28;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 52.8 4:07.173 4:07.173 8.01283E+06 0 31 MPI_Win_unlock() 33.1 2:35.198 2:35.198 17279 0 8982 MPI_Barrier() 99.7 38,872 7:46.947 1 503146 466947651 xx14std 43.1 11,092 3:21.727 511 8.04756E+06 394770 cgca_hxi 50.0 3,349 3:54.249 5 269768 46849913 cgca_clvgp_nocosum 1.2 2,578 5,494 3071 36852 1789 gather 0.5 2,448 2,448 15338 0 160 MPI_Allreduce() 1.1 1,340 5,268 3071 33781 1716 scatter 0.2 964 964 12289 0 79 MPI_Recv() 0.2 962 962 1 0 962943 MPI_Init() 0.2 869 869 7 0 124238 MPI_File_write_all() 30.3 468 2:21.750 1 52197 141750346 cgca_sld 0.1 460 460 135 0 3414 MPI_Bcast() 0.1 429 429 1 0 429139 MPI_Finalize() 0.4 298 1,827 9198 9198 199 dot_product_p 0.1 243 243 12288 0 20 MPI_Isend() 0.1 235 267 100001 100001 3 cgca_clvgn [THROTTLED] 0.0 167 167 100001 0 2 MPI_Win_lock() [THROTTLED] 0.2 166 1,083 3066 6132 353 checon_par 100.0 161 7:48.533 1 473 468533349 .TAU application 0.5 127 2,394 5 20000 478858 cgca_pfem_simg 0.0 101 101 12288 0 8 MPI_Probe() 0.0 90 90 155 0 586 cgca_dacf 0.0 77 77 100001 0 1 beemat [THROTTLED] 0.0 56 56 7 0 8015 MPI_File_open() 0.0 54 54 100001 0 1 MPI_Get() [THROTTLED] 0.0 45 45 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21919 cgca_gbs 0.0 39 39 6144 0 6 MPI_Waitall() 1.1 38 4,937 12284 12284 402 my_barrier 0.0 36 36 100001 0 0 shape_der [THROTTLED] 0.0 36 36 6 0 6068 MPI_Send() 0.0 33 33 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 32 32 100001 0 0 cgca_csym [THROTTLED] 0.0 29 29 67 0 436 MPI_Reduce() 0.0 28 28 100001 0 0 invert [THROTTLED] 0.0 21 22 1 1 22385 load 0.0 21 21 80000 0 0 determinant 0.0 19 151 5 30000 30270 cgca_pfem_intcalc1 0.0 15 15 13 0 1201 MPI_Win_create() 0.0 10 10 1 0 10144 MPI_Comm_dup() 0.0 9 9 1 0 9364 cgca_igb 0.0 9 9 14 0 651 MPI_Type_create_subarray() 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 8 8 7 0 1227 MPI_File_close() 0.0 6 14 1 19 14051 make_ggl 0.0 5 5 2000 0 3 find_g3 0.0 5 5 1 0 5026 allocate_gather_scatter 0.0 4 4 10072 0 0 MPI_Put() 0.0 4 4 1 11 4722 cgca_gcu 0.1 4 360 1 33 360972 read_g_coord_pp 0.0 4 48 1 160 48459 scatter_nodes 0.0 3 64 155 1091 413 cgca_gcupdn 0.0 3 3 7 0 459 MPI_File_set_view() 0.0 3 3 12 0 257 MPI_Win_free() 0.0 2 2 5 0 419 cgca_fv 0.0 1 61 1 576 61832 cgca_pfem_cenc 0.4 0.775 1,733 7 525 247588 cgca_pswci 0.0 0.671 2 1 2 2267 cgca_as 0.0 0.343 0.343 1 0 343 getname 0.0 0.316 0.659 1 9 659 scatter_noadd 0.7 0.25 3,082 1 64 3082759 cgca_ds 0.0 0.223 0.223 1 0 223 calc_nels_pp 0.0 0.214 0.214 1 0 214 rearrange 0.0 0.188 0.188 1 0 188 cgca_pfem_salloc 0.0 0.173 7 1 374 7868 cgca_nr 0.0 0.165 0.165 14 0 12 MPI_Type_commit() 0.0 0.153 5 1 377 5942 cgca_rt 0.0 0.112 4 1 65 4209 gcupd_alloc 0.1 0.112 433 1 2 433516 read_g_num_pp 0.0 0.088 0.088 1 0 88 cgca_ins 0.0 0.085 0.085 5 0 17 cgca_pfem_uym 0.0 0.084 9 1 64 9277 cgca_drt 0.0 0.05 0.05 1 0 50 abaqus2sg 0.0 0.048 0.048 1 0 48 cgca_gdim 0.0 0.045 1 5 5 214 sum_p 0.0 0.041 0.041 7 0 6 elap_time 0.0 0.029 0.029 1 0 29 cgca_pfem_sdalloc 0.0 0.025 0.025 1 0 25 cgca_cadim 0.0 0.022 36 3 6 12143 dismsh_ensi_p 0.0 0.019 0.019 5 0 4 sample 0.0 0.017 0.017 1 2 17 find_pe_procs 0.0 0.016 0.016 1 0 16 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.015 0.015 1 0 15 calc_neq_pp 0.0 0.014 0.014 12 0 1 MPI_Win_fence() 0.0 0.012 4 1 2 4249 read_loads 0.0 0.011 0.011 14 0 1 MPI_Type_free() 0.0 0.011 0.011 1 0 11 cgca_pfem_ctalloc 0.0 0.009 0.009 2 0 4 MPI_Test() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.007 0.007 9 0 1 MPI_Comm_rank() 0.0 0.007 3 1 3 3155 read_p121 0.0 0.006 5 1 2 5012 cgca_art 0.0 0.006 0.006 6 0 1 cgca_gcf 0.0 0.005 0.005 9 0 1 MPI_Comm_size() 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.003 0.003 6 0 0 cgca_agc 0.0 0.003 0.003 5 0 1 cgca_dgc 0.0 0.003 80 1 1 80940 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.002 0.278 1 1 278 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 28, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.187 2.665 3.951 0.8059 MPI-IO Write Bandwidth (MB/s) 7 5.187 2.665 3.951 0.8059 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.187E+04 6.107E+04 1.694E+04 Message size received from all nodes 1.229E+04 8.295E+04 4 6.105E+04 1.694E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 6144 7.21E+04 3.605E+04 5.792E+04 1.418E+04 Message size sent to node 27 3071 4.374E+04 4.374E+04 4.374E+04 0 Message size sent to node 27 : gather => MPI_Isend() 1 3.605E+04 3.605E+04 3.605E+04 0 Message size sent to node 27 : make_ggl => MPI_Isend() 3071 7.21E+04 7.21E+04 7.21E+04 0 Message size sent to node 27 : scatter => MPI_Isend() 1 7.21E+04 7.21E+04 7.21E+04 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 6144 8.295E+04 2.273E+04 6.42E+04 1.875E+04 Message size sent to node 29 3071 8.295E+04 8.295E+04 8.295E+04 0 Message size sent to node 29 : gather => MPI_Isend() 1 2.273E+04 2.273E+04 2.273E+04 0 Message size sent to node 29 : make_ggl => MPI_Isend() 3071 4.546E+04 4.546E+04 4.546E+04 0 Message size sent to node 29 : scatter => MPI_Isend() 1 4.546E+04 4.546E+04 4.546E+04 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 7.21E+04 7.21E+04 7.21E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 7.21E+04 7.21E+04 7.21E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 1 7.21E+04 7.21E+04 7.21E+04 0 [GROUP=MAX_MARKER] Message size sent to node 27 1 7.21E+04 7.21E+04 7.21E+04 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 2 8.295E+04 4.546E+04 6.421E+04 1.874E+04 [GROUP=MAX_MARKER] Message size sent to node 29 1 8.295E+04 8.295E+04 8.295E+04 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 1 4.546E+04 4.546E+04 4.546E+04 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.187E+04 2.187E+04 2.187E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.187E+04 2.187E+04 2.187E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 29;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 54.7 4:16.439 4:16.439 1.12021E+07 0 23 MPI_Win_unlock() 30.4 2:22.358 2:22.358 17279 0 8239 MPI_Barrier() 99.7 38,820 7:46.947 1 503146 466947670 xx14std 51.7 14,114 4:02.358 511 1.12372E+07 474283 cgca_hxi 50.0 4,107 3:54.250 5 269990 46850092 cgca_clvgp_nocosum 1.2 2,541 5,493 3071 36852 1789 gather 0.5 2,374 2,374 15338 0 155 MPI_Allreduce() 1.1 1,297 5,374 3071 33781 1750 scatter 0.2 976 976 12289 0 79 MPI_Recv() 0.2 961 961 1 0 961589 MPI_Init() 0.2 869 869 7 0 124273 MPI_File_write_all() 0.1 453 453 135 0 3361 MPI_Bcast() 0.1 448 448 1 0 448560 MPI_Finalize() 30.3 424 2:21.740 1 51201 141740521 cgca_sld 0.4 311 1,793 9198 9198 195 dot_product_p 0.1 255 255 12288 0 21 MPI_Isend() 0.1 241 264 100001 100001 3 cgca_clvgn [THROTTLED] 0.0 191 191 12288 0 16 MPI_Probe() 0.2 185 1,075 3066 6132 351 checon_par 100.0 162 7:48.552 1 456 468552735 .TAU application 0.7 131 3,451 5 20000 690239 cgca_pfem_simg 0.0 88 88 155 0 574 cgca_dacf 0.0 73 73 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 66 66 100001 0 1 beemat [THROTTLED] 0.0 56 56 7 0 8018 MPI_File_open() 0.0 50 50 100001 0 1 MPI_Get() [THROTTLED] 0.0 46 46 100001 0 0 deemat [THROTTLED] 1.1 45 5,023 12284 12284 409 my_barrier 0.0 43 43 2 0 21904 cgca_gbs 0.0 37 37 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 37 37 100001 0 0 shape_der [THROTTLED] 0.0 36 36 6 0 6027 MPI_Send() 0.0 33 33 6144 0 6 MPI_Waitall() 0.0 29 29 67 0 440 MPI_Reduce() 0.0 29 29 100001 0 0 invert [THROTTLED] 0.0 27 181 5 30000 36339 cgca_pfem_intcalc1 0.0 22 22 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22318 load 0.0 19 19 80000 0 0 determinant 0.0 15 15 13 0 1164 MPI_Win_create() 0.0 10 10 1 0 10109 MPI_Comm_dup() 0.0 9 9 14 0 652 MPI_Type_create_subarray() 0.0 8 8 1 0 8979 cgca_igb 0.0 8 8 12288 0 1 MPI_Get_count() 0.0 7 7 7 0 1040 MPI_File_close() 0.0 6 14 1 19 14215 make_ggl 0.0 5 5 10066 0 1 MPI_Put() 0.0 5 101 155 1711 658 cgca_gcupdn 0.0 5 5 2000 0 3 find_g3 0.0 5 5 1 17 5093 cgca_gcu 0.0 4 4 1 0 4957 allocate_gather_scatter 0.1 4 353 1 33 353982 read_g_coord_pp 0.0 4 48 1 160 48511 scatter_nodes 0.0 3 3 7 0 507 MPI_File_set_view() 0.0 2 2 12 0 248 MPI_Win_free() 0.0 2 2 5 0 418 cgca_fv 0.0 1 62 1 576 62068 cgca_pfem_cenc 0.4 0.944 1,732 7 525 247488 cgca_pswci 0.0 0.603 1 1 2 1440 cgca_as 0.0 0.339 0.339 1 0 339 getname 0.0 0.289 0.613 1 9 613 scatter_noadd 0.7 0.238 3,082 1 64 3082995 cgca_ds 0.0 0.21 0.21 1 0 210 rearrange 0.0 0.191 7 1 374 7925 cgca_nr 0.0 0.189 0.189 1 0 189 cgca_pfem_salloc 0.0 0.15 0.15 14 0 11 MPI_Type_commit() 0.0 0.135 0.135 1 0 135 calc_nels_pp 0.0 0.133 5 1 377 5902 cgca_rt 0.1 0.113 440 1 2 440598 read_g_num_pp 0.0 0.101 0.101 1 0 101 cgca_ins 0.0 0.096 0.096 5 0 19 cgca_pfem_uym 0.0 0.08 4 1 65 4550 gcupd_alloc 0.0 0.072 9 1 64 9285 cgca_drt 0.0 0.067 0.067 222 0 0 cgca_gcr 0.0 0.05 0.05 1 0 50 abaqus2sg 0.0 0.046 1 5 5 214 sum_p 0.0 0.035 0.035 7 0 5 elap_time 0.0 0.029 0.029 1 0 29 cgca_pfem_sdalloc 0.0 0.021 36 3 6 12061 dismsh_ensi_p 0.0 0.019 0.019 1 0 19 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.019 0.019 1 0 19 cgca_pfem_ctalloc 0.0 0.017 0.017 5 0 3 sample 0.0 0.015 0.015 2 0 8 MPI_Test() 0.0 0.015 0.015 12 0 1 MPI_Win_fence() 0.0 0.014 0.014 1 0 14 calc_neq_pp 0.0 0.014 0.016 1 2 16 find_pe_procs 0.0 0.012 0.012 14 0 1 MPI_Type_free() 0.0 0.009 0.009 1 0 9 cgca_gdim 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.008 0.008 1 0 8 cgca_cadim 0.0 0.007 0.007 9 0 1 cgca_agc 0.0 0.007 0.007 6 0 1 cgca_gcf 0.0 0.007 3 1 3 3199 read_p121 0.0 0.005 0.005 9 0 1 MPI_Comm_rank() 0.0 0.005 0.005 9 0 1 MPI_Comm_size() 0.0 0.005 4 1 2 4941 cgca_art 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 0.004 8 0 0 cgca_dgc 0.0 0.004 4 1 2 4125 read_loads 0.0 0.003 81 1 1 81018 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 calc_nodes_pp 0.0 0.002 0.27 1 1 270 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_pfem_ctdalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc 0.0 0 0 1 0 0 cgca_imco --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 29, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.184 2.661 3.95 0.8052 MPI-IO Write Bandwidth (MB/s) 7 5.184 2.661 3.95 0.8052 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.273E+04 6.568E+04 1.433E+04 Message size received from all nodes 1.229E+04 8.295E+04 4 6.566E+04 1.434E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 6144 8.295E+04 4.148E+04 6.421E+04 1.874E+04 Message size sent to node 28 3071 4.546E+04 4.546E+04 4.546E+04 0 Message size sent to node 28 : gather => MPI_Isend() 1 4.148E+04 4.148E+04 4.148E+04 0 Message size sent to node 28 : make_ggl => MPI_Isend() 3071 8.295E+04 8.295E+04 8.295E+04 0 Message size sent to node 28 : scatter => MPI_Isend() 1 8.295E+04 8.295E+04 8.295E+04 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 6144 7.445E+04 2.992E+04 6.714E+04 7315 Message size sent to node 30 3071 7.445E+04 7.445E+04 7.445E+04 0 Message size sent to node 30 : gather => MPI_Isend() 1 2.992E+04 2.992E+04 2.992E+04 0 Message size sent to node 30 : make_ggl => MPI_Isend() 3071 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 30 : scatter => MPI_Isend() 1 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 8.295E+04 8.295E+04 8.295E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 8.295E+04 8.295E+04 8.295E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 1 8.295E+04 8.295E+04 8.295E+04 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 1 8.295E+04 8.295E+04 8.295E+04 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 1 5.985E+04 5.985E+04 5.985E+04 0 [GROUP=MAX_MARKER] Message size sent to node 30 1 5.985E+04 5.985E+04 5.985E+04 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.273E+04 2.273E+04 2.273E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.273E+04 2.273E+04 2.273E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 30;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 55.1 4:18.335 4:18.335 1.12021E+07 0 23 MPI_Win_unlock() 29.8 2:19.754 2:19.754 17279 0 8088 MPI_Barrier() 99.7 38,906 7:46.947 1 503146 466947665 xx14std 53.5 14,717 4:10.536 511 1.12818E+07 490288 cgca_hxi 50.0 2,752 3:54.249 5 269768 46849938 cgca_clvgp_nocosum 1.2 2,545 5,484 3071 36852 1786 gather 0.5 2,394 2,394 15338 0 156 MPI_Allreduce() 0.3 1,450 1,450 145063 0 10 MPI_Win_lock() [THROTTLED] 1.1 1,341 5,273 3071 33781 1717 scatter 0.2 973 973 12289 0 79 MPI_Recv() 0.2 962 962 1 0 962447 MPI_Init() 0.2 870 870 7 0 124404 MPI_File_write_all() 0.1 472 472 1 0 472161 MPI_Finalize() 30.3 458 2:21.750 1 51699 141750394 cgca_sld 0.1 447 447 135 0 3312 MPI_Bcast() 0.4 312 1,807 9198 9198 197 dot_product_p 0.1 267 267 12288 0 22 MPI_Isend() 0.1 223 253 100001 100001 3 cgca_clvgn [THROTTLED] 0.2 182 1,079 3066 6132 352 checon_par 100.0 161 7:48.576 1 463 468576237 .TAU application 0.0 143 143 12288 0 12 MPI_Probe() 0.9 102 4,080 5 20000 816000 cgca_pfem_simg 0.0 89 89 155 0 579 cgca_dacf 0.0 65 65 100001 0 1 beemat [THROTTLED] 0.0 55 55 7 0 7970 MPI_File_open() 0.0 52 52 100001 0 1 MPI_Get() [THROTTLED] 0.0 44 44 100001 0 0 deemat [THROTTLED] 0.0 43 43 2 0 21938 cgca_gbs 1.1 39 4,927 12284 12284 401 my_barrier 0.0 38 38 100001 0 0 shape_der [THROTTLED] 0.0 36 36 100001 0 0 cgca_clvgsd [THROTTLED] 0.0 35 35 6 0 5986 MPI_Send() 0.0 34 34 100001 0 0 invert [THROTTLED] 0.0 30 30 6144 0 5 MPI_Waitall() 0.0 30 30 100001 0 0 cgca_csym [THROTTLED] 0.0 29 29 67 0 441 MPI_Reduce() 0.0 21 21 80000 0 0 determinant 0.0 21 197 5 30000 39512 cgca_pfem_intcalc1 0.0 21 22 1 1 22357 load 0.0 11 11 13 0 847 MPI_Win_create() 0.0 10 10 1 0 10158 MPI_Comm_dup() 0.0 9 9 1 0 9105 cgca_igb 0.0 9 9 14 0 649 MPI_Type_create_subarray() 0.0 7 7 12288 0 1 MPI_Get_count() 0.0 7 7 7 0 1003 MPI_File_close() 0.0 6 14 1 19 14113 make_ggl 0.0 5 5 1 17 5227 cgca_gcu 0.0 5 5 2000 0 3 find_g3 0.0 5 5 1 0 5174 allocate_gather_scatter 0.0 5 88 155 1710 570 cgca_gcupdn 0.0 5 5 10069 0 1 MPI_Put() 0.1 4 347 1 33 347005 read_g_coord_pp 0.0 4 48 1 160 48514 scatter_nodes 0.0 3 3 7 0 452 MPI_File_set_view() 0.0 2 2 12 0 230 MPI_Win_free() 0.0 2 2 5 0 412 cgca_fv 0.0 1 66 1 576 66127 cgca_pfem_cenc 0.4 0.797 1,732 7 525 247536 cgca_pswci 0.0 0.601 1 1 2 1484 cgca_as 0.0 0.341 0.341 1 0 341 getname 0.0 0.312 0.696 1 9 696 scatter_noadd 0.7 0.237 3,083 1 64 3083272 cgca_ds 0.0 0.203 0.203 1 0 203 rearrange 0.0 0.202 0.202 1 0 202 calc_nels_pp 0.0 0.172 8 1 374 8138 cgca_nr 0.0 0.163 0.163 14 0 12 MPI_Type_commit() 0.0 0.159 0.159 1 0 159 cgca_pfem_salloc 0.0 0.136 0.136 5 0 27 cgca_pfem_uym 0.0 0.125 5 1 377 5641 cgca_rt 0.1 0.122 447 1 2 447566 read_g_num_pp 0.0 0.091 4 1 65 4110 gcupd_alloc 0.0 0.086 0.086 1 0 86 cgca_ins 0.0 0.082 9 1 64 9269 cgca_drt 0.0 0.052 0.974 5 5 195 sum_p 0.0 0.049 0.049 1 0 49 abaqus2sg 0.0 0.034 0.034 7 0 5 elap_time 0.0 0.032 0.032 1 0 32 cgca_pfem_sdalloc 0.0 0.029 0.029 1 0 29 calc_neq_pp 0.0 0.024 35 3 6 11980 dismsh_ensi_p 0.0 0.019 0.019 2 0 10 MPI_Test() 0.0 0.019 0.019 1 2 19 find_pe_procs 0.0 0.017 0.017 5 0 3 sample 0.0 0.015 0.015 1 0 15 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.014 0.014 12 0 1 MPI_Win_fence() 0.0 0.013 0.013 1 0 13 cgca_pfem_ctalloc 0.0 0.009 0.009 14 0 1 MPI_Type_free() 0.0 0.009 0.009 1 0 9 cgca_cadim 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.008 0.008 1 0 8 cgca_gdim 0.0 0.006 0.006 9 0 1 MPI_Comm_size() 0.0 0.006 0.006 9 0 1 cgca_agc 0.0 0.006 0.772 1 2 772 cgca_art 0.0 0.006 3 1 3 3217 read_p121 0.0 0.005 0.005 9 0 1 MPI_Comm_rank() 0.0 0.005 0.005 8 0 1 cgca_dgc 0.0 0.005 0.005 5 0 1 cgca_gcf 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 81 1 1 81105 read_rest 0.0 0.003 4 1 2 4300 read_loads 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.185 1 1 185 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc 0.0 0 0 1 0 0 cgca_imco --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 30, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.181 2.658 3.946 0.8057 MPI-IO Write Bandwidth (MB/s) 7 5.181 2.658 3.946 0.8057 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 1.229E+04 1.6E+05 2.992E+04 6.544E+04 1.006E+04 Message size received from all nodes 1.229E+04 7.562E+04 4 6.541E+04 1.008E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 6144 7.445E+04 3.722E+04 6.714E+04 7309 Message size sent to node 29 3071 5.985E+04 5.985E+04 5.985E+04 0 Message size sent to node 29 : gather => MPI_Isend() 1 3.722E+04 3.722E+04 3.722E+04 0 Message size sent to node 29 : make_ggl => MPI_Isend() 3071 7.445E+04 7.445E+04 7.445E+04 0 Message size sent to node 29 : scatter => MPI_Isend() 1 7.445E+04 7.445E+04 7.445E+04 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 0 0 0 0 0 Message size sent to node 30 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 6144 7.562E+04 2.591E+04 6.372E+04 1.191E+04 Message size sent to node 31 3071 7.562E+04 7.562E+04 7.562E+04 0 Message size sent to node 31 : gather => MPI_Isend() 1 2.591E+04 2.591E+04 2.591E+04 0 Message size sent to node 31 : make_ggl => MPI_Isend() 3071 5.182E+04 5.182E+04 5.182E+04 0 Message size sent to node 31 : scatter => MPI_Isend() 1 5.182E+04 5.182E+04 5.182E+04 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 7.445E+04 7.445E+04 7.445E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 7.445E+04 7.445E+04 7.445E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 1 7.445E+04 7.445E+04 7.445E+04 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 1 7.445E+04 7.445E+04 7.445E+04 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 1 5.182E+04 5.182E+04 5.182E+04 0 [GROUP=MAX_MARKER] Message size sent to node 31 1 5.182E+04 5.182E+04 5.182E+04 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 3.781E+04 3.781E+04 3.781E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 3.781E+04 3.781E+04 3.781E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- NODE 31;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 55.9 4:21.756 4:21.756 8.01284E+06 0 33 MPI_Win_unlock() 30.7 2:23.757 2:23.757 17279 0 8320 MPI_Barrier() 99.7 38,875 7:46.949 1 503146 466949760 xx14std 48.5 11,069 3:47.101 511 8.04756E+06 444426 cgca_hxi 0.5 2,421 2,421 15338 0 158 MPI_Allreduce() 1.2 2,219 5,418 3071 24568 1764 gather 1.1 1,290 5,345 3071 21497 1740 scatter 0.2 967 967 1 0 967460 MPI_Init() 0.2 870 870 7 0 124427 MPI_File_write_all() 0.2 826 826 6145 0 135 MPI_Recv() 50.0 759 3:54.250 5 169767 46850052 cgca_clvgp_nocosum 30.3 510 2:21.750 1 52197 141750393 cgca_sld 0.1 454 454 1 0 454692 MPI_Finalize() 0.1 441 441 135 0 3270 MPI_Bcast() 0.4 307 1,831 9198 9198 199 dot_product_p 0.1 233 257 100001 100001 3 cgca_clvgn [THROTTLED] 0.0 220 220 6144 0 36 MPI_Probe() 0.2 183 1,078 3066 6132 352 checon_par 100.0 156 7:48.558 1 489 468558863 .TAU application 0.0 153 153 6144 0 25 MPI_Isend() 0.8 112 3,961 5 20000 792287 cgca_pfem_simg 0.0 108 108 6144 0 18 MPI_Waitall() 0.0 107 107 100001 0 1 MPI_Win_lock() [THROTTLED] 0.0 92 92 155 0 595 cgca_dacf 0.0 66 66 100001 0 1 beemat [THROTTLED] 0.0 56 56 100001 0 1 MPI_Get() [THROTTLED] 0.0 55 55 7 0 7975 MPI_File_open() 0.0 50 50 100001 0 1 deemat [THROTTLED] 1.1 44 5,338 12284 12284 435 my_barrier 0.0 43 43 2 0 21946 cgca_gbs 0.0 42 42 100001 0 0 shape_der [THROTTLED] 0.0 34 34 6 0 5764 MPI_Send() 0.0 29 29 67 0 442 MPI_Reduce() 0.0 28 28 100001 0 0 invert [THROTTLED] 0.0 27 27 80000 0 0 determinant 0.0 24 228 5 30000 45777 cgca_pfem_intcalc1 0.0 23 23 100001 0 0 cgca_csym [THROTTLED] 0.0 21 22 1 1 22257 load 0.0 13 13 13 0 1015 MPI_Win_create() 0.0 10 10 1 0 10034 MPI_Comm_dup() 0.0 9 9 1 0 9309 cgca_igb 0.0 9 9 6144 0 2 MPI_Get_count() 0.0 9 9 14 0 652 MPI_Type_create_subarray() 0.0 6 14 1 14 14124 make_ggl 0.0 6 6 7 0 886 MPI_File_close() 0.0 5 5 10078 0 1 MPI_Put() 0.0 5 5 2000 0 3 find_g3 0.0 4 4 1 0 4863 allocate_gather_scatter 0.0 4 4 1 5 4811 cgca_gcu 0.0 4 48 1 160 48420 scatter_nodes 0.1 4 340 1 33 340039 read_g_coord_pp 0.0 3 67 155 1085 438 cgca_gcupdn 0.0 3 3 7 0 483 MPI_File_set_view() 0.0 2 2 12 0 238 MPI_Win_free() 0.0 2 2 5 0 418 cgca_fv 0.0 1 62 1 576 62064 cgca_pfem_cenc 0.4 0.758 1,732 7 525 247450 cgca_pswci 0.0 0.673 0.673 1 0 673 getname 0.0 0.606 1 1 2 1423 cgca_as 0.0 0.35 0.511 1 5 511 scatter_noadd 0.0 0.215 0.215 1 0 215 rearrange 0.0 0.185 8 1 374 8202 cgca_nr 0.0 0.162 0.162 1 0 162 cgca_pfem_salloc 0.7 0.153 3,081 1 64 3081531 cgca_ds 0.0 0.143 0.143 14 0 10 MPI_Type_commit() 0.0 0.136 0.136 1 0 136 calc_nels_pp 0.0 0.132 5 1 377 5623 cgca_rt 0.1 0.119 454 1 2 454711 read_g_num_pp 0.0 0.085 0.085 1 0 85 cgca_ins 0.0 0.084 9 1 64 9273 cgca_drt 0.0 0.082 0.082 5 0 16 cgca_pfem_uym 0.0 0.082 4 1 65 4460 gcupd_alloc 0.0 0.049 0.049 1 0 49 abaqus2sg 0.0 0.045 1 5 5 211 sum_p 0.0 0.038 0.038 7 0 5 elap_time 0.0 0.034 0.034 1 0 34 calc_neq_pp 0.0 0.031 0.031 1 0 31 cgca_pfem_sdalloc 0.0 0.025 0.025 14 0 2 MPI_Type_free() 0.0 0.023 0.023 1 0 23 cgca_pfem_ctalloc 0.0 0.023 34 3 6 11537 dismsh_ensi_p 0.0 0.022 0.023 1 2 23 find_pe_procs 0.0 0.017 0.017 5 0 3 sample 0.0 0.016 0.016 1 0 16 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.016 0.016 12 0 1 MPI_Win_fence() 0.0 0.009 5 1 3 5023 read_p121 0.0 0.008 0.008 9 0 1 MPI_Comm_rank() 0.0 0.008 0.008 1 0 8 MPI_Test() 0.0 0.008 0.008 1 0 8 caflaunch$MAIN$blk 0.0 0.007 0.007 1 0 7 cgca_cadim 0.0 0.007 0.007 1 0 7 cgca_gdim 0.0 0.005 0.005 9 0 1 MPI_Comm_size() 0.0 0.005 4 1 2 4872 cgca_art 0.0 0.005 4 1 2 4349 read_loads 0.0 0.004 0.004 1 0 4 MPI_Alloc_mem() 0.0 0.004 0.004 3 0 1 cgca_agc 0.0 0.003 80 1 1 80921 read_rest 0.0 0.002 0.002 1 0 2 MPI_Finalized() 0.0 0.002 0.002 2 0 1 cgca_dgc 0.0 0.002 0.002 1 0 2 cgca_pfem_ctdalloc 0.0 0.002 0.002 1 0 2 cgca_pfem_integalloc 0.0 0.002 0.412 1 1 412 max_p 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.001 0.001 1 0 1 calc_nodes_pp 0.0 0.001 0.001 1 0 1 calc_npes_pp 0.0 0.001 0.001 1 0 1 cgca_imco 0.0 0.001 0.001 1 0 1 cgca_pfem_ealloc 0.0 0.001 0.001 1 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 cgca_pfem_integdalloc --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 31, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written 7 4.685E+05 4.685E+05 4.685E+05 0 MPI-IO Bytes Written : cgca_pswci => MPI_File_write_all() 7 5.183 2.661 3.945 0.8045 MPI-IO Write Bandwidth (MB/s) 7 5.183 2.661 3.945 0.8045 MPI-IO Write Bandwidth (MB/s) : cgca_pswci => MPI_File_write_all() 1.534E+04 128 4 8.015 1.371 Message size for all-reduce 135 3.866E+05 4 5.103E+04 9.093E+04 Message size for broadcast 67 2.03E+05 4 1.454E+05 5.874E+04 Message size for reduce 6145 1.6E+05 2.591E+04 6.373E+04 1.197E+04 Message size received from all nodes 6150 7.562E+04 4 6.369E+04 1.198E+04 Message size sent to all nodes 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 6 6.766E+04 4 3.383E+04 3.383E+04 Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 0 0 0 0 0 Message size sent to node 10 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 0 0 0 0 0 Message size sent to node 11 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 0 0 0 0 0 Message size sent to node 12 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 0 0 0 0 0 Message size sent to node 13 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 0 0 0 0 0 Message size sent to node 14 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 0 0 0 0 0 Message size sent to node 15 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 0 0 0 0 0 Message size sent to node 16 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 0 0 0 0 0 Message size sent to node 17 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 0 0 0 0 0 Message size sent to node 18 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 0 0 0 0 0 Message size sent to node 19 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 0 0 0 0 0 Message size sent to node 20 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 0 0 0 0 0 Message size sent to node 21 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 0 0 0 0 0 Message size sent to node 22 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 0 0 0 0 0 Message size sent to node 23 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 0 0 0 0 0 Message size sent to node 24 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 0 0 0 0 0 Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 0 0 0 0 0 Message size sent to node 26 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 0 0 0 0 0 Message size sent to node 27 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 0 0 0 0 0 Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 0 0 0 0 0 Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 : scatter_noadd => MPI_Isend() 6144 7.562E+04 3.781E+04 6.372E+04 1.19E+04 Message size sent to node 30 3071 5.182E+04 5.182E+04 5.182E+04 0 Message size sent to node 30 : gather => MPI_Isend() 1 3.781E+04 3.781E+04 3.781E+04 0 Message size sent to node 30 : make_ggl => MPI_Isend() 3071 7.562E+04 7.562E+04 7.562E+04 0 Message size sent to node 30 : scatter => MPI_Isend() 1 7.562E+04 7.562E+04 7.562E+04 0 Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 0 0 0 0 0 Message size sent to node 31 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : gather => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : make_ggl => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 : scatter_noadd => MPI_Isend() 2 3.866E+05 2.03E+05 2.948E+05 9.18E+04 [GROUP=MAX_MARKER] Message size for broadcast 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_g_coord_pp => MPI_Bcast() 1 3.866E+05 3.866E+05 3.866E+05 0 [GROUP=MAX_MARKER] Message size for broadcast : read_rest => MPI_Bcast() 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce 1 2.03E+05 2.03E+05 2.03E+05 0 [GROUP=MAX_MARKER] Message size for reduce : scatter_nodes => MPI_Reduce() 1 7.562E+04 7.562E+04 7.562E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to all nodes : make_ggl => MPI_Isend() 1 7.562E+04 7.562E+04 7.562E+04 0 [GROUP=MAX_MARKER] Message size sent to all nodes : scatter_noadd => MPI_Isend() 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 1 6.766E+04 6.766E+04 6.766E+04 0 [GROUP=MAX_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 0 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 1 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 10 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 11 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 12 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 13 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 14 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 15 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 16 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 17 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 18 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 19 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 2 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 20 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 21 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 22 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 23 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 24 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 25 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 26 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 27 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 28 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 29 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 3 : scatter_noadd => MPI_Isend() 1 7.562E+04 7.562E+04 7.562E+04 0 [GROUP=MAX_MARKER] Message size sent to node 30 1 7.562E+04 7.562E+04 7.562E+04 0 [GROUP=MAX_MARKER] Message size sent to node 30 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 31 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 4 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 5 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : gather => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 6 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 7 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 8 : scatter_noadd => MPI_Isend() 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 0 0 0 0 0 [GROUP=MAX_MARKER] Message size sent to node 9 : scatter_noadd => MPI_Isend() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for all-reduce : max_p => MPI_Allreduce() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for broadcast : scatter_nodes => MPI_Bcast() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce 1 4 4 4 0 [GROUP=MIN_MARKER] Message size for reduce : make_ggl => MPI_Reduce() 1 2.591E+04 2.591E+04 2.591E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes 1 2.591E+04 2.591E+04 2.591E+04 0 [GROUP=MIN_MARKER] Message size received from all nodes : make_ggl => MPI_Recv() 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes 1 4 4 4 0 [GROUP=MIN_MARKER] Message size sent to all nodes : dismsh_ensi_p => MPI_Send() 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 0 0 0 0 0 [GROUP=MIN_MARKER] Message size sent to node 0 : dismsh_ensi_p => MPI_Send() --------------------------------------------------------------------------------------- FUNCTION SUMMARY (total): --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 59.1 2:27:37.188 2:27:37.188 3.51095E+08 0 25 MPI_Win_unlock() 22.5 56:16.147 56:16.147 535649 0 6303 MPI_Barrier() 96.5 20:11.200 4:01:15.412 31 1.55975E+07 466948776 xx14std 54.9 8:13.038 2:17:08.315 15841 3.52318E+08 519432 cgca_hxi 3.1 7:50.146 7:50.146 32 0 14692077 caflaunch$MAIN$blk 48.4 2:25.825 2:01:01.752 155 8.54893E+06 46850015 cgca_clvgp_nocosum 1.1 1:20.845 2:48.721 95201 1.13013E+06 1772 gather 0.5 1:14.776 1:14.776 475478 0 157 MPI_Allreduce() 1.1 41,820 2:39.092 95201 1.03493E+06 1671 scatter 0.3 39,207 39,207 31 0 1264749 MPI_Init() 0.2 34,888 34,888 186 0 187575 MPI_Send() 0.2 26,962 26,962 217 0 124253 MPI_File_write_all() 0.2 24,628 24,628 374815 0 66 MPI_Recv() 0.1 16,859 16,859 4185 0 4029 MPI_Bcast() 29.3 14,110 1:13:14.181 31 1.59221E+06 141747803 cgca_sld 0.4 9,695 56,329 285138 285138 198 dot_product_p 0.1 9,560 9,560 374784 0 26 MPI_Isend() 0.1 9,548 9,548 3.24385E+06 0 3 MPI_Win_lock() [THROTTLED] 0.1 8,439 8,439 31 0 272246 MPI_Finalize() 0.1 7,579 7,579 374784 0 20 MPI_Probe() 0.1 6,977 7,699 3.00003E+06 3.00003E+06 3 cgca_clvgn [THROTTLED] 0.2 5,601 33,640 95046 190092 354 checon_par 100.0 5,188 4:09:59.346 32 16070 468729564 .TAU application 3.3 3,203 8:15.619 155 620000 3197545 cgca_pfem_simg 0.0 2,983 2,983 4805 0 621 cgca_dacf 0.0 2,141 2,141 3.10003E+06 0 1 beemat [THROTTLED] 0.0 1,814 1,814 3.10003E+06 0 1 MPI_Get() [THROTTLED] 0.0 1,656 1,656 217 0 7635 MPI_File_open() 0.0 1,465 1,465 3.10003E+06 0 0 deemat [THROTTLED] 0.0 1,428 1,428 190464 0 8 MPI_Waitall() 0.0 1,363 1,363 62 0 21999 cgca_gbs 0.1 1,274 21,635 155 930000 139587 cgca_pfem_intcalc1 1.0 1,271 2:23.434 380804 380804 377 my_barrier 0.0 1,191 1,191 3.10003E+06 0 0 shape_der [THROTTLED] 0.0 1,094 1,094 2.90003E+06 0 0 cgca_clvgsd [THROTTLED] 0.0 1,054 1,054 3.10003E+06 0 0 invert [THROTTLED] 0.0 867 867 2077 0 418 MPI_Reduce() 0.0 745 745 3.10003E+06 0 0 cgca_csym [THROTTLED] 0.0 682 682 2.48E+06 0 0 determinant 0.0 674 674 403 0 1674 MPI_Win_create() 0.0 673 693 31 31 22382 load 0.0 311 311 312629 0 1 MPI_Put() 0.0 282 282 31 0 9119 cgca_igb 0.0 262 262 217 0 1211 MPI_File_close() 0.0 260 260 374784 0 1 MPI_Get_count() 0.0 233 256 98270 100001 3 cgca_clvgn 0.0 216 4,163 4805 56340 866 cgca_gcupdn 0.0 199 437 31 584 14110 make_ggl 0.0 191 192 31 1077 6206 cgca_gcu 0.0 180 180 217 0 833 MPI_File_set_view() 0.0 169 169 31 0 5458 MPI_Comm_dup() 0.0 161 161 62000 0 3 find_g3 0.0 146 146 434 0 338 MPI_Type_create_subarray() 0.0 145 145 31 0 4683 allocate_gather_scatter 0.0 142 1,482 31 4960 47833 scatter_nodes 0.1 131 13,777 31 1023 444422 read_g_coord_pp 0.0 97 97 372 0 261 MPI_Win_free() 0.0 76 76 300003 0 0 cgca_gcr [THROTTLED] 0.0 70 70 155 0 456 cgca_fv 0.0 64 1,825 31 17856 58895 cgca_pfem_cenc 0.4 29 53,730 217 16275 247604 cgca_pswci 0.0 21 21 87825 0 0 cgca_gcr 0.0 19 49 31 62 1604 cgca_as 0.0 18 18 31 0 582 getname 0.0 9 20 31 275 645 scatter_noadd 0.4 8 1:01.830 31 1984 1994538 cgca_ds 0.0 6 6 31 0 212 rearrange 0.0 6 244 31 11594 7894 cgca_nr 0.1 6 10,829 31 62 349339 read_g_num_pp 0.0 6 6 31 0 194 cgca_pfem_salloc 0.0 5 5 31 0 171 calc_nels_pp 0.0 5 5 434 0 12 MPI_Type_commit() 0.0 4 183 31 11687 5915 cgca_rt 0.0 4 4 31 0 135 cgca_ins 0.0 2 2 155 0 19 cgca_pfem_uym 0.0 2 286 31 1984 9246 cgca_drt 0.0 2 136 31 2015 4398 gcupd_alloc 0.0 1 36 155 155 236 sum_p 0.0 1 1 31 0 44 abaqus2sg 0.0 1 1 217 0 6 elap_time 0.0 1 1 31 0 38 cgca_pfem_sdalloc 0.0 1 1 31 0 38 cgca_gdim 0.2 1 34,889 93 186 375161 dismsh_ensi_p 0.0 1 1 31 0 33 cgca_cadim 0.0 0.738 0.738 155 0 5 sample 0.0 0.609 0.609 31 0 20 calc_neq_pp 0.0 0.591 0.591 408 0 1 cgca_gcf 0.0 0.536 0.536 31 0 17 cgca_pfem_ctalloc 0.0 0.526 0.555 31 62 18 find_pe_procs 0.0 0.522 0.522 32 0 16 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.5 0.5 372 0 1 MPI_Win_fence() 0.0 0.492 0.492 61 0 8 MPI_Test() 0.0 0.446 0.446 434 0 1 MPI_Type_free() 0.0 0.323 0.323 554 0 1 cgca_agc 0.0 0.264 0.264 279 0 1 MPI_Comm_size() 0.0 0.261 0.261 279 0 1 MPI_Comm_rank() 0.0 0.259 0.259 523 0 0 cgca_dgc 0.0 0.23 113 31 93 3664 read_p121 0.0 0.197 248 31 62 8019 cgca_art 0.0 0.154 120 31 62 3902 read_loads 0.0 0.14 0.14 31 0 5 MPI_Alloc_mem() 0.0 0.111 0.111 31 0 4 cgca_pfem_integalloc 0.0 0.101 2,485 31 31 80166 read_rest 0.0 0.079 8 31 31 262 max_p 0.0 0.069 0.069 31 0 2 MPI_Finalized() 0.0 0.057 0.057 31 0 2 cgca_pfem_ctdalloc 0.0 0.047 0.047 31 0 2 calc_nodes_pp 0.0 0.041 0.041 31 0 1 cgca_pfem_ealloc 0.0 0.038 0.038 32 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.034 0.034 31 0 1 cgca_pfem_edalloc 0.0 0.032 0.032 32 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.032 0.032 32 0 1 CI0000$cgca_m3pfem_ 0.0 0.032 0.032 31 0 1 calc_npes_pp 0.0 0.031 0.031 31 0 1 cgca_pfem_integdalloc 0.0 0.022 0.022 31 0 1 cgca_imco FUNCTION SUMMARY (mean): --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 59.1 4:36.787 4:36.787 1.09717E+07 0 25 MPI_Win_unlock() 22.5 1:45.504 1:45.504 16739 0 6303 MPI_Barrier() 96.5 37,850 7:32.356 0.96875 487423 466948776 xx14std 54.9 15,407 4:17.134 495.031 1.101E+07 519432 cgca_hxi 3.1 14,692 14,692 1 0 14692077 caflaunch$MAIN$blk 48.4 4,557 3:46.929 4.84375 267154 46850015 cgca_clvgp_nocosum 1.1 2,526 5,272 2975.03 35316.5 1772 gather 0.5 2,336 2,336 14858.7 0 157 MPI_Allreduce() 1.1 1,306 4,971 2975.03 32341.5 1671 scatter 0.3 1,225 1,225 0.96875 0 1264749 MPI_Init() 0.2 1,090 1,090 5.8125 0 187575 MPI_Send() 0.2 842 842 6.78125 0 124253 MPI_File_write_all() 0.2 769 769 11713 0 66 MPI_Recv() 0.1 526 526 130.781 0 4029 MPI_Bcast() 29.3 440 2:17.318 0.96875 49756.6 141747803 cgca_sld 0.4 302 1,760 8910.56 8910.56 198 dot_product_p 0.1 298 298 11712 0 26 MPI_Isend() 0.1 298 298 101370 0 3 MPI_Win_lock() [THROTTLED] 0.1 263 263 0.96875 0 272246 MPI_Finalize() 0.1 236 236 11712 0 20 MPI_Probe() 0.1 218 240 93750.9 93750.9 3 cgca_clvgn [THROTTLED] 0.2 175 1,051 2970.19 5940.38 354 checon_par 100.0 162 7:48.729 1 502.188 468729564 .TAU application 3.3 100 15,488 4.84375 19375 3197545 cgca_pfem_simg 0.0 93 93 150.156 0 621 cgca_dacf 0.0 66 66 96876 0 1 beemat [THROTTLED] 0.0 56 56 96876 0 1 MPI_Get() [THROTTLED] 0.0 51 51 6.78125 0 7635 MPI_File_open() 0.0 45 45 96876 0 0 deemat [THROTTLED] 0.0 44 44 5952 0 8 MPI_Waitall() 0.0 42 42 1.9375 0 21999 cgca_gbs 0.1 39 676 4.84375 29062.5 139587 cgca_pfem_intcalc1 1.0 39 4,482 11900.1 11900.1 377 my_barrier 0.0 37 37 96876 0 0 shape_der [THROTTLED] 0.0 34 34 90625.9 0 0 cgca_clvgsd [THROTTLED] 0.0 32 32 96876 0 0 invert [THROTTLED] 0.0 27 27 64.9062 0 418 MPI_Reduce() 0.0 23 23 96876 0 0 cgca_csym [THROTTLED] 0.0 21 21 77500 0 0 determinant 0.0 21 21 12.5938 0 1674 MPI_Win_create() 0.0 21 21 0.96875 0.96875 22382 load 0.0 9 9 9769.66 0 1 MPI_Put() 0.0 8 8 0.96875 0 9119 cgca_igb 0.0 8 8 6.78125 0 1211 MPI_File_close() 0.0 8 8 11712 0 1 MPI_Get_count() 0.0 7 8 3070.94 3125.03 3 cgca_clvgn 0.0 6 130 150.156 1760.62 866 cgca_gcupdn 0.0 6 13 0.96875 18.25 14110 make_ggl 0.0 5 6 0.96875 33.6562 6206 cgca_gcu 0.0 5 5 6.78125 0 833 MPI_File_set_view() 0.0 5 5 0.96875 0 5458 MPI_Comm_dup() 0.0 5 5 1937.5 0 3 find_g3 0.0 4 4 13.5625 0 338 MPI_Type_create_subarray() 0.0 4 4 0.96875 0 4683 allocate_gather_scatter 0.0 4 46 0.96875 155 47833 scatter_nodes 0.1 4 430 0.96875 31.9688 444422 read_g_coord_pp 0.0 3 3 11.625 0 261 MPI_Win_free() 0.0 2 2 9375.09 0 0 cgca_gcr [THROTTLED] 0.0 2 2 4.84375 0 456 cgca_fv 0.0 2 57 0.96875 558 58895 cgca_pfem_cenc 0.4 0.91 1,679 6.78125 508.594 247604 cgca_pswci 0.0 0.664 0.664 2744.53 0 0 cgca_gcr 0.0 0.606 1 0.96875 1.9375 1604 cgca_as 0.0 0.564 0.564 0.96875 0 582 getname 0.0 0.306 0.625 0.96875 8.59375 645 scatter_noadd 0.4 0.257 1,932 0.96875 62 1994538 cgca_ds 0.0 0.206 0.206 0.96875 0 212 rearrange 0.0 0.196 7 0.96875 362.312 7894 cgca_nr 0.1 0.195 338 0.96875 1.9375 349339 read_g_num_pp 0.0 0.188 0.188 0.96875 0 194 cgca_pfem_salloc 0.0 0.166 0.166 0.96875 0 171 calc_nels_pp 0.0 0.158 0.158 13.5625 0 12 MPI_Type_commit() 0.0 0.148 5 0.96875 365.219 5915 cgca_rt 0.0 0.131 0.131 0.96875 0 135 cgca_ins 0.0 0.0927 0.0927 4.84375 0 19 cgca_pfem_uym 0.0 0.0927 8 0.96875 62 9246 cgca_drt 0.0 0.092 4 0.96875 62.9688 4398 gcupd_alloc 0.0 0.0498 1 4.84375 4.84375 236 sum_p 0.0 0.043 0.043 0.96875 0 44 abaqus2sg 0.0 0.0387 0.0387 6.78125 0 6 elap_time 0.0 0.0368 0.0368 0.96875 0 38 cgca_pfem_sdalloc 0.0 0.0367 0.0367 0.96875 0 38 cgca_gdim 0.2 0.0322 1,090 2.90625 5.8125 375161 dismsh_ensi_p 0.0 0.0318 0.0318 0.96875 0 33 cgca_cadim 0.0 0.0231 0.0231 4.84375 0 5 sample 0.0 0.019 0.019 0.96875 0 20 calc_neq_pp 0.0 0.0185 0.0185 12.75 0 1 cgca_gcf 0.0 0.0168 0.0168 0.96875 0 17 cgca_pfem_ctalloc 0.0 0.0164 0.0173 0.96875 1.9375 18 find_pe_procs 0.0 0.0163 0.0163 1 0 16 CI0001$cgca_m3sld_mp_cgca_sld2_$BLK 0.0 0.0156 0.0156 11.625 0 1 MPI_Win_fence() 0.0 0.0154 0.0154 1.90625 0 8 MPI_Test() 0.0 0.0139 0.0139 13.5625 0 1 MPI_Type_free() 0.0 0.0101 0.0101 17.3125 0 1 cgca_agc 0.0 0.00825 0.00825 8.71875 0 1 MPI_Comm_size() 0.0 0.00816 0.00816 8.71875 0 1 MPI_Comm_rank() 0.0 0.00809 0.00809 16.3438 0 0 cgca_dgc 0.0 0.00719 3 0.96875 2.90625 3664 read_p121 0.0 0.00616 7 0.96875 1.9375 8019 cgca_art 0.0 0.00481 3 0.96875 1.9375 3902 read_loads 0.0 0.00438 0.00438 0.96875 0 5 MPI_Alloc_mem() 0.0 0.00347 0.00347 0.96875 0 4 cgca_pfem_integalloc 0.0 0.00316 77 0.96875 0.96875 80166 read_rest 0.0 0.00247 0.253 0.96875 0.96875 262 max_p 0.0 0.00216 0.00216 0.96875 0 2 MPI_Finalized() 0.0 0.00178 0.00178 0.96875 0 2 cgca_pfem_ctdalloc 0.0 0.00147 0.00147 0.96875 0 2 calc_nodes_pp 0.0 0.00128 0.00128 0.96875 0 1 cgca_pfem_ealloc 0.0 0.00119 0.00119 1 0 1 CI0000$cgca_m3sld_mp_cgca_sld_$BLK 0.0 0.00106 0.00106 0.96875 0 1 cgca_pfem_edalloc 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3clvg_mp_cgca_clvgp_nocosum_$BLK 0.0 0.001 0.001 1 0 1 CI0000$cgca_m3pfem_ 0.0 0.001 0.001 0.96875 0 1 calc_npes_pp 0.0 0.000969 0.000969 0.96875 0 1 cgca_pfem_integdalloc 0.0 0.000687 0.000687 0.96875 0 1 cgca_imco