Reading Profile files in profile.* NODE 0;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 58,737 58,737 1 0 58737030 .TAU application => caflaunch$MAIN$blk 100.0 58,737 58,737 1 0 58737030 caflaunch$MAIN$blk 100.0 3 58,740 1 1 58740842 .TAU application NODE 1;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 49.5 28,914 28,914 2.59846E+06 0 11 MPI_Win_unlock() 44.9 26,251 26,251 1.80009E+06 0 15 .TAU application => coback1tau => MPI_Win_unlock() 82.6 13,118 48,247 1 2.56008E+06 48247865 .TAU application => coback1tau 82.6 13,118 48,247 1 2.56008E+06 48247865 coback1tau 13.2 7,706 7,706 360000 0 21 .TAU application => coback1tau => MPI_Recv() 13.2 7,706 7,706 360000 0 21 MPI_Recv() 100.0 7,343 58,442 1 798468 58442218 .TAU application 4.6 2,663 2,663 798370 0 3 .TAU application => MPI_Win_unlock() 1.4 814 814 100001 0 8 .TAU application => coback1tau => MPI_Isend() 1.4 814 814 100001 0 8 MPI_Isend() [THROTTLED] 0.2 137 137 100001 0 1 .TAU application => coback1tau => MPI_Wait() 0.2 137 137 100001 0 1 MPI_Wait() [THROTTLED] 0.2 107 107 1 0 107522 .TAU application => MPI_Init() 0.2 107 107 1 0 107522 MPI_Init() 0.2 105 105 100001 0 1 MPI_Win_lock() [THROTTLED] 0.2 105 105 99980 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.1 79 79 100001 0 1 MPI_Get() [THROTTLED] 0.1 79 79 99991 0 1 .TAU application => coback1tau => MPI_Get() 0.1 78 78 1 0 78029 .TAU application => MPI_Finalize() 0.1 78 78 1 0 78029 MPI_Finalize() 0.1 35 35 8 0 4386 MPI_Win_create() 0.1 34 34 4 0 8641 .TAU application => coback1tau => MPI_Win_create() 0.0 1 1 8 0 170 MPI_Barrier() 0.0 0.883 0.883 5 0 177 .TAU application => coback1tau => MPI_Barrier() 0.0 0.539 0.539 7 0 77 .TAU application => MPI_Win_free() 0.0 0.539 0.539 7 0 77 MPI_Win_free() 0.0 0.521 0.521 4 0 130 .TAU application => MPI_Win_create() 0.0 0.475 0.475 3 0 158 .TAU application => MPI_Barrier() 0.0 0.121 0.121 1 0 121 .TAU application => MPI_Comm_dup() 0.0 0.121 0.121 1 0 121 MPI_Comm_dup() 0.0 0.056 0.056 40 0 1 .TAU application => MPI_Put() 0.0 0.056 0.056 40 0 1 MPI_Put() 0.0 0.041 0.041 1 0 41 .TAU application => caflaunch$MAIN$blk 0.0 0.041 0.041 1 0 41 caflaunch$MAIN$blk 0.0 0.037 0.037 21 0 2 .TAU application => MPI_Win_lock() 0.0 0.03 0.03 7 0 4 MPI_Win_fence() 0.0 0.021 0.021 10 0 2 .TAU application => MPI_Get() 0.0 0.017 0.017 1 0 17 .TAU application => CI0000$coback1tau_$BLK 0.0 0.017 0.017 1 0 17 CI0000$coback1tau_$BLK 0.0 0.015 0.015 3 0 5 .TAU application => MPI_Win_fence() 0.0 0.015 0.015 4 0 4 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.014 0.014 1 0 14 .TAU application => MPI_Comm_rank() 0.0 0.014 0.014 1 0 14 MPI_Comm_rank() 0.0 0.011 0.011 1 0 11 .TAU application => MPI_Finalized() 0.0 0.011 0.011 1 0 11 MPI_Finalized() 0.0 0.01 0.01 1 0 10 .TAU application => MPI_Alloc_mem() 0.0 0.01 0.01 1 0 10 MPI_Alloc_mem() 0.0 0.007 0.007 1 0 7 .TAU application => MPI_Comm_size() 0.0 0.007 0.007 1 0 7 MPI_Comm_size() --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 1, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 3.6E+05 4 4 4 0 Message size received from all nodes 3.6E+05 4 4 4 0 Message size sent to all nodes 1.8E+05 4 4 4 0 Message size sent to node 0 1.3E+05 4 4 4 0 Message size sent to node 0 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 0 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 2 1.3E+05 4 4 4 0 Message size sent to node 2 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 2 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau => MPI_Isend() --------------------------------------------------------------------------------------- NODE 2;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 51.4 30,029 30,029 2.60424E+06 0 12 MPI_Win_unlock() 47.0 27,463 27,463 1.80009E+06 0 15 .TAU application => coback1tau => MPI_Win_unlock() 82.6 12,901 48,247 1 2.56008E+06 48247985 .TAU application => coback1tau 82.6 12,901 48,247 1 2.56008E+06 48247985 coback1tau 100.0 7,441 58,442 1 804237 58442292 .TAU application 11.6 6,783 6,783 360000 0 19 .TAU application => coback1tau => MPI_Recv() 11.6 6,783 6,783 360000 0 19 MPI_Recv() 4.4 2,565 2,565 804142 0 3 .TAU application => MPI_Win_unlock() 1.2 713 713 100001 0 7 .TAU application => coback1tau => MPI_Isend() 1.2 713 713 100001 0 7 MPI_Isend() [THROTTLED] 0.3 151 151 100001 0 2 .TAU application => coback1tau => MPI_Wait() 0.3 151 151 100001 0 2 MPI_Wait() [THROTTLED] 0.2 112 112 100001 0 1 MPI_Win_lock() [THROTTLED] 0.2 112 112 99980 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.2 107 107 1 0 107347 .TAU application => MPI_Init() 0.2 107 107 1 0 107347 MPI_Init() 0.1 86 86 100001 0 1 MPI_Get() [THROTTLED] 0.1 86 86 99991 0 1 .TAU application => coback1tau => MPI_Get() 0.1 78 78 1 0 78101 .TAU application => MPI_Finalize() 0.1 78 78 1 0 78101 MPI_Finalize() 0.1 35 35 8 0 4384 MPI_Win_create() 0.1 34 34 4 0 8634 .TAU application => coback1tau => MPI_Win_create() 0.0 1 1 8 0 155 MPI_Barrier() 0.0 0.678 0.678 5 0 136 .TAU application => coback1tau => MPI_Barrier() 0.0 0.561 0.561 3 0 187 .TAU application => MPI_Barrier() 0.0 0.536 0.536 4 0 134 .TAU application => MPI_Win_create() 0.0 0.463 0.463 7 0 66 .TAU application => MPI_Win_free() 0.0 0.463 0.463 7 0 66 MPI_Win_free() 0.0 0.101 0.101 1 0 101 .TAU application => MPI_Comm_dup() 0.0 0.101 0.101 1 0 101 MPI_Comm_dup() 0.0 0.044 0.044 7 0 6 MPI_Win_fence() 0.0 0.04 0.04 37 0 1 .TAU application => MPI_Put() 0.0 0.04 0.04 37 0 1 MPI_Put() 0.0 0.038 0.038 1 0 38 .TAU application => caflaunch$MAIN$blk 0.0 0.038 0.038 1 0 38 caflaunch$MAIN$blk 0.0 0.029 0.029 4 0 7 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.027 0.027 1 0 27 .TAU application => MPI_Alloc_mem() 0.0 0.027 0.027 21 0 1 .TAU application => MPI_Win_lock() 0.0 0.027 0.027 1 0 27 MPI_Alloc_mem() 0.0 0.019 0.019 1 0 19 .TAU application => MPI_Comm_rank() 0.0 0.019 0.019 10 0 2 .TAU application => MPI_Get() 0.0 0.019 0.019 1 0 19 MPI_Comm_rank() 0.0 0.017 0.017 1 0 17 .TAU application => CI0000$coback1tau_$BLK 0.0 0.017 0.017 1 0 17 CI0000$coback1tau_$BLK 0.0 0.015 0.015 3 0 5 .TAU application => MPI_Win_fence() 0.0 0.01 0.01 1 0 10 .TAU application => MPI_Finalized() 0.0 0.01 0.01 1 0 10 MPI_Finalized() 0.0 0.006 0.006 1 0 6 .TAU application => MPI_Comm_size() 0.0 0.006 0.006 1 0 6 MPI_Comm_size() --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 2, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 3.6E+05 4 4 4 0 Message size received from all nodes 3.6E+05 4 4 4 0 Message size sent to all nodes 0 0 0 0 0 Message size sent to node 0 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 1 1.3E+05 4 4 4 0 Message size sent to node 1 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 1 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 3 1.3E+05 4 4 4 0 Message size sent to node 3 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 3 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau => MPI_Isend() --------------------------------------------------------------------------------------- NODE 3;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 51.3 29,975 29,975 2.60287E+06 0 12 MPI_Win_unlock() 46.8 27,344 27,344 1.8001E+06 0 15 .TAU application => coback1tau => MPI_Win_unlock() 82.6 13,020 48,248 1 2.56008E+06 48248195 .TAU application => coback1tau 82.6 13,020 48,248 1 2.56008E+06 48248195 coback1tau 100.0 7,375 58,442 1 802861 58442326 .TAU application 11.5 6,736 6,736 360000 0 19 .TAU application => coback1tau => MPI_Recv() 11.5 6,736 6,736 360000 0 19 MPI_Recv() 4.5 2,631 2,631 802769 0 3 .TAU application => MPI_Win_unlock() 1.3 758 758 100001 0 8 .TAU application => coback1tau => MPI_Isend() 1.3 758 758 100001 0 8 MPI_Isend() [THROTTLED] 0.3 154 154 100001 0 2 .TAU application => coback1tau => MPI_Wait() 0.3 154 154 100001 0 2 MPI_Wait() [THROTTLED] 0.2 113 113 100001 0 1 MPI_Win_lock() [THROTTLED] 0.2 113 113 99980 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.2 107 107 1 0 107681 .TAU application => MPI_Init() 0.2 107 107 1 0 107681 MPI_Init() 0.1 85 85 100001 0 1 MPI_Get() [THROTTLED] 0.1 85 85 99991 0 1 .TAU application => coback1tau => MPI_Get() 0.1 78 78 1 0 78141 .TAU application => MPI_Finalize() 0.1 78 78 1 0 78141 MPI_Finalize() 0.1 35 35 8 0 4383 MPI_Win_create() 0.1 34 34 4 0 8638 .TAU application => coback1tau => MPI_Win_create() 0.0 1 1 8 0 141 MPI_Barrier() 0.0 0.683 0.683 5 0 137 .TAU application => coback1tau => MPI_Barrier() 0.0 0.541 0.541 7 0 77 .TAU application => MPI_Win_free() 0.0 0.541 0.541 7 0 77 MPI_Win_free() 0.0 0.514 0.514 4 0 128 .TAU application => MPI_Win_create() 0.0 0.445 0.445 3 0 148 .TAU application => MPI_Barrier() 0.0 0.103 0.103 1 0 103 .TAU application => MPI_Comm_dup() 0.0 0.103 0.103 1 0 103 MPI_Comm_dup() 0.0 0.076 0.076 35 0 2 MPI_Put() 0.0 0.052 0.052 34 0 2 .TAU application => MPI_Put() 0.0 0.052 0.052 1 0 52 .TAU application => caflaunch$MAIN$blk 0.0 0.052 0.052 1 0 52 caflaunch$MAIN$blk 0.0 0.032 0.032 7 0 5 MPI_Win_fence() 0.0 0.031 0.031 21 0 1 .TAU application => MPI_Win_lock() 0.0 0.024 0.024 1 0 24 .TAU application => MPI_Alloc_mem() 0.0 0.024 0.024 1 0 24 .TAU application => coback1tau => MPI_Put() 0.0 0.024 0.024 1 0 24 MPI_Alloc_mem() 0.0 0.02 0.02 10 0 2 .TAU application => MPI_Get() 0.0 0.019 0.019 1 0 19 .TAU application => CI0000$coback1tau_$BLK 0.0 0.019 0.019 1 0 19 CI0000$coback1tau_$BLK 0.0 0.017 0.017 4 0 4 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.015 0.015 1 0 15 .TAU application => MPI_Comm_rank() 0.0 0.015 0.015 3 0 5 .TAU application => MPI_Win_fence() 0.0 0.015 0.015 1 0 15 MPI_Comm_rank() 0.0 0.011 0.011 1 0 11 .TAU application => MPI_Finalized() 0.0 0.011 0.011 1 0 11 MPI_Finalized() 0.0 0.004 0.004 1 0 4 .TAU application => MPI_Comm_size() 0.0 0.004 0.004 1 0 4 MPI_Comm_size() --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 3, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 3.6E+05 4 4 4 0 Message size received from all nodes 3.6E+05 4 4 4 0 Message size sent to all nodes 0 0 0 0 0 Message size sent to node 0 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 2 1.3E+05 4 4 4 0 Message size sent to node 2 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 2 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 4 1.3E+05 4 4 4 0 Message size sent to node 4 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 4 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau => MPI_Isend() --------------------------------------------------------------------------------------- NODE 4;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 52.8 30,885 30,885 2.60385E+06 0 12 MPI_Win_unlock() 48.4 28,315 28,315 1.8001E+06 0 16 .TAU application => coback1tau => MPI_Win_unlock() 82.6 13,007 48,248 1 2.56009E+06 48248345 .TAU application => coback1tau 82.6 13,007 48,248 1 2.56009E+06 48248345 coback1tau 100.0 7,432 58,443 1 803841 58443744 .TAU application 9.9 5,798 5,798 360000 0 16 .TAU application => coback1tau => MPI_Recv() 9.9 5,798 5,798 360000 0 16 MPI_Recv() 4.4 2,569 2,569 803752 0 3 .TAU application => MPI_Win_unlock() 1.3 744 744 100001 0 7 .TAU application => coback1tau => MPI_Isend() 1.3 744 744 100001 0 7 MPI_Isend() [THROTTLED] 0.3 152 152 100001 0 2 .TAU application => coback1tau => MPI_Wait() 0.3 152 152 100001 0 2 MPI_Wait() [THROTTLED] 0.2 111 111 1 0 111990 .TAU application => MPI_Init() 0.2 111 111 1 0 111990 MPI_Init() 0.2 108 108 100001 0 1 MPI_Win_lock() [THROTTLED] 0.2 108 108 99980 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.1 85 85 100001 0 1 MPI_Get() [THROTTLED] 0.1 85 85 99991 0 1 .TAU application => coback1tau => MPI_Get() 0.1 79 79 1 0 79543 .TAU application => MPI_Finalize() 0.1 79 79 1 0 79543 MPI_Finalize() 0.1 35 35 8 0 4390 MPI_Win_create() 0.1 34 34 4 0 8639 .TAU application => coback1tau => MPI_Win_create() 0.0 0.826 0.826 8 0 103 MPI_Barrier() 0.0 0.566 0.566 7 0 81 .TAU application => MPI_Win_free() 0.0 0.566 0.566 7 0 81 MPI_Win_free() 0.0 0.562 0.562 4 0 140 .TAU application => MPI_Win_create() 0.0 0.49 0.49 5 0 98 .TAU application => coback1tau => MPI_Barrier() 0.0 0.336 0.336 3 0 112 .TAU application => MPI_Barrier() 0.0 0.096 0.096 1 0 96 .TAU application => MPI_Comm_dup() 0.0 0.096 0.096 1 0 96 MPI_Comm_dup() 0.0 0.061 0.061 32 0 2 MPI_Put() 0.0 0.053 0.053 1 0 53 .TAU application => caflaunch$MAIN$blk 0.0 0.053 0.053 1 0 53 caflaunch$MAIN$blk 0.0 0.046 0.046 31 0 1 .TAU application => MPI_Put() 0.0 0.031 0.031 21 0 1 .TAU application => MPI_Win_lock() 0.0 0.031 0.031 7 0 4 MPI_Win_fence() 0.0 0.03 0.03 1 0 30 .TAU application => MPI_Alloc_mem() 0.0 0.03 0.03 1 0 30 MPI_Alloc_mem() 0.0 0.019 0.019 1 0 19 .TAU application => CI0000$coback1tau_$BLK 0.0 0.019 0.019 1 0 19 CI0000$coback1tau_$BLK 0.0 0.018 0.018 10 0 2 .TAU application => MPI_Get() 0.0 0.017 0.017 4 0 4 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.016 0.016 1 0 16 .TAU application => MPI_Comm_rank() 0.0 0.016 0.016 1 0 16 MPI_Comm_rank() 0.0 0.015 0.015 1 0 15 .TAU application => coback1tau => MPI_Put() 0.0 0.014 0.014 3 0 5 .TAU application => MPI_Win_fence() 0.0 0.01 0.01 1 0 10 .TAU application => MPI_Finalized() 0.0 0.01 0.01 1 0 10 MPI_Finalized() 0.0 0.004 0.004 1 0 4 .TAU application => MPI_Comm_size() 0.0 0.004 0.004 1 0 4 MPI_Comm_size() --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 4, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 3.6E+05 4 4 4 0 Message size received from all nodes 3.6E+05 4 4 4 0 Message size sent to all nodes 0 0 0 0 0 Message size sent to node 0 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 3 1.3E+05 4 4 4 0 Message size sent to node 3 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 3 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 5 1.3E+05 4 4 4 0 Message size sent to node 5 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 5 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau => MPI_Isend() --------------------------------------------------------------------------------------- NODE 5;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 51.9 30,358 30,358 2.58808E+06 0 12 MPI_Win_unlock() 47.5 27,774 27,774 1.8001E+06 0 15 .TAU application => coback1tau => MPI_Win_unlock() 82.6 13,102 48,248 1 2.56009E+06 48248434 .TAU application => coback1tau 82.6 13,102 48,248 1 2.56009E+06 48248434 coback1tau 100.0 7,417 58,441 1 788062 58441346 .TAU application 10.6 6,212 6,212 360000 0 17 .TAU application => coback1tau => MPI_Recv() 10.6 6,212 6,212 360000 0 17 MPI_Recv() 4.4 2,583 2,583 787976 0 3 .TAU application => MPI_Win_unlock() 1.3 779 779 100001 0 8 .TAU application => coback1tau => MPI_Isend() 1.3 779 779 100001 0 8 MPI_Isend() [THROTTLED] 0.3 153 153 100001 0 2 .TAU application => coback1tau => MPI_Wait() 0.3 153 153 100001 0 2 MPI_Wait() [THROTTLED] 0.2 111 111 1 0 111917 .TAU application => MPI_Init() 0.2 111 111 1 0 111917 MPI_Init() 0.2 106 106 100001 0 1 MPI_Win_lock() [THROTTLED] 0.2 106 106 99980 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.1 83 83 100001 0 1 MPI_Get() [THROTTLED] 0.1 83 83 99991 0 1 .TAU application => coback1tau => MPI_Get() 0.1 78 78 1 0 78167 .TAU application => MPI_Finalize() 0.1 78 78 1 0 78167 MPI_Finalize() 0.1 35 35 8 0 4396 MPI_Win_create() 0.1 34 34 4 0 8658 .TAU application => coback1tau => MPI_Win_create() 0.0 0.762 0.762 8 0 95 MPI_Barrier() 0.0 0.581 0.581 7 0 83 .TAU application => MPI_Win_free() 0.0 0.581 0.581 7 0 83 MPI_Win_free() 0.0 0.542 0.542 4 0 136 .TAU application => MPI_Win_create() 0.0 0.425 0.425 5 0 85 .TAU application => coback1tau => MPI_Barrier() 0.0 0.337 0.337 3 0 112 .TAU application => MPI_Barrier() 0.0 0.096 0.096 1 0 96 .TAU application => MPI_Comm_dup() 0.0 0.096 0.096 1 0 96 MPI_Comm_dup() 0.0 0.067 0.067 29 0 2 MPI_Put() 0.0 0.052 0.052 1 0 52 .TAU application => caflaunch$MAIN$blk 0.0 0.052 0.052 1 0 52 caflaunch$MAIN$blk 0.0 0.044 0.044 28 0 2 .TAU application => MPI_Put() 0.0 0.033 0.033 1 0 33 .TAU application => MPI_Alloc_mem() 0.0 0.033 0.033 21 0 2 .TAU application => MPI_Win_lock() 0.0 0.033 0.033 1 0 33 MPI_Alloc_mem() 0.0 0.031 0.031 7 0 4 MPI_Win_fence() 0.0 0.023 0.023 1 0 23 .TAU application => coback1tau => MPI_Put() 0.0 0.018 0.018 1 0 18 .TAU application => MPI_Comm_rank() 0.0 0.018 0.018 10 0 2 .TAU application => MPI_Get() 0.0 0.018 0.018 4 0 4 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.018 0.018 1 0 18 MPI_Comm_rank() 0.0 0.017 0.017 1 0 17 .TAU application => CI0000$coback1tau_$BLK 0.0 0.017 0.017 1 0 17 CI0000$coback1tau_$BLK 0.0 0.013 0.013 3 0 4 .TAU application => MPI_Win_fence() 0.0 0.01 0.01 1 0 10 .TAU application => MPI_Finalized() 0.0 0.01 0.01 1 0 10 MPI_Finalized() 0.0 0.007 0.007 1 0 7 .TAU application => MPI_Comm_size() 0.0 0.007 0.007 1 0 7 MPI_Comm_size() --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 5, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 3.6E+05 4 4 4 0 Message size received from all nodes 3.6E+05 4 4 4 0 Message size sent to all nodes 0 0 0 0 0 Message size sent to node 0 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 4 1.3E+05 4 4 4 0 Message size sent to node 4 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 4 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 6 1.3E+05 4 4 4 0 Message size sent to node 6 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 6 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau => MPI_Isend() --------------------------------------------------------------------------------------- NODE 6;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 49.3 28,785 28,785 2.59369E+06 0 11 MPI_Win_unlock() 44.8 26,186 26,186 1.8001E+06 0 15 .TAU application => coback1tau => MPI_Win_unlock() 82.6 13,057 48,248 1 2.56008E+06 48248504 .TAU application => coback1tau 82.6 13,057 48,248 1 2.56008E+06 48248504 coback1tau 13.4 7,848 7,848 360000 0 22 .TAU application => coback1tau => MPI_Recv() 13.4 7,848 7,848 360000 0 22 MPI_Recv() 100.0 7,397 58,444 1 793678 58444848 .TAU application 4.4 2,599 2,599 793595 0 3 .TAU application => MPI_Win_unlock() 1.3 773 773 100001 0 8 .TAU application => coback1tau => MPI_Isend() 1.3 773 773 100001 0 8 MPI_Isend() [THROTTLED] 0.3 152 152 100001 0 2 .TAU application => coback1tau => MPI_Wait() 0.3 152 152 100001 0 2 MPI_Wait() [THROTTLED] 0.2 119 119 1 0 119712 .TAU application => MPI_Init() 0.2 119 119 1 0 119712 MPI_Init() 0.2 112 112 100001 0 1 MPI_Win_lock() [THROTTLED] 0.2 112 112 99980 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.1 83 83 100001 0 1 MPI_Get() [THROTTLED] 0.1 83 83 99991 0 1 .TAU application => coback1tau => MPI_Get() 0.1 78 78 1 0 78057 .TAU application => MPI_Finalize() 0.1 78 78 1 0 78057 MPI_Finalize() 0.1 35 35 8 0 4413 MPI_Win_create() 0.1 34 34 4 0 8688 .TAU application => coback1tau => MPI_Win_create() 0.0 0.84 0.84 8 0 105 MPI_Barrier() 0.0 0.571 0.571 7 0 82 .TAU application => MPI_Win_free() 0.0 0.571 0.571 7 0 82 MPI_Win_free() 0.0 0.55 0.55 4 0 138 .TAU application => MPI_Win_create() 0.0 0.48 0.48 3 0 160 .TAU application => MPI_Barrier() 0.0 0.36 0.36 5 0 72 .TAU application => coback1tau => MPI_Barrier() 0.0 0.101 0.101 1 0 101 .TAU application => MPI_Comm_dup() 0.0 0.101 0.101 1 0 101 MPI_Comm_dup() 0.0 0.08 0.08 1 0 80 .TAU application => caflaunch$MAIN$blk 0.0 0.08 0.08 1 0 80 caflaunch$MAIN$blk 0.0 0.042 0.042 25 0 2 .TAU application => MPI_Put() 0.0 0.042 0.042 25 0 2 MPI_Put() 0.0 0.042 0.042 7 0 6 MPI_Win_fence() 0.0 0.037 0.037 21 0 2 .TAU application => MPI_Win_lock() 0.0 0.026 0.026 4 0 6 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.025 0.025 1 0 25 .TAU application => CI0000$coback1tau_$BLK 0.0 0.025 0.025 1 0 25 CI0000$coback1tau_$BLK 0.0 0.016 0.016 10 0 2 .TAU application => MPI_Get() 0.0 0.016 0.016 3 0 5 .TAU application => MPI_Win_fence() 0.0 0.015 0.015 1 0 15 .TAU application => MPI_Alloc_mem() 0.0 0.015 0.015 1 0 15 .TAU application => MPI_Comm_rank() 0.0 0.015 0.015 1 0 15 MPI_Alloc_mem() 0.0 0.015 0.015 1 0 15 MPI_Comm_rank() 0.0 0.012 0.012 1 0 12 .TAU application => MPI_Finalized() 0.0 0.012 0.012 1 0 12 MPI_Finalized() 0.0 0.007 0.007 1 0 7 .TAU application => MPI_Comm_size() 0.0 0.007 0.007 1 0 7 MPI_Comm_size() --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 6, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 3.6E+05 4 4 4 0 Message size received from all nodes 3.6E+05 4 4 4 0 Message size sent to all nodes 0 0 0 0 0 Message size sent to node 0 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 5 1.3E+05 4 4 4 0 Message size sent to node 5 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 5 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 7 1.3E+05 4 4 4 0 Message size sent to node 7 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 7 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau => MPI_Isend() --------------------------------------------------------------------------------------- NODE 7;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 46.3 27,082 27,082 2.60224E+06 0 10 MPI_Win_unlock() 41.6 24,322 24,322 1.8001E+06 0 14 .TAU application => coback1tau => MPI_Win_unlock() 82.6 12,994 48,248 1 2.56008E+06 48248538 .TAU application => coback1tau 82.6 12,994 48,248 1 2.56008E+06 48248538 coback1tau 16.7 9,783 9,783 360000 0 27 .TAU application => coback1tau => MPI_Recv() 16.7 9,783 9,783 360000 0 27 MPI_Recv() 100.0 7,245 58,440 1 802220 58440936 .TAU application 4.7 2,759 2,759 802140 0 3 .TAU application => MPI_Win_unlock() 1.3 772 772 100001 0 8 .TAU application => coback1tau => MPI_Isend() 1.3 772 772 100001 0 8 MPI_Isend() [THROTTLED] 0.3 146 146 100001 0 1 .TAU application => coback1tau => MPI_Wait() 0.3 146 146 100001 0 1 MPI_Wait() [THROTTLED] 0.2 110 110 100001 0 1 MPI_Win_lock() [THROTTLED] 0.2 110 110 99980 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.2 107 107 1 0 107476 .TAU application => MPI_Init() 0.2 107 107 1 0 107476 MPI_Init() 0.1 82 82 100001 0 1 MPI_Get() [THROTTLED] 0.1 82 82 99991 0 1 .TAU application => coback1tau => MPI_Get() 0.1 77 77 1 0 77755 .TAU application => MPI_Finalize() 0.1 77 77 1 0 77755 MPI_Finalize() 0.1 35 35 8 0 4396 MPI_Win_create() 0.1 34 34 4 0 8660 .TAU application => coback1tau => MPI_Win_create() 0.0 1 1 8 0 132 MPI_Barrier() 0.0 0.605 0.605 5 0 121 .TAU application => coback1tau => MPI_Barrier() 0.0 0.583 0.583 7 0 83 .TAU application => MPI_Win_free() 0.0 0.583 0.583 7 0 83 MPI_Win_free() 0.0 0.53 0.53 4 0 132 .TAU application => MPI_Win_create() 0.0 0.452 0.452 3 0 151 .TAU application => MPI_Barrier() 0.0 0.098 0.098 1 0 98 .TAU application => MPI_Comm_dup() 0.0 0.098 0.098 1 0 98 MPI_Comm_dup() 0.0 0.045 0.045 1 0 45 .TAU application => caflaunch$MAIN$blk 0.0 0.045 0.045 1 0 45 caflaunch$MAIN$blk 0.0 0.044 0.044 23 0 2 MPI_Put() 0.0 0.036 0.036 7 0 5 MPI_Win_fence() 0.0 0.029 0.029 1 0 29 .TAU application => MPI_Alloc_mem() 0.0 0.029 0.029 22 0 1 .TAU application => MPI_Put() 0.0 0.029 0.029 1 0 29 MPI_Alloc_mem() 0.0 0.027 0.027 21 0 1 .TAU application => MPI_Win_lock() 0.0 0.021 0.021 1 0 21 .TAU application => MPI_Comm_rank() 0.0 0.021 0.021 4 0 5 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.021 0.021 1 0 21 MPI_Comm_rank() 0.0 0.019 0.019 10 0 2 .TAU application => MPI_Get() 0.0 0.018 0.018 1 0 18 .TAU application => CI0000$coback1tau_$BLK 0.0 0.018 0.018 1 0 18 CI0000$coback1tau_$BLK 0.0 0.015 0.015 3 0 5 .TAU application => MPI_Win_fence() 0.0 0.015 0.015 1 0 15 .TAU application => coback1tau => MPI_Put() 0.0 0.012 0.012 1 0 12 .TAU application => MPI_Finalized() 0.0 0.012 0.012 1 0 12 MPI_Finalized() 0.0 0.007 0.007 1 0 7 .TAU application => MPI_Comm_size() 0.0 0.007 0.007 1 0 7 MPI_Comm_size() --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 7, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 3.6E+05 4 4 4 0 Message size received from all nodes 3.6E+05 4 4 4 0 Message size sent to all nodes 0 0 0 0 0 Message size sent to node 0 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 6 1.3E+05 4 4 4 0 Message size sent to node 6 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 6 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 8 1.3E+05 4 4 4 0 Message size sent to node 8 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 8 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau => MPI_Isend() --------------------------------------------------------------------------------------- NODE 8;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 45.7 26,720 26,720 2.59537E+06 0 10 MPI_Win_unlock() 40.8 23,862 23,862 1.8001E+06 0 13 .TAU application => coback1tau => MPI_Win_unlock() 82.6 13,421 48,248 1 2.56008E+06 48248625 .TAU application => coback1tau 82.6 13,421 48,248 1 2.56008E+06 48248625 coback1tau 16.7 9,753 9,753 360000 0 27 .TAU application => coback1tau => MPI_Recv() 16.7 9,753 9,753 360000 0 27 MPI_Recv() 100.0 7,139 58,444 1 795353 58444614 .TAU application 4.9 2,857 2,857 795276 0 4 .TAU application => MPI_Win_unlock() 1.4 834 834 100001 0 8 .TAU application => coback1tau => MPI_Isend() 1.4 834 834 100001 0 8 MPI_Isend() [THROTTLED] 0.3 148 148 100001 0 1 .TAU application => coback1tau => MPI_Wait() 0.3 148 148 100001 0 1 MPI_Wait() [THROTTLED] 0.2 119 119 1 0 119830 .TAU application => MPI_Init() 0.2 119 119 1 0 119830 MPI_Init() 0.2 110 110 100001 0 1 MPI_Win_lock() [THROTTLED] 0.2 110 110 99980 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.1 81 81 100001 0 1 MPI_Get() [THROTTLED] 0.1 81 81 99991 0 1 .TAU application => coback1tau => MPI_Get() 0.1 77 77 1 0 77813 .TAU application => MPI_Finalize() 0.1 77 77 1 0 77813 MPI_Finalize() 0.1 35 35 8 0 4392 MPI_Win_create() 0.1 34 34 4 0 8648 .TAU application => coback1tau => MPI_Win_create() 0.0 0.954 0.954 8 0 119 MPI_Barrier() 0.0 0.578 0.578 7 0 83 .TAU application => MPI_Win_free() 0.0 0.578 0.578 7 0 83 MPI_Win_free() 0.0 0.546 0.546 4 0 136 .TAU application => MPI_Win_create() 0.0 0.542 0.542 5 0 108 .TAU application => coback1tau => MPI_Barrier() 0.0 0.412 0.412 3 0 137 .TAU application => MPI_Barrier() 0.0 0.098 0.098 1 0 98 .TAU application => MPI_Comm_dup() 0.0 0.098 0.098 1 0 98 MPI_Comm_dup() 0.0 0.057 0.057 1 0 57 .TAU application => caflaunch$MAIN$blk 0.0 0.057 0.057 1 0 57 caflaunch$MAIN$blk 0.0 0.038 0.038 20 0 2 MPI_Put() 0.0 0.027 0.027 7 0 4 MPI_Win_fence() 0.0 0.026 0.026 1 0 26 .TAU application => MPI_Alloc_mem() 0.0 0.026 0.026 21 0 1 .TAU application => MPI_Win_lock() 0.0 0.026 0.026 1 0 26 MPI_Alloc_mem() 0.0 0.023 0.023 19 0 1 .TAU application => MPI_Put() 0.0 0.02 0.02 1 0 20 .TAU application => CI0000$coback1tau_$BLK 0.0 0.02 0.02 1 0 20 CI0000$coback1tau_$BLK 0.0 0.019 0.019 1 0 19 .TAU application => MPI_Comm_rank() 0.0 0.019 0.019 1 0 19 MPI_Comm_rank() 0.0 0.017 0.017 10 0 2 .TAU application => MPI_Get() 0.0 0.015 0.015 1 0 15 .TAU application => coback1tau => MPI_Put() 0.0 0.015 0.015 4 0 4 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.013 0.013 1 0 13 .TAU application => MPI_Finalized() 0.0 0.013 0.013 1 0 13 MPI_Finalized() 0.0 0.012 0.012 3 0 4 .TAU application => MPI_Win_fence() 0.0 0.004 0.004 1 0 4 .TAU application => MPI_Comm_size() 0.0 0.004 0.004 1 0 4 MPI_Comm_size() --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 8, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 3.6E+05 4 4 4 0 Message size received from all nodes 3.6E+05 4 4 4 0 Message size sent to all nodes 0 0 0 0 0 Message size sent to node 0 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 7 1.3E+05 4 4 4 0 Message size sent to node 7 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 7 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 8 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 8 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 9 1.3E+05 4 4 4 0 Message size sent to node 9 : .TAU application => coback1tau 5E+04 4 4 4 0 Message size sent to node 9 : .TAU application => coback1tau => MPI_Isend() --------------------------------------------------------------------------------------- NODE 9;CONTEXT 0;THREAD 0: --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 34.3 20,055 20,055 180000 0 111 .TAU application => coback1tau => MPI_Recv() 34.3 20,055 20,055 180000 0 111 MPI_Recv() 34.1 19,921 19,921 1.6922E+06 0 12 MPI_Win_unlock() 29.1 17,001 17,001 900093 0 19 .TAU application => coback1tau => MPI_Win_unlock() 82.6 10,052 48,248 1 1.48008E+06 48248542 .TAU application => coback1tau 82.6 10,052 48,248 1 1.48008E+06 48248542 coback1tau 100.0 7,086 58,442 1 792184 58442274 .TAU application 5.0 2,920 2,920 792106 0 4 .TAU application => MPI_Win_unlock() 1.1 651 651 100001 0 7 .TAU application => coback1tau => MPI_Isend() 1.1 651 651 100001 0 7 MPI_Isend() [THROTTLED] 0.4 221 221 100001 0 2 .TAU application => coback1tau => MPI_Wait() 0.4 221 221 100001 0 2 MPI_Wait() [THROTTLED] 0.2 132 132 100001 0 1 MPI_Win_lock() [THROTTLED] 0.2 132 132 99980 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.2 107 107 1 0 107469 .TAU application => MPI_Init() 0.2 107 107 1 0 107469 MPI_Init() 0.2 98 98 100001 0 1 MPI_Get() [THROTTLED] 0.2 98 98 99991 0 1 .TAU application => coback1tau => MPI_Get() 0.1 77 77 1 0 77780 .TAU application => MPI_Finalize() 0.1 77 77 1 0 77780 MPI_Finalize() 0.1 35 35 8 0 4386 MPI_Win_create() 0.1 34 34 4 0 8644 .TAU application => coback1tau => MPI_Win_create() 0.0 1 1 8 0 160 MPI_Barrier() 0.0 1 1 5 0 205 .TAU application => coback1tau => MPI_Barrier() 0.0 0.588 0.588 7 0 84 .TAU application => MPI_Win_free() 0.0 0.588 0.588 7 0 84 MPI_Win_free() 0.0 0.511 0.511 4 0 128 .TAU application => MPI_Win_create() 0.0 0.257 0.257 3 0 86 .TAU application => MPI_Barrier() 0.0 0.1 0.1 1 0 100 .TAU application => MPI_Comm_dup() 0.0 0.1 0.1 1 0 100 MPI_Comm_dup() 0.0 0.045 0.045 1 0 45 .TAU application => caflaunch$MAIN$blk 0.0 0.045 0.045 1 0 45 caflaunch$MAIN$blk 0.0 0.034 0.034 21 0 2 .TAU application => MPI_Win_lock() 0.0 0.029 0.029 20 0 1 .TAU application => MPI_Put() 0.0 0.029 0.029 20 0 1 MPI_Put() 0.0 0.028 0.028 1 0 28 .TAU application => MPI_Alloc_mem() 0.0 0.028 0.028 1 0 28 MPI_Alloc_mem() 0.0 0.028 0.028 7 0 4 MPI_Win_fence() 0.0 0.019 0.019 10 0 2 .TAU application => MPI_Get() 0.0 0.018 0.018 1 0 18 .TAU application => CI0000$coback1tau_$BLK 0.0 0.018 0.018 1 0 18 .TAU application => MPI_Comm_rank() 0.0 0.018 0.018 1 0 18 CI0000$coback1tau_$BLK 0.0 0.018 0.018 1 0 18 MPI_Comm_rank() 0.0 0.015 0.015 3 0 5 .TAU application => MPI_Win_fence() 0.0 0.013 0.013 4 0 3 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.011 0.011 1 0 11 .TAU application => MPI_Finalized() 0.0 0.011 0.011 1 0 11 MPI_Finalized() 0.0 0.004 0.004 1 0 4 .TAU application => MPI_Comm_size() 0.0 0.004 0.004 1 0 4 MPI_Comm_size() --------------------------------------------------------------------------------------- USER EVENTS Profile :NODE 9, CONTEXT 0, THREAD 0 --------------------------------------------------------------------------------------- NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name --------------------------------------------------------------------------------------- 1.8E+05 4 4 4 0 Message size received from all nodes 1.8E+05 4 4 4 0 Message size sent to all nodes 0 0 0 0 0 Message size sent to node 0 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 0 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 1 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 1 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 2 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 2 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 3 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 3 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 4 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 4 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 5 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 5 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 6 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 6 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 7 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 7 : .TAU application => coback1tau => MPI_Isend() 1.8E+05 4 4 4 0 Message size sent to node 8 8E+04 4 4 4 0 Message size sent to node 8 : .TAU application => coback1tau 1E+05 4 4 4 0 Message size sent to node 8 : .TAU application => coback1tau => MPI_Isend() 0 0 0 0 0 Message size sent to node 9 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau 0 0 0 0 0 Message size sent to node 9 : .TAU application => coback1tau => MPI_Isend() --------------------------------------------------------------------------------------- FUNCTION SUMMARY (total): --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 43.2 4:12.672 4:12.672 2.2481E+07 0 11 MPI_Win_unlock() 39.1 3:48.522 3:48.522 1.53009E+07 0 15 .TAU application => coback1tau => MPI_Win_unlock() 74.3 1:54.675 7:14.235 9 2.19607E+07 48248337 .TAU application => coback1tau 74.3 1:54.675 7:14.235 9 2.19607E+07 48248337 coback1tau 13.8 1:20.680 1:20.680 3.06E+06 0 26 .TAU application => coback1tau => MPI_Recv() 13.8 1:20.680 1:20.680 3.06E+06 0 26 MPI_Recv() 100.0 1:05.882 9:44.725 10 7.1809E+06 58472544 .TAU application 10.0 58,737 58,737 10 0 5873749 .TAU application => caflaunch$MAIN$blk 10.0 58,737 58,737 10 0 5873749 caflaunch$MAIN$blk 4.1 24,150 24,150 7.18013E+06 0 3 .TAU application => MPI_Win_unlock() 1.2 6,842 6,842 900009 0 8 .TAU application => coback1tau => MPI_Isend() 1.2 6,842 6,842 900009 0 8 MPI_Isend() [THROTTLED] 0.2 1,418 1,418 900009 0 2 .TAU application => coback1tau => MPI_Wait() 0.2 1,418 1,418 900009 0 2 MPI_Wait() [THROTTLED] 0.2 1,012 1,012 900009 0 1 MPI_Win_lock() [THROTTLED] 0.2 1,012 1,012 899820 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.2 1,000 1,000 9 0 111216 .TAU application => MPI_Init() 0.2 1,000 1,000 9 0 111216 MPI_Init() 0.1 767 767 900009 0 1 MPI_Get() [THROTTLED] 0.1 766 766 899919 0 1 .TAU application => coback1tau => MPI_Get() 0.1 703 703 9 0 78154 .TAU application => MPI_Finalize() 0.1 703 703 9 0 78154 MPI_Finalize() 0.1 316 316 72 0 4392 MPI_Win_create() 0.1 311 311 36 0 8650 .TAU application => coback1tau => MPI_Win_create() 0.0 9 9 72 0 131 MPI_Barrier() 0.0 5 5 45 0 126 .TAU application => coback1tau => MPI_Barrier() 0.0 5 5 63 0 80 .TAU application => MPI_Win_free() 0.0 5 5 63 0 80 MPI_Win_free() 0.0 4 4 36 0 134 .TAU application => MPI_Win_create() 0.0 3 3 27 0 139 .TAU application => MPI_Barrier() 0.0 0.914 0.914 9 0 102 .TAU application => MPI_Comm_dup() 0.0 0.914 0.914 9 0 102 MPI_Comm_dup() 0.0 0.453 0.453 261 0 2 MPI_Put() 0.0 0.361 0.361 256 0 1 .TAU application => MPI_Put() 0.0 0.301 0.301 63 0 5 MPI_Win_fence() 0.0 0.283 0.283 189 0 1 .TAU application => MPI_Win_lock() 0.0 0.222 0.222 9 0 25 .TAU application => MPI_Alloc_mem() 0.0 0.222 0.222 9 0 25 MPI_Alloc_mem() 0.0 0.171 0.171 36 0 5 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.17 0.17 9 0 19 .TAU application => CI0000$coback1tau_$BLK 0.0 0.17 0.17 9 0 19 CI0000$coback1tau_$BLK 0.0 0.167 0.167 90 0 2 .TAU application => MPI_Get() 0.0 0.155 0.155 9 0 17 .TAU application => MPI_Comm_rank() 0.0 0.155 0.155 9 0 17 MPI_Comm_rank() 0.0 0.13 0.13 27 0 5 .TAU application => MPI_Win_fence() 0.0 0.1 0.1 9 0 11 .TAU application => MPI_Finalized() 0.0 0.1 0.1 9 0 11 MPI_Finalized() 0.0 0.092 0.092 5 0 18 .TAU application => coback1tau => MPI_Put() 0.0 0.05 0.05 9 0 6 .TAU application => MPI_Comm_size() 0.0 0.05 0.05 9 0 6 MPI_Comm_size() FUNCTION SUMMARY (mean): --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 43.2 25,267 25,267 2.2481E+06 0 11 MPI_Win_unlock() 39.1 22,852 22,852 1.53009E+06 0 15 .TAU application => coback1tau => MPI_Win_unlock() 74.3 11,467 43,423 0.9 2.19607E+06 48248337 .TAU application => coback1tau 74.3 11,467 43,423 0.9 2.19607E+06 48248337 coback1tau 13.8 8,068 8,068 306000 0 26 .TAU application => coback1tau => MPI_Recv() 13.8 8,068 8,068 306000 0 26 MPI_Recv() 100.0 6,588 58,472 1 718090 58472544 .TAU application 10.0 5,873 5,873 1 0 5873749 .TAU application => caflaunch$MAIN$blk 10.0 5,873 5,873 1 0 5873749 caflaunch$MAIN$blk 4.1 2,415 2,415 718013 0 3 .TAU application => MPI_Win_unlock() 1.2 684 684 90000.9 0 8 .TAU application => coback1tau => MPI_Isend() 1.2 684 684 90000.9 0 8 MPI_Isend() [THROTTLED] 0.2 141 141 90000.9 0 2 .TAU application => coback1tau => MPI_Wait() 0.2 141 141 90000.9 0 2 MPI_Wait() [THROTTLED] 0.2 101 101 90000.9 0 1 MPI_Win_lock() [THROTTLED] 0.2 101 101 89982 0 1 .TAU application => coback1tau => MPI_Win_lock() 0.2 100 100 0.9 0 111216 .TAU application => MPI_Init() 0.2 100 100 0.9 0 111216 MPI_Init() 0.1 76 76 90000.9 0 1 MPI_Get() [THROTTLED] 0.1 76 76 89991.9 0 1 .TAU application => coback1tau => MPI_Get() 0.1 70 70 0.9 0 78154 .TAU application => MPI_Finalize() 0.1 70 70 0.9 0 78154 MPI_Finalize() 0.1 31 31 7.2 0 4392 MPI_Win_create() 0.1 31 31 3.6 0 8650 .TAU application => coback1tau => MPI_Win_create() 0.0 0.945 0.945 7.2 0 131 MPI_Barrier() 0.0 0.569 0.569 4.5 0 126 .TAU application => coback1tau => MPI_Barrier() 0.0 0.501 0.501 6.3 0 80 .TAU application => MPI_Win_free() 0.0 0.501 0.501 6.3 0 80 MPI_Win_free() 0.0 0.481 0.481 3.6 0 134 .TAU application => MPI_Win_create() 0.0 0.376 0.376 2.7 0 139 .TAU application => MPI_Barrier() 0.0 0.0914 0.0914 0.9 0 102 .TAU application => MPI_Comm_dup() 0.0 0.0914 0.0914 0.9 0 102 MPI_Comm_dup() 0.0 0.0453 0.0453 26.1 0 2 MPI_Put() 0.0 0.0361 0.0361 25.6 0 1 .TAU application => MPI_Put() 0.0 0.0301 0.0301 6.3 0 5 MPI_Win_fence() 0.0 0.0283 0.0283 18.9 0 1 .TAU application => MPI_Win_lock() 0.0 0.0222 0.0222 0.9 0 25 .TAU application => MPI_Alloc_mem() 0.0 0.0222 0.0222 0.9 0 25 MPI_Alloc_mem() 0.0 0.0171 0.0171 3.6 0 5 .TAU application => coback1tau => MPI_Win_fence() 0.0 0.017 0.017 0.9 0 19 .TAU application => CI0000$coback1tau_$BLK 0.0 0.017 0.017 0.9 0 19 CI0000$coback1tau_$BLK 0.0 0.0167 0.0167 9 0 2 .TAU application => MPI_Get() 0.0 0.0155 0.0155 0.9 0 17 .TAU application => MPI_Comm_rank() 0.0 0.0155 0.0155 0.9 0 17 MPI_Comm_rank() 0.0 0.013 0.013 2.7 0 5 .TAU application => MPI_Win_fence() 0.0 0.01 0.01 0.9 0 11 .TAU application => MPI_Finalized() 0.0 0.01 0.01 0.9 0 11 MPI_Finalized() 0.0 0.0092 0.0092 0.5 0 18 .TAU application => coback1tau => MPI_Put() 0.0 0.005 0.005 0.9 0 6 .TAU application => MPI_Comm_size() 0.0 0.005 0.005 0.9 0 6 MPI_Comm_size()