CGPACK > FEB-2013
1-FEB-2013: more large array images made from coarrays.
The two examples below were run with 18 images as
allocate(coarray(10,10,10)[3,2,*])
making the third codimension equal to 3, and then with 64 images as
allocate(coarray(10,10,10)[4,4,*])
making the third codimension equal to 4. The sizes of the super arrays are thus 18 and 64 times the size of the individual coarrays respectively.
Importantly, the total number of images is set by FOR_COARRAY_NUM_IMAGES environment variable and codimensions along 1 and are read by the program via get_command_argument Fortran 2008 intrinsic. Thus the program does not need to be recompiled for another set of coarray dimensions!
![A super array obtained by joining 64 coarrays from 64 images arranged in a [4,4,4] grid.](64images.png)
![A super array obtained by joining 18 coarrays from 18 images arranged in a [3,2,3] grid.](18images.png)
5-FEB-2013: halo exchange between coarrays.
Halo exchange is a key element in many parallel algorithms. Typically, wherever the neighbourhood of a point is required for calculating properties at point, halo exchange will be required. This can be FE, MD, or, in our case, cellular automata (CA) calculations.
In our 3D CA implementation we assume the immediate 26 cell neighbourhood (3x3x3=27 minus the central cell =26). We arrange coarrays in a 3D grid [codim1,codim2,codim3], e.g. [4,4,4]. This means that cell neighbourhood might span multiple processors. A common solution is to define the coarrays with extents bigger by 2 in each direction. This then creates "halo" elements. Halo exchange is the process of copying "real" values from coarrays in the neighbouring images to the corresponding halo locations. Then the neighbourhood of any "real" cell is correctly defined.
In the example below we used 4x4x4=64 images. Each coarray is (10,10,10). Initially each coarray is assigned this_image(). Then new coarray is created with the difference between the original value and the average value for a 3x3x3 neighbourhood for each cell. This will make the internal values in each coarray zero, but in contrast, the values close to the boundaries will be non-zero, which can be easily visualised.
Below is a series of slices of the super array made from 64 local coarrays. The colour scheme is the same in all slices. Note how the boudary values change from one slice to another.
XY plane, position 10. One side of the boundary.
The images shown are 1 to 16, counting in standard
fortran "row major" format, i.e. first along X,
then along Y.
![Slice of the [4x4x4]=64 halo image super array, on XY plane at position 9.](halo64_xy9.png)
XY plane, position 11. The other side of the boundary.
![Slice of the [4x4x4]=64 halo image super array, on XY plane at position 10.](halo64_xy10.png)
XY plane, position 12. Passed the boundary, most cells are zero.
![Slice of the [4x4x4]=64 halo image super array, on XY plane at position 11.](halo64_xy11.png)
YZ plane, position 30. One side of the boundary.
Images shown are [3,:,:].
![Slice of the [4x4x4]=64 halo image super array, on YZ plane at position 30.](halo64_yz29.png)
YZ plane, position 31. The other side of the boundary.
Images shown are [4,:,:].
![Slice of the [4x4x4]=64 halo image super array, on YZ plane at position 31.](halo64_yz30.png)
ZX plane, position 9. Not at the boundary.
Images shown are [:,1,:].
![Slice of the [4x4x4]=64 halo image super array, on ZX plane at position 9.](halo64_zx8.png)
ZX plane, position 10. One side of the boudary.
Images shown are [:,1,:].
![Slice of the [4x4x4]=64 halo image super array, on ZX plane at position 10.](halo64_zx9.png)
ZX plane, position 11. The other side of the boudary.
Images shown are [:,2,:].
![Slice of the [4x4x4]=64 halo image super array, on ZX plane at position 11.](halo64_zx10.png)
15-FEB-2013: grain nucleation across "super" array, i.e. all image arrays put together.
This is a simple example of a [2x2x2] coarray grid. Initially array in each image is given -this_image(). So all neagative values in the image below just show arrays belonging to either of the eight images.
![Three orthogonal slices of the [2x2x2] super array showing grain nuclei.](nuc1.png)
Then 10 nuclei are scattered randomly across the whole or "super" array. These are the 10 positive labels on the key. The important aspect of this is that all nuclei labels (just a single integer for now) are unique. Hence only a single image is doing all the work. The three orthogonal slices are chosen to show the locations of the nuclei.
The nuclei appear somewhat diluted, smeared in this image. Not sure why. Probably some problem with Paraview's save screenshot filter. These are indeed single cells, i.e. isolated entries in the "super" array.