In create_point_sources() and create_point_sinks() in the main()

function, when we calculate the coordinates of the MPI process
corresponding to the (x,y,z) position of the Basic Agent, we were not
subtracting the initial starting coordinates of the physical domain from
the tempPoint[] array. This will not work when the physical domain
starts from a non-zero value. Thus added
tempPoint[0] = tempPoint[0] - lower_x_boundary_coordinate_of_domain;
similary for the y, z dimensions.
Have not compiled it but committing it.

BioFVM_parallel.cpp

@@ -192,7 +192,7 @@ void Cartesian_Mesh::resize( double x_start, double x_end, double y_start, doubl
 	
 	for( int i=0; i < connected_voxel_indices.size() ; i++ )
 	{ 
-        connected_voxel_indices[i].clear(); 
+        connected_voxel_indices[i].clear();             
     }
 	
     

HOW_TO_COMPILE

+Just do:
+
+$ make tutorial1
+
+If we change N in tutorial1.cpp, then only tutorial1.cpp is recompiled.
+
+When we do $ make clean, it completely removes all .o files and hence doing
+
+$ make tutorial1
+
+will again recompile ALL the .cpp files. 

error-7513445

-[s01r1b47:254716] *** Process received signal ***
-[s01r1b47:254716] Signal: Segmentation fault (11)
-[s01r1b47:254716] Signal code: Address not mapped (1)
-[s01r1b47:254716] Failing at address: 0xd1
-[s01r1b47:254716] [ 0] /lib64/libpthread.so.0(+0x10b10)[0x7f1531a87b10]
-[s01r1b47:254716] [ 1] ./examples/tutorial1[0x407ade]
-[s01r1b47:254716] [ 2] ./examples/tutorial1[0x405594]
-[s01r1b47:254716] [ 3] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f15316f46e5]
-[s01r1b47:254716] [ 4] ./examples/tutorial1[0x405cf9]
-[s01r1b47:254716] *** End of error message ***
---------------------------------------------------------------------------
-mpiexec noticed that process rank 0 with PID 254716 on node s01r1b47 exited on signal 11 (Segmentation fault).
---------------------------------------------------------------------------

examples/tutorial1_BioFVM.cpp

@@ -68,7 +68,7 @@ using namespace BioFVM;
 //int omp_num_threads = 48; // set number of threads for parallel computing, set this to # of CPU cores x 2 (for hyperthreading)
 
 
-#define N 3840
+#define N 1000
 double pi= 3.1415926535897932384626433832795;
 
 double UniformRandom()
@@ -134,6 +134,15 @@ void create_point_sources(double cell_radius, double dt, int num_sources, Microe
                 /* Use the floor() function, as type promotion was creating a problem in proc_x_coord expression  */
                 /*------------------------------------------------------------------------------------------------*/
                 
+                /*------------------------------------------------------------------------------------------------------------------------------------------------------*/
+                /* When the starting point of domain is not zero i.e. minX != 0, minY != 0 and minZ != 0, then we should subtract these starting points from tempPoints */
+                /* otherwise the strategy below fails                                                                                                                   */
+                /*------------------------------------------------------------------------------------------------------------------------------------------------------*/
+                
+                tempPoint[0] = tempPoint[0] - microenvironment.mesh.bounding_box[0];
+                tempPoint[1] = tempPoint[1] - microenvironment.mesh.bounding_box[1];
+                tempPoint[2] = tempPoint[2] - microenvironment.mesh.bounding_box[2];
+                
                 proc_y_coord = tempPoint[0]/(local_x_voxels * dx);                             //Can use floor() here but no need
                 proc_x_coord = (mpi_Dims[0] - 1) - floor(tempPoint[1]/(local_y_voxels * dy));  //Automatic promotion to double was creating problems
                 proc_z_coord = tempPoint[2]/(local_z_voxels * dz);                             //Can use floor() here but no need 
@@ -304,6 +313,15 @@ void create_point_sinks(double cell_radius, double dt, int num_sinks, Microenvir
                 /* MPI Process X direction is top to bottom                                                       */
                 /*------------------------------------------------------------------------------------------------*/
                 
+                /*------------------------------------------------------------------------------------------------------------------------------------------------------*/
+                /* When the starting point of domain is not zero i.e. minX != 0, minY != 0 and minZ != 0, then we should subtract these starting points from tempPoints */
+                /* otherwise the strategy below fails                                                                                                                   */
+                /*------------------------------------------------------------------------------------------------------------------------------------------------------*/
+                
+                tempPoint[0] = tempPoint[0] - microenvironment.mesh.bounding_box[0];
+                tempPoint[1] = tempPoint[1] - microenvironment.mesh.bounding_box[1];
+                tempPoint[2] = tempPoint[2] - microenvironment.mesh.bounding_box[2];
+                
                 proc_y_coord = tempPoint[0]/(local_x_voxels * dx);
                 proc_x_coord = (mpi_Dims[0] - 1) - floor(tempPoint[1]/(local_y_voxels * dy));
                 proc_z_coord = tempPoint[2]/(local_z_voxels * dz); 

output-7513445

-Total MPI processes = 8
-Thread level provided = 1
-Total Voxels = 1
-
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 0 microns^2 / minutes
-     decay rate: 0 minutes^-1
-     diffusion length scale: 0 microns
-
-
-
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 0 microns^2 / minutes
-     decay rate: 0 minutes^-1
-     diffusion length scale: 0 microns
-
-
-
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 0 microns^2 / minutes
-     decay rate: 0 minutes^-1
-     diffusion length scale: 0 microns
-
-
-
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 0 microns^2 / minutes
-     decay rate: 0 minutes^-1
-     diffusion length scale: 0 microns
-
-
-
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 0 microns^2 / minutes
-     decay rate: 0 minutes^-1
-     diffusion length scale: 0 microns
-
-
-
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 0 microns^2 / minutes
-     decay rate: 0 minutes^-1
-     diffusion length scale: 0 microns
-
-
-
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 0 microns^2 / minutes
-     decay rate: 0 minutes^-1
-     diffusion length scale: 0 microns
-
-
-
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 0 microns^2 / minutes
-     decay rate: 0 minutes^-1
-     diffusion length scale: 0 microns
-
-
-Total voxels are = 120000
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 1000 microns^2 / minutes
-     decay rate: 0.01 minutes^-1
-     diffusion length scale: 316.228 microns
-
-
-Total voxels are = 120000
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 1000 microns^2 / minutes
-     decay rate: 0.01 minutes^-1
-     diffusion length scale: 316.228 microns
-
-
-Total voxels are = 120000
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 1000 microns^2 / minutes
-     decay rate: 0.01 minutes^-1
-     diffusion length scale: 316.228 microns
-
-
-Total voxels are = 120000
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 1000 microns^2 / minutes
-     decay rate: 0.01 minutes^-1
-     diffusion length scale: 316.228 microns
-
-
-Total voxels are = 120000
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 1000 microns^2 / minutes
-     decay rate: 0.01 minutes^-1
-     diffusion length scale: 316.228 microns
-
-
-Total voxels are = 120000
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 1000 microns^2 / minutes
-     decay rate: 0.01 minutes^-1
-     diffusion length scale: 316.228 microns
-
-
-Total voxels are = 120000
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 1000 microns^2 / minutes
-     decay rate: 0.01 minutes^-1
-     diffusion length scale: 316.228 microns
-
-
-Total voxels are = 120000
-Microenvironment summary: substrate scale: 
-
-Mesh information: 
-type: uniform Cartesian
-Domain: [0,1000] microns x [0,1000] microns x [0,1000] microns
-   resolution: dx = 10 microns
-   voxels: 120000
-   voxel faces: 0
-   volume: 1e+09 cubic microns
-Densities: (1 total)
-   substrate1:
-     units: dimensionless
-     diffusion coefficient: 1000 microns^2 / minutes
-     decay rate: 0.01 minutes^-1
-     diffusion length scale: 316.228 microns
-
-
--------------------------------------------------------
-Primary job  terminated normally, but 1 process returned
-a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------

output-7584197

-TIME FOR RESIZING MICROENVIRONMENT = 0.758827
-TIME FOR GENERATING GAUSSIAN PROFILE = 0.00794553
-TIME FOR WRITING INITIAL CONCENTRATION FILE = 0.34451
-TIME FOR CREATING ALL BASIC AGENTS = 0.000949931
-TIME FOR SIMULATING (SOURCES+SINKS+DIFFUSION) = 8.48919
-TIME FOR WRITING FINAL FILE = 0.280464
-TOTAL PROGRAM EXECUTION TIME = 9.88661

script_ddt.sh

 #!/bin/bash
-#SBATCH --nodes=16
+#SBATCH --nodes=2
 #SBATCH --ntasks-per-node=2
 #SBATCH -t 08:00:00
 #SBATCH --cpus-per-task=1
@@ -9,6 +9,7 @@
 #SBATCH --exclusive
 
 # set application and parameters
-export OMP_NUM_THREADS=24
+export OMP_NUM_THREADS=1
 export OMP_PROC_BIND=spread
-ddt mpiexec --map-by socket --bind-to socket ./examples/tutorial1
+#ddt mpiexec --map-by socket --bind-to socket ./examples/tutorial1
+ddt srun ./examples/tutorial1