readme.txt 13.4 KB
Newer Older
gsaxena's avatar
gsaxena committed
1
####################################
gsaxena's avatar
gsaxena committed
2
BioFVM_X is based on BioFVM v 1.1.6. 
gsaxena's avatar
gsaxena committed
3
####################################
gsaxena's avatar
gsaxena committed
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

+ BioFVM (http://BioFVM.MathCancer.org) supports only on-node shared memory 
  parallelism using OpenMP.

+ BioFVM_X adds MPI to BioFVM to parallelize the core kernels of BioFVM using  
  MPI.

+ Some core kernels are: Domain Partitioning, Basic Agent Generation, Gaussian 
  Profile building, File Writing, Direct Solver (Thomas).
  
+ Most of the MPI-based parallel functions are made available in 
  BioFVM_parallel.cpp
  
+ A Makefile is given and a sample example in ./examples/tutorial1_BioFVM.cpp 
  is completely parallelized using MPI + OpenMP. 
  
+ Procedure to run:
gsaxena's avatar
gsaxena committed
21 22 23 24 25 26
  (1) git clone the repository
  (2) Change to the top level directory i.e. $ cd GS_BioFVM_X/GS_BioFVM
  (3) Make sure GCC 8.1 (or other version) and OpenMPI 3.1.1 (or other version) 
      present (or the modules are loaded in a module environment)
  (4) Execute $ make tutorial1
      This results in an executable file "tutorial1" in the directory ./examples
gsaxena's avatar
gsaxena committed
27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
      
      Example output:
      
bsc99102@login1:~/GS_BioFVM_X/GS_BioFVM> make tutorial1
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c BioFVM_vector.cpp 
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c BioFVM_matlab.cpp
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c BioFVM_utilities.cpp 
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c BioFVM_mesh.cpp 
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c BioFVM_microenvironment.cpp 
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c BioFVM_solvers.cpp 
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c BioFVM_basic_agent.cpp 
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c BioFVM_agent_container.cpp 
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c BioFVM_MultiCellDS.cpp
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c BioFVM_parallel.cpp
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -c pugixml.cpp
mpic++ -march=native  -O2 -fomit-frame-pointer -mfpmath=both -fopenmp -m64 -std=c++11 -g  -o ./examples/tutorial1 ./examples/tutorial1_BioFVM.cpp BioFVM_vector.o BioFVM_matlab.o BioFVM_utilities.o BioFVM_mesh.o BioFVM_microenvironment.o BioFVM_solvers.o BioFVM_basic_agent.o BioFVM_agent_container.o BioFVM_MultiCellDS.o BioFVM_parallel.o  pugixml.o
bsc99102@login1:~/GS_BioFVM_X/GS_BioFVM>
gsaxena's avatar
gsaxena committed
44 45 46 47 48 49 50 51 52 53

Check if the executable has been made:

bsc99102@login1:~/GS_BioFVM_X/GS_BioFVM> ls -lrt ./examples | grep tutorial1
-rw-r--r-- 1 bsc99102 bsc99     420 Sep  4  2019 tutorial1_BioFVM.h
-rw-r--r-- 1 bsc99102 bsc99   34502 Feb 14 13:46 tutorial1_BioFVM.cpp
-rwxr-xr-x 1 bsc99102 bsc99 6991728 Apr 16 16:03 tutorial1

  (5)  For the SLURM scheduler, the following script can be used 
  (script_tuturial1.sh):
gsaxena's avatar
gsaxena committed
54
 
gsaxena's avatar
gsaxena committed
55
============================================================================
gsaxena's avatar
gsaxena committed
56 57 58 59 60 61 62 63 64 65 66 67 68 69
 #!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=24
#SBATCH -t 00:15:00
#SBATCH -o output-%j
#SBATCH -e error-%j
#SBATCH --exclusive

export OMP_DISPLAY_ENV=true
export OMP_NUM_THREADS=24
export OMP_PROC_BIND=spread
export OMP_PLACES=threads
mpiexec --map-by ppr:1:socket:pe=24  --report-bindings ./examples/tutorial1
gsaxena's avatar
gsaxena committed
70
============================================================================
gsaxena's avatar
gsaxena committed
71

gsaxena's avatar
gsaxena committed
72
This script implies :
gsaxena's avatar
gsaxena committed
73

gsaxena's avatar
gsaxena committed
74 75
(a) uses 1 single node 
(b) spawns 2 MPI processes in that node (Since we have 2 sockets per node, we 
gsaxena's avatar
gsaxena committed
76
    create 2 MPI processes i.e. 1 MPI process per socket).
gsaxena's avatar
gsaxena committed
77
(c) create 24 threads per MPI process  (Since we have 24 cores in 1 socket,
gsaxena's avatar
gsaxena committed
78
    we create 24 threads i.e. 1 thread per core).
gsaxena's avatar
gsaxena committed
79

gsaxena's avatar
gsaxena committed
80 81 82
IMPORTANT: 

(a) Please make sure that the dimension of the Physical domain 
gsaxena's avatar
gsaxena committed
83
(specified in ./examples/tutorial1_BioFVM.cpp) 
gsaxena's avatar
gsaxena committed
84
is completely divisible by the Voxel dimensions (given by 
gsaxena's avatar
gsaxena committed
85
'mesh_resolution' in ./examples/tutorial1_BioFVM.cpp).
gsaxena's avatar
gsaxena committed
86 87 88 89 90 91 92 93 94

(b) Please make sure that the number of Voxels in the X direction are perfectly 
divisible by the total number of processes. 

For the given example: N=1000, minX=0, maxX=N, mesh_resolution=10. This gives 
total Voxels in X (or Y or Z) direction as (maxX-minX)/mesh_resolution = 
(1000 - 0)/10 = 100 Voxels. Now the total number of Voxels in the X-direction 
must be perfectly divisible by total number of MPI processes (in this case P=2).
Thus, we have 100/2 = 50 Voxels per process. 
gsaxena's avatar
gsaxena committed
95

gsaxena's avatar
gsaxena committed
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154
(6) Submit the batch script using: $sbatch script_tutorial1.sh

(7) After the simulation completes, two text files are produced. One is the
error file that contains the bindings of MPI processes and threads:

For example:
========================================================================================================================================================
[s05r2b10:340036] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], socket 0[core 7[hwt 0]], socket 0[core 8[hwt 0]], socket 0[core 9[hwt 0]], socket 0[core 10[hwt 0]], socket 0[core 11[hwt 0]], socket 0[core 12[hwt 0]], socket 0[core 13[hwt 0]], socket 0[core 14[hwt 0]], socket 0[core 15[hwt 0]], socket 0[core 16[hwt 0]], socket 0[core 17[hwt 0]], socket 0[core 18[hwt 0]], socket 0[core 19[hwt 0]], socket 0[core 20[hwt 0]], socket 0[core 21[hwt 0]], socket 0[core 22[hwt 0]], socket 0[core 23[hwt 0]]: [B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B][./././././././././././././././././././././././.]

[s05r2b10:340036] MCW rank 1 bound to socket 1[core 24[hwt 0]], socket 1[core 25[hwt 0]], socket 1[core 26[hwt 0]], socket 1[core 27[hwt 0]], socket 1[core 28[hwt 0]], socket 1[core 29[hwt 0]], socket 1[core 30[hwt 0]], socket 1[core 31[hwt 0]], socket 1[core 32[hwt 0]], socket 1[core 33[hwt 0]], socket 1[core 34[hwt 0]], socket 1[core 35[hwt 0]], socket 1[core 36[hwt 0]], socket 1[core 37[hwt 0]], socket 1[core 38[hwt 0]], socket 1[core 39[hwt 0]], socket 1[core 40[hwt 0]], socket 1[core 41[hwt 0]], socket 1[core 42[hwt 0]], socket 1[core 43[hwt 0]], socket 1[core 44[hwt 0]], socket 1[core 45[hwt 0]], socket 1[core 46[hwt 0]], socket 1[core 47[hwt 0]]: [./././././././././././././././././././././././.][B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B/B]

OPENMP DISPLAY ENVIRONMENT BEGIN
  _OPENMP = '201511'
  OMP_DYNAMIC = 'FALSE'
  OMP_NESTED = 'FALSE'
  OMP_NUM_THREADS = '24'
  OMP_SCHEDULE = 'DYNAMIC'
  OMP_PROC_BIND = 'SPREAD'
  OMP_PLACES = '{0},{1},{2},{3},{4},{5},{6},{7},{8},{9},{10},{11},{12},{13},{14},{15},{16},{17},{18},{19},{20},{21},{22},{23}'
  OMP_STACKSIZE = '0'
  OMP_WAIT_POLICY = 'PASSIVE'
  OMP_THREAD_LIMIT = '4294967295'
  OMP_MAX_ACTIVE_LEVELS = '2147483647'
  OMP_CANCELLATION = 'FALSE'
  OMP_DEFAULT_DEVICE = '0'
  OMP_MAX_TASK_PRIORITY = '0'
OPENMP DISPLAY ENVIRONMENT END

OPENMP DISPLAY ENVIRONMENT BEGIN
  _OPENMP = '201511'
  OMP_DYNAMIC = 'FALSE'
  OMP_NESTED = 'FALSE'
  OMP_NUM_THREADS = '24'
  OMP_SCHEDULE = 'DYNAMIC'
  OMP_PROC_BIND = 'SPREAD'
  OMP_PLACES = '{24},{25},{26},{27},{28},{29},{30},{31},{32},{33},{34},{35},{36},{37},{38},{39},{40},{41},{42},{43},{44},{45},{46},{47}'
  OMP_STACKSIZE = '0'
  OMP_WAIT_POLICY = 'PASSIVE'
  OMP_THREAD_LIMIT = '4294967295'
  OMP_MAX_ACTIVE_LEVELS = '2147483647'
  OMP_CANCELLATION = 'FALSE'
  OMP_DEFAULT_DEVICE = '0'
  OMP_MAX_TASK_PRIORITY = '0'
OPENMP DISPLAY ENVIRONMENT END
========================================================================================================================================================

and the second file is that of the output.

For example:
========================================================
TIME FOR RESIZING MICROENVIRONMENT = 1.0491
TIME FOR GENERATING GAUSSIAN PROFILE = 0.011721
TIME FOR WRITING INITIAL CONCENTRATION FILE = 0.154976
TIME FOR CREATING ALL BASIC AGENTS = 0.000926388
TIME FOR SIMULATING (SOURCES+SINKS+DIFFUSION) = 2.60654
TIME FOR WRITING FINAL FILE = 0.170324
TOTAL PROGRAM EXECUTION TIME = 3.99776
========================================================

gsaxena's avatar
gsaxena committed
155 156 157
####################################################
BioFVM: Finite Volume Solver for Biological Problems.
####################################################
gsaxena's avatar
gsaxena committed
158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293

Version:      1.1.6
Release date: 19 August 2017

Homepage:     http://BioFVM.MathCancer.org
Downloads:    http://BioFVM.sf.net

Summary:  
This update includes small bug fixes and enhancements for 
compatibiltiy with PhysiCell. It is focused on improving 
its handling of default settings and Dirichlet conditions. 

New features:
+ Added ability to enable / disable Dirichlet conditions for 
  individual substrates. Use: 
  
  void Microenvironment::set_substrate_dirichlet_activation(
       int substrate_index , bool new_value )
  
  to set substrate with index substrate_index to apply or 
  not apply Dirichlet conditions, for all defined Dirichlet 
  nodes. (e.g., you want to set Dirichlet conditions for the
  first substrate, but not teh others). 
  
+ Added Boolean new use_oxygen_as_first_field to Microenvironment_Options,
  so that initialize_microenvironment() only uses these if the flag 
  is set to true. By default, it is set to true. 
  
+ Calling "Microenvironment.set_density()" or 
  "Microenvironment.add_density()" now sets: 
  
  default_microenvironment_options.use_oxygen_as_first_field = false
  
  so that calling initialize_microenvironment() after defining fields 
  does not overwrite prior user choices. 
  
  These functions also correctly resize 
  
  default_microenvironment_options.Dirichlet_condition_vector and 
  default_microenvironment_options.Dirichlet_activation_vector
  
+ Updated all the add/update_density functions in the Mircoenvironment,
  so that adding a new density sets   
  
+ Updated Microenvironment::apply_dirichlet_conditions() so that 
  it only applies to the substrates j for which 
  
  microenvironment.dirichlet_activation_vector[j] == true 

Bugfixes: 
+ Fixed initialize_microenvironment() so that it now only applies Dirichlet 
  conditions if 
  
  default_microenvironment_options.outer_Dirichlet_conditions == true 
  
+ Updated the makefiles to use ARCH := native, which simplifies setup. 
  This generally has your compiler query your processor for available 
  hardware optimizations, so you don't have to choose this yourself 
  (e.g., ARCH := haswell). I suggest using a compiler at least as 
  recent as gcc/g++ 6.5.x   
  
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
  
Version:      1.1.5
Release date: 19 July 2017

Homepage:     http://BioFVM.MathCancer.org
Downloads:    http://BioFVM.sf.net

Summary:  
This update includes small bug fixes and enhancements for 
compatibiltiy with PhysiCell. 

New features:
+ Added Microenvironment::find_substrate_index( std::string ) to make it 
  easier to find . 
  
+ Added Basic_Agent::nearest_gradient( int substrate_index ) to directly 
  access the gradient of the indicated substrate at the agent's  
  current_voxel_index. 
  
+ Added Basic_Agent::nearest_gradient_vector( void ) to directly 
  access a vector of gradients (one for each substrate in the 
  microenvironment) at the agent's current_voxel_index. 
 
+ Added Microenvironment::is_dirichlet_node( int voxel_index ) to 
  easily check if the voxel is a Dirichlet node. 
  
+ Added new class Microenvironment_Options, with a default 
  default_microenvironment_options, to simplify Microenvironment
  setup. The defaults are dx = dy = dz = 20 microns, on 1 cubic mm. 
  
+ Added function initialize_microenvironment() to set up the 
  microenvironment using the options in 
  default_microenvironment_options. The code sets oxygen as the 
  default field, with D = 1e5 micron^2/min, and decay rate = 0.1 1/min
  (a 1 mm diffusion length scale). If 
  default_microenvironment_options.outer_Dirichlet_conditions = true, 
  then we set a 38 mmHg condition on the outer edges, corresponding to
  5% oxygenation (physioxic conditions). 

+ Updated the makefile to default to MARCH := native, which allows
  automated performance tuning without user editing of the Makefile.

Bugfixes: 
+ correct typos in citation information in all source files 

+ updated citation information 

+ added void set_default_microenvironment( Microenvironment* M ) declaration to 
  BioFVM_Microenvironment.h
 
+ set volume_is_changed = false in Basic_Agent::set_internal_uptake_constants(); 

+ Set the MultiCellDS options Booleans to extern bool in BioFVM_MultiCellDS.h 
  so that PhysiCell can read these options. 

+ Updated the simlified_data field in MultiCellDS XML output to include a 
  new "source" attribute with value "BioFVM".

+ Updated Microenvironment::update_dirichlet_node() and 
  Microenvironment::remove_dirichlet_node() to check against 
  mesh.voxels[].is_Dirichlet to provide a cheap and early exit 
  if the node isn't Dirichlet in the first place. 
  
+ Changed to a thread-safe data structure for Dirichlet nodes 
  (e.g., if a custom cell function in an OMP loop adds or removes 
  Dirichlet nodes). 
  
+ Fixed convergence_test3.cpp, convergence_test4_1.cpp, 
  convergence_test4_2.cpp, convergence_test5.cpp, 
  performance_test_numcells.cpp, tutorial1_BioFVM.cpp, 
  and main_experiment.cpp to use Basic_Agent::set_total_volume(). 

+ Fixed tutorial3_BioFVM.cpp to use dirichlet_one instead of 
  dirichlet_zero.