Prepared the physiboss_tnf_model_mpi for testing, some core files have also changed, most significantly Vincent added more variables to pack and unpack, there was a problem of intracellular_type which was not being packed, now in current code it is packed OUTSIDE maboss, later change this packing on intracellular_type to inside maboss (as Vincent suggest and this is the correct form).

This commit represents an attempt to run physiboss_cell_lines_mpi example from Vincent. Had to bring that example in and lots of other files which we "thought" were not important. For e.g., files in beta directory which help to set-up MaBoss (this is downloaded then installed when we "make" physiboss_cell_lines_mpi project). Thus, execute on MN0 node (has outside internet connection). Doing "make clean" removes MaBoss and it needs to be downloded again with "make". The example does not crash but produces identical SVG files. Thus, committing and requesting Vincent to check the example.

This is the first commit when making PhysiCell-X comptaible with PhysiCell v1.9.0. The cell-cell atachment facility is not implemented.
Tested the predator-prey model - working fine.
To test MaBoss will need an example.
Remember NetworkSimulator state; object has become private so it might cause problems while packing/unpacking.
There are some functions which are not in PhysiCell v1.9.0 but are in physiboss-dev (might be needed for packing/unpacking) so be very careful.
As of now there is only one type of intracellular type i.e. PhysiBoss

Predator-Prey example is working properly.
Have not checked Predator-Prey-Farmer example.
Many changes have been done (Big commit).
This is the last commit on the development branch before I start incorporating changes of v1.9.

Merging the development branch with the master branch, fixed conflicts in /modules/PhysiCell_settings.cpp (only place where it showed a conflict).

Multiple changes have been made since last commit : possibly incorporation of v1.7 serial PhysiCell into PhysiCell-X, difficult to examine each change
but the local development branch that I am commiting was working for Predator-Prey example. For heterogeneity example, MAYBE the syntax of cell_create() in the unpack() function has to be made the same as the one in create_tissue() function in custom.cpp.

This is the last commit to the development branch BEFORE I start incorporating changes in v1.8 for cell-cell-adhesion and in general.

(1) This is the current working version of Predator-Prey
(2) In the unpack() function, I am now using pCell = create_cell(get_cell_definition(temp_str),cell_ID); to create a cell AFTER I unpack the ID, position and the type of the cell. The 'type' is used by get_cell_definition(temp_str).
(3) pCell = create_cell(get_cell_definition(temp_str),cell_ID); is needed in the unpack() function because we are using the same call in the setup_tissue() function of the custom.cpp in sample_projects/template.
(4) For the heterogeneity-sample example there is a POSSIBILITY that it might not work with pCell = create_cell(get_cell_definition(temp_str),cell_ID); so replace the call in unpack() by a simple create_cell(ID,...) i.e. no need for get_cell_definition().
(5) Now, the XML file is being written by a single MPI process i.e. root process - so this will work in parallel settings (just that only a single process will write the XML file)
(6) The MATLAB file writing is parallelized now (Both for BioFVM MATLAB and PhysiCell MATLAB).
(7) I have removed the statement pReference_live_phenotype = &phenotype; from unpack() function [as Paul Macklin says these pointers are set automatically].
(8) To check correctness of Cell Transmission, one CELLS_RANK_X file is written per rank (this can be disabled by commenting out the print_cell() function from packing(), unpacking() functions.

In function bool load_PhysiCell_config_file( std::string filename, mpi_Environment &world ), only the root process was calling the setup_microenvironment_from_XML() function. This is wrong. All processes must call this function but only the root process should print the statements. This has been corrected on the development branch as well.

(1) In PhysiCell_settings.cpp, in function bool load_PhysiCell_config_file( std::string filename, mpi_Environment &world ), only the root process was calling the setup_microenvironemnt_from_XML() function. This was a mistake. All processes must call this function but only the root process should print the cout statements.
(2) Added -g flag to template Makefile.
(3) Changed script_project.sh

Makefiles for "template" have been altered to make $make template & $make reset work. Both (1) $make heterogeneity-sample (for parallel execution) and $make template (for serial execution) are working. This is the correct point for git commit.

Large commit, work from 10th Dec to 15th Dec 2020 is being commit (inclusive of these dates), for details see Work_Log on https://docs.google.com/document/d/1YMGO_RXxZ3ZRqGgN_uDNx3UQy6bxUiOMZB1pmsGONwM/edit . Overall
(1) I have incorporated all (hopefully) the changes from v1.7 to v1.6 so that the serial version of Predator-Prey can run.
(2) I have NOT tested the Predator-Prey in v1.6 as yet. This commit is just before this stage.
(3) The parallel version of heterogeneity-sample is working and producing the expected output.

Multiple changes done on 7th and 9th December 2020 are incorporated in this commit (see Work_Log on Google Documents).

This is the first commit towards changing PhysiCell_X for the Predator-Prey example. Multiple changes have been made and tested thatheterogeneity-sample parallelized version works with the code till now. All changes listed at: https://docs.google.com/document/d/1YMGO_RXxZ3ZRqGgN_uDNx3UQy6bxUiOMZB1pmsGONwM/edit?usp=sharing from 1st Dec 2020 to 4th Dec 2020 (including both the dates).

Changed all SVG_plot calls to parallel calls i.e. with world and cart_topo object (checked its working)

Step 1 - Clone the repository
Step 2 - Execute $make heterogeneity-sample to check out correct files into the main directories to prepare the parallel version of the heterogeneity 3D program
Step 3 - $make to compile
Step 4 - A copy of script_hetero.sh is present in the main directory for sample execution, hence $ sbatch script_hetero.sh
Step 5 - After program runs ... $make clean to remove executable + .o files then
Step 6 - $make reset to restore the initial state of PhysiCell.

(1) This commit removes DistPhy as a separate repository. (2) Deleted the .git and .gitignore inside DistPhy (3)From GS_PhysiCell_X did git git rm --cached DistPhy then did git add DistPhy and then commit.

This commit contains (1) SVG parallelization - this functionality is fully parallelized but I have used the parallel SVG function for plotting the initial.svg and not the svg generation at other steps and the final step. Simply use the parallel SVG function for the intermediate SVG generation and final file generation. (2) As Migule informed that the .xml file was not present in the config folders - I have now removed *.xml from .gitignore and added output/*.xml in .gitignore to just ignore the XML files (and output/*.svg and output/*.mat files also) in the output directory. The XML, SVG and MAT files in other directories will be written. This commit is the major commit BEFORE i start resetting the project. In case of problems, fall back to this commit.

1. Created 2 lists that contain the global mesh index of all the neighbouring voxels of all the sub-domain boundary voxels, called them moore_connected_voxel_global_indices_left/right.
2. Packed some voxel-based, some cell-based (majorly geometry & mechanics) information to be exchanged.
3. Exchanging the above information.
4. Unpack the above information in a class named Moore_Voxel_Info which in addition to the Voxel information contains a Moore_Cell_Info class vector containg the packed information above. Further, associated the global mesh index with a Moore_Voxel_Info object using std::unordered_map<int,Moore_Voxel_Info> to reduce the time taken to search the Voxel+Cell Info associated with a global mesh index.Every boundary voxel stores its global voxel neighbours in the moore_list and this global mesh index becomes a key to search the correct Moore_Voxel_Indo object which stores the needed information.

(1) When cells divide, parent OR daughter or both maybe pushed out of the subdomain. I have added the functionality to bring them back to the sub-domain (as cell transmission is not implemented for movement during Cell DIVISION)
(2) Since PhysiCell implements deletion of cells by exchanging the cell with the last cell in the "all_cells" list, in case both the current cell AND the last cell is to be deleted, this becomes a problems as the last cell comes into the position of the current cell and the loop counter is incremented ! So whenever we delete a cell, we can decrement the loop counter by 1 so that it again comes to examine the last cell which is in the current position. This solved the problem of cell deletion.

(a) Cell Packing
(b) Cell sending/receving
(c) Cell deletion from old subdomain
(d) Cell creation in new subdomain
(e) Need to now start with Unpacking ...

Basically now we have (1) capabiity to detect if a cell leaves subdomain (2) Functionality to pack such cells using a single buffer i.e. one buffer contains data of all cells crossing the subdomain. (3) MPI_Isend and MPI_Irecv for sending and receiving this buffer in the other process. (4) The cell that is sent is then deleted from the process. (5) As of now the functionlality of unpacking is only dummy i.e. only unpacks the incoming cell ID to check if we are receiving the cell. The unpack facility will take a long time to write and then we have to consider the pointer fields.

I had not commited for a long time and have made many changes since the last commit.
I have parallelized cell division and some part after it.
Updating cell position is still not parallelized.
Have completed writing the print function.

1. I am parallelizing divide() cells first.
2. Each process knows how many cells in it will divide.
3. When 'x' cells divide 'x' more cells are created.
4. These new 'x' cells must have unique IDs.
5. The root process has the duty of allocating unique IDs.
6. These processes contact root process and request 'x' number of IDs.
7. Root process knows which was the last ID used, and gives a range of continuous IDs to processes in sequence.
8. Root process then increases the current ID to the last ID which was allocated.

1. The main.cpp has commented out the un-necessary XML, full data plotting. The SVG plots are working with individual ranks.
2. This commit is exactly the same as the last commit on master branch except for commenting out in main.cpp (see above).
3. This commit is the first commit BEFORE beginning parallelization of the update_all_cells function.

(1) the main function calls simulate_diffusion_decay() function.
(2) This function had a serial prototype which passed only the time dt.
(3) Created a new prototype and version for this function which accepts mpi_Environment &world, mpi_Cartesian &cart_topo objects
(4) simulate_diffusion_decay() then calls the parallel solver using a function pointer called diffusion_decay_solver - this is used at a lot of places and hence created another function pointer: diffusion_decay_solver_mpi that points to the new parallel solver 3-D function.
(5) This new function pointer must be set at some places - have tried to check that is done at all places but can later again check.
The program is compiling.

1. Wrote DistPhy_Collective.cpp & .h that contain pure collective MPI functions only.
2. These functions are wrapped inside functions like distribute_global_sum() etc.
3. Not sure if git is going to commit the two files that I wrote in DistPhy (so check)

Program is compiling.

(1) this function directly calls the mesh function
(2) we want the nearest_voxel_local_index(position, world, cart_topo) function to be called instead of this.
(3) replace call with underlying_mesh.nearest_voxel_local_index( position, world, cart_topo);
since the underlying_mesh means the mechanical voxel mesh, it automatically provides particulars of that mesh.
The program is compiling.

(1) parallelizing the call assign_position()
(2) Now it takes 3 cooridnates and mpi_Environment, mpi_Cartesian objects
(3) assign_position() is in PhysiCell_cell.cpp
(4) this calls update_voxel(world, cart_topo)
(5) update_voxel() is in BioFVM_basic_agent.cpp
(6) this calls a new parallel function nearest_voxel_local_index()
(7) this function is in BioFVM_microenvironment.cpp but actually
(8) this calls a new function by the same name in BioFVM_cartesian_mesh.cpp
(9) this is the file in which this function is implemented and returns the
process specifc local voxel index which contains the basic agent.

Had updated the README  file onine so now needed to pull and then merge

(2)  Parallelizing setup_tissue() in heterogeneity.cpp - divided into 3 phases
	(a) 1st Phase: Create new Cells or Basic Agents - done
 	(b) 2nd Phase: Assign positions to Cells - doing
	(c) 3rd Phase: pCell->default[0], p_min, p_max data etc. --> global communications needed - To be done
(3) Program is compiling AFTER 1st phase completed and BEFORE 2nd Phase begins.
(4) A special syntax exists when calling Base class parametrized constructor from Derived class constructor.
Cell::Cell(int p_ID):Basic_Agent(p_ID) is the correct syntax. We CANNOT say Basic_Agent(p_ID); inside body of the constructor of Cell(int p_ID){ }.

custome_modules.
Extended the DistPhy_Utils.cpp to contain a function
distribute_cell_positions() function that distributes the cell
coordinates, cell IDs and the number of cell IDs.
Program is compiling but need to check the functioning through DDT.

commit is the first commit when I am changing the setup_tissue()
function.
I have written two new files in DistPhy framework: DistPhy_Utils.cpp and
corresponding .h file. The class mpi_CellPositions function : void mpi_CellPositions::positions_to_rank_list(std::vector<std::vector<double>> &generated_list, int xmin, int xmax, int ymin, int ymax, int zmin, int zmax, int dx, int dy, int dz, mpi_Environment &world, mpi_Cartesian &cart_topo, int strt_cell_ID )
takes as input a list of position coordinates in 3-D space and
determines as to which MPI process rank would create the cell at that
position. For each rank we have : cell_coords_all_procs[rank] list which
contains all the coordinate positions.
Then for each rank we have cell_IDs_all_procs[rank] which contains ID of
the cells that that "rank" would be handling.
We also have a count of the IDs in no_of_IDs_all_procs[rank] for a
particular rank.
Added a "static" variable called max_ID_in_parallel to the Basic_Agent
class that can be accessed using get_max_ID_in_parallel() and
set_max_ID_in_parallel(). Obtain this ID BEFORE creating positions for
cells and set this ID AFTER creating the positions.

Cell_Container* cell_container = create_cell_container_for_microenvironment( microenvironment, mechanics_voxel_size );
in the main() function.
Ultimately it calls the mesh.resize( ) function which takes care of
domain partitioning w.r.t to the mechanical voxel size (=30). So there
are two meshes (and most probably its the mechanical mesh that is being
plotted along with cells. That is the reason why it produces a complete
image of cells because *till now* cell data was not partitioned among
processes. *Now* it should produce only a partial image (hopefully).
Check through DDT if the right parallel version of functions are being
called now starting with Cell_Container* cell_container = create_cell_container_for_microenvironment( microenvironment, mechanics_voxel_size, world, cart_topo );

function. Then it prints the microenvironment information but it calls a
display_information function in BioFVM_Mesh.cpp which prints the number
of voxels. NOW this number of voxels is the number of local voxels so I
have added the word "per-process" over there ! Compiled sucessfully and
ran (some output but very importantly no error).

versions of functions from initialize_microenvironment(). Now calling
parallel versions of functions - compiling successfully.
*can also revert back to this version without problems
*now need to check Dirichlet conditions setting and then
*integrate the solver code.

Integrated code for function: void
Cartesian::resize(x1,x2,y1,y2,z1,z2,dx,dy,dz,world,cart_topo) where
domain partitioning takes place. Adjacency lists of voxels in all three
directions are also made here. Compiled sucessfully but I have not run
it as data should now be on different processes and without the partial
parallel solver, there is not much point in executing it. Now next step
will be to go up the stack to the outer functions which take care of
dirichlet conditions. In the stand-alone version of BioFVM, the
dirichlet conditions were not set correctly for 3-D but in this BioFVM
version integrated with PhysiCell V1.1.6, they should be ok (but still
check if Dirichlet conditions being set properly).

Cartesian_Mesh::resize(...) the parallel version in BioFVM_mesh.cpp.
So I am doing a commit at this point that just in case something goes
wrong, I should revert back to this commit.