Task-Based-Model.md

@@ -4,11 +4,10 @@ We want to go from a for join model to a task based one. ![backward_transformer_
 
 Basically, we would like to avoid the black regions (sequential portions) that we can see in these two traces. Thus, creating a task based model will allow us to overlap the computation of 2 distinct layers, avoiding any CPU inactivity.
 
-# Analysis of the for join model current state
+# Data dependency
 
 First, we need to analyze what parts of the model can be overlapped. Therefore, we will inspect each layer and check for their data dependencies. Then, depending on these dependencies, we will create a data flow diagram that will show the different tasks and how each one depends on a previous one.
 
-## Data dependency
 
 On a general point of view, each layer of the forward needs to have its weights and bias updated before being computed.
 
@@ -20,4 +19,17 @@ With this general / too wide view of the model, it does not seems like we will b
 
 During a forward pass, nearly each of the calculations made are channel-independent, as there is only the attention layer that is using multiple channels a one calculation. Therefore, we can set up tasks with k - channels as an input data, k a divider of T in \[1, T\], allowing us to overlap two different layers using different sets of k-channels. However, the attention layer is using all the tokens of a sentence for its computation. Thus, it is mandatory to wait for all the k-channels of a sentence before entering this layer (this is also why we choose k as a divider of T in \[1, T\]). But as the attention layer is only computing one sentence at once, we can still compute k-channels sets from other sentences at the same time as the attention layer is running.
 
-During a backward pass, the principle of k-channels is a bit different, as the weights and biases do not have a shape depending of T. Basically, we can see this as k=1. However this is not problem
\ No newline at end of file
+During a backward pass, the principle of k-channels is a bit different, as the weights and biases do not have a shape depending of T. Basically, we can see this as k=1. However this is not problem as we still need to compute the data arrays for the next backward layer. Those arrays do have a shape depending of T, thus making it possible to use the notion of k-channels, which will be related to the output data (called dinp) and not to the weights or biases. We will then be able to overlap backward layers likewise the forward layers, with the same exception for the attention layer.
+
+Moreover, the aim of the backward layer is to compute the gradients, which will then be added to the weights and biases of each layer parameters. In the code, it is done this way :
+1. Forward pass
+2. Set all gradients values to 0
+3. Backward pass to update gradients
+4. Update each forward layer's parameters using the gradients
+5. New forward pass
+
+The particularity of this given order is that we don't have to fully complete a step before going to the next one. In fact, the gradients values can be set to 0 just before computing there new values during there related backward layer. This way, we can, for example, overlap a backward layer will setting the gradients of the next backward layer to 0. Also, this same idea works for the update of the forward layers' parameters, as we can update the weights of one layer during the its previous forward layer.
+However, keep in mind that the backward pass is the forward pass but mirrored, so the first backward layers are making data dependencies (the update of the weights) towards the last forward layer.
+
+
+With everything that has been stated, we can now create the following data flow diagram : 
\ No newline at end of file