| ... | @@ -30,47 +30,11 @@ Moreover, the aim of the backward layer is to compute the gradients, which will |
... | @@ -30,47 +30,11 @@ Moreover, the aim of the backward layer is to compute the gradients, which will |
|
|
|
|
|
|
|
The particularity of this given order is that we don't have to fully complete a step before going to the next one. In fact, the gradients values can be set to 0 just before computing their new values during their related backward layer. This way, we can, for example, overlap a backward layer will setting the gradients of the next backward layer to 0. Also, this same idea works for the update of the forward layers' parameters, as we can update the weights of one layer during the its previous forward layer. However, keep in mind that the backward pass is the forward pass but mirrored, so the first backward layers are implying data dependencies (the update of the weights) towards the last forward layer.
|
|
The particularity of this given order is that we don't have to fully complete a step before going to the next one. In fact, the gradients values can be set to 0 just before computing their new values during their related backward layer. This way, we can, for example, overlap a backward layer will setting the gradients of the next backward layer to 0. Also, this same idea works for the update of the forward layers' parameters, as we can update the weights of one layer during the its previous forward layer. However, keep in mind that the backward pass is the forward pass but mirrored, so the first backward layers are implying data dependencies (the update of the weights) towards the last forward layer.
|
|
|
|
|
|
|
|
With everything that has been stated, we can now create the following data flow diagram : !
|
|
With everything that has been stated, we can now create the following data flow diagram : [GPT-2_task_based_model](uploads/0b71cb63f70b983d90c46001bcc29702/GPT-2_task_based_model.png)
|
|
|
|
|
|
|
|
# First implementation
|
|
# Nested task implementation
|
|
|
|
|
|
|
|
To begin with, we will put a taskwait between each call of the main training loop. However the rest of the program will be task-based, as shown by the following sketch:
|
|
We will define two types of dependencies. The first one will be dependencies around tokens, to allow the overlapping of the different layers computation. The second one will allow the overlapping of the layers with the gradients reset and update phase.
|
|
|
|
|
For the dependencies around tokens, we create an array which we will use to define our dependencies. One element of this array defines the dependencies of one block of tokens.
|
|
|
|
|
For the dependencies around the update and reset phase, we use the first element of each parameters, gradients, activation and gradients activation array slices. In fact, each of these arrays can be decomposed into subarrays, (buffers per kernels), which can, for some of them, be sliced again according if they are used inside a transformer block or not. We are using as dependencies the first element of each innermost slice.
|
|
|
|
|
Then, most of the task implementation will be composed of tasks defined around group of tokens, and you will find another taskloop inside of this task. |
|
|
Which gives us these paraver traces :
|
|
\ No newline at end of file |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This lead us to the following performances :
|
|
|
|
|
|
|
|
|
|
Here we tried to slice according to the token sequence only (BATCH_SUBSIZE=4), or according to the sequence token and the sequence batch (BATCH_SUBSIZE=1). The slope obtained for the second version seems better so we will keep it for the following tests.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Because we have a taskwait between each call of the main training loop, it is possible to get the runtime, speedup and efficiency for the forward and backward pass independently. The following diagrams give us the insight that the application is memory bound.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
We can compare now the efficiency between this version of the task based model and the fork join model
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# Final implementation
|
|
|
|
|
|
|
|
|
|
Now, we want to remove the taskloops put across the main training loop to improve tasks management and to allow the use of taskiter.
|
|
|
|
|
|
|
|
|
|
**_The implementation has changed, this diagram is no longer relevant_**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
 |
|
|
|
\ No newline at end of file |
|
|
