Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Register
  • Sign in
  • L llm.c - GPT2
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 4
    • Issues 4
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Terraform modules
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • tchatela
  • llm.c - GPT2
  • Wiki
  • Task Based Model

Task Based Model · Changes

Page history
Update Task Based Model authored Sep 20, 2024 by tchatela's avatar tchatela
Hide whitespace changes
Inline Side-by-side
Task-Based-Model.md
View page @ 1f1f4fc7
...@@ -37,4 +37,10 @@ With everything that has been stated, we can now create the following data flow ...@@ -37,4 +37,10 @@ With everything that has been stated, we can now create the following data flow
We will define two types of dependencies. The first one will be dependencies around tokens, to allow the overlapping of the different layers computation. The second one will allow the overlapping of the layers with the gradients reset and update phase. We will define two types of dependencies. The first one will be dependencies around tokens, to allow the overlapping of the different layers computation. The second one will allow the overlapping of the layers with the gradients reset and update phase.
For the dependencies around tokens, we create an array which we will use to define our dependencies. One element of this array defines the dependencies of one block of tokens. For the dependencies around tokens, we create an array which we will use to define our dependencies. One element of this array defines the dependencies of one block of tokens.
For the dependencies around the update and reset phase, we use the first element of each parameters, gradients, activation and gradients activation array slices. In fact, each of these arrays can be decomposed into subarrays, (buffers per kernels), which can, for some of them, be sliced again according if they are used inside a transformer block or not. We are using as dependencies the first element of each innermost slice. For the dependencies around the update and reset phase, we use the first element of each parameters, gradients, activation and gradients activation array slices. In fact, each of these arrays can be decomposed into subarrays, (buffers per kernels), which can, for some of them, be sliced again according if they are used inside a transformer block or not. We are using as dependencies the first element of each innermost slice.
Then, most of the task implementation will be composed of tasks defined around group of tokens, and you will find another taskloop inside of this task. Then, most of the task implementation will be composed of tasks defined around group of tokens, and you will find another taskloop inside of this task.
\ No newline at end of file
# Remove nested tasks
We would like to leverage taskiter to remove the overhead of task creation. However, as taskiter is not applying on nested task, we would like to remove them as most as we can. The current structure is :
`for (int token=0; token < T; token++) {
`
\ No newline at end of file
Clone repository

GPT2 Parallelization and Porting

  • Model Description
  • Runtime and Performances
  • Fork Join Model
  • Task Based Model
  • Distributed Model
  • Various Informations