Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Register
  • Sign in
  • L llm.c - GPT2
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 4
    • Issues 4
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Terraform modules
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • tchatela
  • llm.c - GPT2
  • Wiki
  • Task Based Model

Task Based Model · Changes

Page history
Update Task Based Model authored Jul 05, 2024 by tchatela's avatar tchatela
Hide whitespace changes
Inline Side-by-side
Task-Based-Model.md
View page @ 840d59e1
# Why going towards a task based model
We want to go from a for join model to a task based one.
![backward_transformer_block](uploads/41e0ed558170fe7d13bd7efbf848bb57/backward_transformer_block.png)
\ No newline at end of file
![backward_transformer_block](uploads/41e0ed558170fe7d13bd7efbf848bb57/backward_transformer_block.png)
![Forward_pass](uploads/4e68b0a08e2153e3e885cd805fd1f8e3/Forward_pass.png)
Basically, we would like to avoid the black regions (sequential portions) that we can see in these two traces. Thus, creating a task based model will allow us to overlap the computation of 2 distinct layers, avoiding any CPU inactivity.
# Analysis of the for join model current state
First, we need to analyze what are the parts of the model that can be overlapped. Therefore, we will inspect each layer and check for their data dependencies. Then, depending on these dependencies, we will create a data flow diagram that will show the different tasks and how each one depends on a previous one.
## Data dependency
Clone repository

GPT2 Parallelization and Porting

  • Model Description
  • Runtime and Performances
  • Improvements
  • Traces