Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Register
  • Sign in
  • L llm.c - GPT2
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 4
    • Issues 4
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Terraform modules
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • tchatela
  • llm.c - GPT2
  • Wiki
  • Distributed Model

Distributed Model · Changes

Page history
Update Distributed Model authored Aug 08, 2024 by tchatela's avatar tchatela
Show whitespace changes
Inline Side-by-side
Distributed-Model.md
View page @ d7042e42
...@@ -9,12 +9,19 @@ Note that : ...@@ -9,12 +9,19 @@ Note that :
- The data sent for each of these operations are about 250 000 floats so about 1 Gb. - The data sent for each of these operations are about 250 000 floats so about 1 Gb.
- Accroding to [MN5 overview](https://www.bsc.es/supportkc/docs/MareNostrum5/overview), the transfer speed of a node is about 1Gb/s - Accroding to [MN5 overview](https://www.bsc.es/supportkc/docs/MareNostrum5/overview), the transfer speed of a node is about 1Gb/s
## Using 4 workers and 1 server
Using 8 workers and 1 server. Data transfer is done through tasks using high priority. **One worker is computing a full token sequence (64 tokens / worker)**
## Using 8 workers and 1 server
**One worker is computing half a token sequence (32 tokens / worker)**
Data transfer is done through tasks using high priority.
Dt = 2.5 seconds Dt = 2.5 seconds
![base-9-priority](uploads/7193fd34722329b1c593279365ff7ef1/base-9-priority.png) ![base-9-priority](uploads/7193fd34722329b1c593279365ff7ef1/base-9-priority.png)
Using 8 workers and 1 server. Data transfer is done through tasks using high priority. Data transfer is done through tasks using high priority.
Dt = 2.5 s Dt = 2.5 s
![all-nine](uploads/2d471a1e2cf8dc0bf30f957a4554d9e5/all-nine.png) ![all-nine](uploads/2d471a1e2cf8dc0bf30f957a4554d9e5/all-nine.png)
......
Clone repository

GPT2 Parallelization and Porting

  • Model Description
  • Runtime and Performances
  • Improvements
  • Traces
  • Fork Join Model
  • Task Based Model
  • Distributed Model