Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Register
  • Sign in
  • L llm.c - GPT2
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 4
    • Issues 4
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Terraform modules
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • tchatela
  • llm.c - GPT2
  • Wiki
  • Distributed Model

Distributed Model · Changes

Page history
Update Distributed Model authored Sep 18, 2024 by tchatela's avatar tchatela
Show whitespace changes
Inline Side-by-side
Distributed-Model.md
View page @ 489593d3
......@@ -142,21 +142,21 @@ With B=16, T=1024, 1 rank/socket
| Number of ranks | time/iteration | tokens/s | tokens/(s.cpus) |
| ------ | ------ | ------ | ------ |
| 1 | 67000 ms | | |
| 1 | 67925 ms | 241 | 4.3 |
| 2 | 31739 ms | 516 | 4.6 |
| 4 | 15084 ms | 1086 | 4.9 |
| 8 | 7624 ms | 2149 | 4.8 |
| 16 | 4084 ms | 4011 | 4.48 |
# Increasing the batch size per rank
# Increasing the batch size
4 ranks, 1 rank/socket, T=1024
| Batch size (B) | time/iteration | tokens/s | tokens/(s.cpus) |
| ------ | ------ | ------ | ------ |
| 4 | | | |
| 8 | | | |
| 16 | | | |
| 32 | | | |
| 64 | | | |
| Batch size (B) | Batch size/rank | time/iteration | tokens/s | tokens/(s.cpus) |
| ------ | ------ | ------ | ------ | ------ |
| 4 | 1 | 3984 ms | 1028 | 4.6 |
| 8 | 2 | 7421 ms | 1104 | 4.9 |
| 16 | 4 | 15084 ms | 1086 | 4.9 |
| 32 | 8 | | | |
| 64 | 16 | | | |
Clone repository

GPT2 Parallelization and Porting

  • Model Description
  • Runtime and Performances
  • Improvements
  • Traces
  • Fork Join Model
  • Task Based Model
  • Distributed Model