|
|
|
# Introduction
|
|
|
|
# Overview
|
|
|
|
|
|
|
|
```mermaid
|
|
|
|
flowchart TD
|
|
|
|
start -->|1 - start trigger| eventHandler[Event Handler]
|
|
|
|
eventHandler -->|2 - create pod| Task-A
|
|
|
|
FA-A --> |3c - sends ready| eventHandler
|
|
|
|
eventHandler -->|4 - create pod| Task-B
|
|
|
|
FA-B --> |5a - asks if available| FA-A
|
|
|
|
FA-A --> |5b - establish socket| FA-B
|
|
|
|
FA-A --> |6a - finish trigger| eventHandler
|
|
|
|
FA-B --> |6d - sends ready| eventHandler
|
|
|
|
|
|
|
|
subgraph Task-A
|
|
|
|
UA-A[User app A] --> |2b - waits for inputs | UA-A
|
|
|
|
UA-A -->|3a - writes first packet| Out-A
|
|
|
|
Out-A --> |3b - detects first packet| FA-A[Forward Agent A]
|
|
|
|
end
|
|
|
|
|
|
|
|
subgraph Task-B
|
|
|
|
UA-B[User app B] --> |4b - waits for inputs | UA-B
|
|
|
|
UA-B --> |6b - writes first packet| Out-B
|
|
|
|
Out-B --> |6c - detects first packet| FA-B[Forward Agent B]
|
|
|
|
end
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
## Expected behavior
|
|
|
|
1. A trigger is sent to the `event-handler` to start the tasks workflow
|
|
|
|
2. The `event-handler` receives the `run` trigger with the workflow configuration (DAG-style with input and output nodes and objects) and creates the first pod/pods
|
|
|
|
- Each pod has two containers:
|
|
|
|
1. The `user-application` container: contains the user app that processes inputs and generates outputs
|
|
|
|
2. The `forwarding-agent` container: responsible of sending and receiving events from the event manager and establish the socket connections between task pods
|
|
|
|
- The `user-application` shall wait for all of its input objects to be `ready` to consume; then, it shall start writing the output to the appropriate sockets
|
|
|
|
3. The `forwarding-agent` shall communicate to the `event-handler` the `ready`ness of the `user-application` when the first packet of each output socket is written (can be)
|
|
|
|
4. For each object `ready`ness trigger received, the `event-handler` shall spawn the consumer tasks for that particular object (i.e. if `task-a` outputs `object-1` that is consumed by `task-b` and `task-c`, then the `event-handler` shall create both `task-b` and `task-c` pods, each one with its `user-application` and `forwarding-agent` containers)
|
|
|
|
- If a consumer task is already created due to other input objects, then it shall be skipped
|
|
|
|
- `event-handler` should keep track of the already created tasks
|
|
|
|
5. The newly created task's `forwarding-agent` shall inform its `forwarding-agent` providers of its availability (i.e. an API request):
|
|
|
|
- If reachable, each available `forwarding-agent` provider shall start a thread to establish a TCP socket with its consumer
|
|
|
|
- The providers shall only start sending the packages when all of its providers are `ready`
|
|
|
|
6. When a provider `forwarding-agent` has finished sending all of its objects to all of its consumers, it shall send a `finish` trigger to the `event-handler` to update the status of the workflow; then, that `forwarding-agent` shall exit and the Pod shall be `Completed`
|
|
|
|
7. Repeat the steps until all tasks are `Completed`
|
|
|
|
|
|
|
|
## Corner-cases
|
|
|
|
### Multiple chained dependencies including several level-ahead ancestors
|
|
|
|
Given the following scenario:
|
|
|
|
|
|
|
|
```mermaid
|
|
|
|
graph TD
|
|
|
|
A[Task A] --> B[Task B]
|
|
|
|
B --> D[Task D]
|
|
|
|
A --> C[Task C]
|
|
|
|
C --> D
|
|
|
|
A --> D
|
|
|
|
```
|
|
|
|
|
|
|
|
- `task-b`, `task-c` and `task-d` should receive the output objects from `task-a`, but only `task-b` and `task-c` would be able to run as `task-d` also depends on output from these two; thus, `task-d` would be created much earlier than when it will run
|
|
|
|
- Objects for `task-d` execution would have to be stored for a long time instead of immediately consumed
|
|
|
|
|
|
|
|
# Tools
|
|
|
|
## Argo Events
|
|
|
|
Home page: [LINK](https://argoproj.github.io/argo-events/) |
|
|
\ No newline at end of file |
