Orchestrating Knowledge/ML Workflows at Scale With Netflix Maestro | by Netflix Know-how Weblog

Scalability

Usability

Figure 1. Maestro high level architecture
Determine 1. Maestro excessive stage structure

Workflow Engine

Time-Primarily based Scheduling Service

Sign Service

Figure 2. Signal service high level architecture
Determine 2. Sign service excessive stage structure
Figure 3. Maestro’s scalable foreach design to support super large iterations
Determine 3. Maestro’s scalable foreach design to assist tremendous giant iterations

Consumer Interfaces

Figure 4. An example workflow defined by YAML, Python, and Java DSLs
Determine 4. An instance workflow outlined by YAML, Python, and Java DSLs

Parameterized Workflows

Workflow:
id: demo.pipeline
FROM_DATE: 20210101 #inclusive
TO_DATE: 20220101 #unique
jobs:
- job:
id: step1
kind: NoOp
'!dates': dateIntsBetween(FROM_DATE, TO_DATE, 1); #SEL expr
- foreach:
id: step2
params:
date: [email protected] #reference upstream step parameter
jobs:
- job:
id: backfill
kind: Pocket book
pocket book:
input_path: s3://path/to/pocket book.ipynb
arg1: $date #go the foreach parameter into pocket book
Figure 5. An example of using parameterized workflow to backfill data
Determine 5. An instance of utilizing parameterized workflow to backfill information

Execution Abstractions