Orchestrating Knowledge/ML Workflows at Scale With Netflix Maestro | by Netflix Know-how Weblog



Figure 1. Maestro high level architecture
Determine 1. Maestro excessive stage structure

Workflow Engine

Time-Primarily based Scheduling Service

Sign Service

Figure 2. Signal service high level architecture
Determine 2. Sign service excessive stage structure
Figure 3. Maestro’s scalable foreach design to support super large iterations
Determine 3. Maestro’s scalable foreach design to assist tremendous giant iterations

Consumer Interfaces

Figure 4. An example workflow defined by YAML, Python, and Java DSLs
Determine 4. An instance workflow outlined by YAML, Python, and Java DSLs

Parameterized Workflows

id: demo.pipeline
FROM_DATE: 20210101 #inclusive
TO_DATE: 20220101 #unique
- job:
id: step1
kind: NoOp
'!dates': dateIntsBetween(FROM_DATE, TO_DATE, 1); #SEL expr
- foreach:
id: step2
date: [email protected] #reference upstream step parameter
- job:
id: backfill
kind: Pocket book
pocket book:
input_path: s3://path/to/pocket book.ipynb
arg1: $date #go the foreach parameter into pocket book
Figure 5. An example of using parameterized workflow to backfill data
Determine 5. An instance of utilizing parameterized workflow to backfill information

Execution Abstractions