Orchestrating Knowledge/ML Workflows at Scale With Netflix Maestro | by Netflix Know-how Weblog

by Jun He, Akash Dwivedi, Natallia Dzenisenka, Snehal Chennuru, Praneeth Yenugutala, Pawan Dixit

At Netflix, Knowledge and Machine Studying (ML) pipelines are broadly used and have turn out to be central for the enterprise, representing various use instances that transcend suggestions, predictions and information transformations. A lot of batch workflows run day by day to serve numerous enterprise wants. These embrace ETL pipelines, ML mannequin coaching workflows, batch jobs, and so on. As Large information and ML grew to become extra prevalent and impactful, the scalability, reliability, and value of the orchestrating ecosystem have more and more turn out to be extra essential for our information scientists and the corporate.

On this weblog publish, we introduce and share learnings on Maestro, a workflow orchestrator that may schedule and handle workflows at a large scale.

Scalability and value are important to allow large-scale workflows and assist a variety of use instances. Our current orchestrator (Meson) has labored effectively for a number of years. It schedules round 70 hundreds of workflows and half one million jobs per day. Resulting from its recognition, the variety of workflows managed by the system has grown exponentially. We began seeing indicators of scale points, like:

• Slowness throughout peak visitors moments like 12 AM UTC, resulting in elevated operational burden. The scheduler on-call has to intently monitor the system throughout non-business hours.
• Meson was based mostly on a single chief structure with excessive availability. Because the utilization elevated, we needed to vertically scale the system to maintain up and have been approaching AWS occasion kind limits.

With the excessive development of workflows prior to now few years — rising at > 100% a yr, the necessity for a scalable information workflow orchestrator has turn out to be paramount for Netflix’s enterprise wants. After perusing the present panorama of workflow orchestrators, we determined to develop a subsequent technology system that may scale horizontally to unfold the roles throughout the cluster consisting of 100’s of nodes. It addresses the important thing challenges we face with Meson and achieves operational excellence.

Scalability

The orchestrator has to schedule lots of of hundreds of workflows, thousands and thousands of jobs on daily basis and function with a strict SLO of lower than 1 minute of scheduler launched delay even when there are spikes within the visitors. At Netflix, the height visitors load generally is a few orders of magnitude greater than the typical load. For instance, loads of our workflows are run round midnight UTC. Therefore, the system has to face up to bursts in visitors whereas nonetheless sustaining the SLO necessities. Moreover, we wish to have a single scheduler cluster to handle most of person workflows for operational and value causes.

One other dimension of scalability to think about is the scale of the workflow. Within the information area, it is not uncommon to have an excellent giant variety of jobs inside a single workflow. For instance, a workflow to backfill hourly information for the previous 5 years can result in 43800 jobs (24 * 365 * 5), every of which processes information for an hour. Equally, ML mannequin coaching workflows normally encompass tens of hundreds (and even thousands and thousands) of coaching jobs inside a single workflow. These large-scale workflows would possibly create hotspots and overwhelm the orchestrator and downstream methods. Due to this fact, the orchestrator has to handle a workflow consisting of lots of of hundreds of jobs in a performant method, which can be fairly difficult.

Usability

Netflix is a data-driven firm, the place key choices are pushed by information insights, from the pixel coloration used on the touchdown web page to the renewal of a TV-series. Knowledge scientists, engineers, non-engineers, and even content material producers all run their information pipelines to get the mandatory insights. Given the varied backgrounds, usability is a cornerstone of a profitable orchestrator at Netflix.

We want our customers to deal with their enterprise logic and let the orchestrator clear up cross-cutting considerations like scheduling, processing, error dealing with, safety and so on. It wants to offer completely different grains of abstractions for fixing comparable issues, high-level to cater to non-engineers and low-level for engineers to unravel their particular issues. It must also present all of the knobs for configuring their workflows to go well with their wants. As well as, it’s vital for the system to be debuggable and floor all of the errors for customers to troubleshoot, as they enhance the UX and cut back the operational burden.

Offering abstractions for the customers can be wanted to save lots of beneficial time on creating workflows and jobs. We would like customers to depend on shared templates and reuse their workflow definitions throughout their group, saving effort and time on creating the identical performance. Utilizing job templates throughout the corporate additionally helps with upgrades and fixes: when the change is made in a template it’s routinely up to date for all workflows that use it.

Nevertheless, usability is difficult as it’s usually opinionated. Completely different customers have completely different preferences and would possibly ask for various options. Typically, the customers would possibly ask for the other options or ask for some area of interest instances, which could not essentially be helpful for a broader viewers.

Maestro is the subsequent technology Knowledge Workflow Orchestration platform to fulfill the present and future wants of Netflix. It’s a general-purpose workflow orchestrator that gives a completely managed workflow-as-a-service (WAAS) to the information platform at Netflix. It serves hundreds of customers, together with information scientists, information engineers, machine studying engineers, software program engineers, content material producers, and enterprise analysts, for numerous use instances.

Maestro is extremely scalable and extensible to assist current and new use instances and gives enhanced usability to finish customers. Determine 1 exhibits the high-level structure.

In Maestro, a workflow is a DAG (Directed acyclic graph) of particular person models of job definition referred to as Steps. Steps can have dependencies, triggers, workflow parameters, metadata, step parameters, configurations, and branches (conditional or unconditional). On this weblog, we use step and job interchangeably. A workflow occasion is an execution of a workflow, equally, an execution of a step is known as a step occasion. Occasion information embrace the evaluated parameters and different data collected at runtime to offer completely different sorts of execution insights. The system consists of three principal micro providers which we are going to develop upon within the following sections.

Maestro ensures the enterprise logic is run in isolation. Maestro launches a unit of labor (a.ok.a. Steps) in a container and ensures the container is launched with the customers/purposes id. Launching with id ensures the work is launched on-behalf-of the person/software, the id is later utilized by the downstream methods to validate if an operation is allowed or not, for an instance person/software id is checked by the information warehouse to validate if a desk learn/write is allowed or not.

Workflow Engine

Workflow engine is the core part, which manages workflow definitions, the lifecycle of workflow cases, and step cases. It offers wealthy options to assist:

• Any legitimate DAG patterns
• In style information move constructs like sub workflow, foreach, conditional branching and so on.
• A number of failure modes to deal with step failures with completely different error retry insurance policies
• Versatile concurrency management to throttle the variety of executions at workflow/step stage
• Step templates for widespread job patterns like operating a Spark question or transferring information to Google sheets
• Assist parameter code injection utilizing personalized expression language
• Workflow definition and possession administration.
  Timeline together with all state adjustments and associated debug data.

We use Netflix open source project Conductor as a library to handle the workflow state machine in Maestro. It ensures to enqueue and dequeue every step outlined in a workflow with at the least as soon as assure.

Time-Primarily based Scheduling Service

Time-based scheduling service begins new workflow cases on the scheduled time laid out in workflow definitions. Customers can outline the schedule utilizing cron expression or utilizing periodic schedule templates like hourly, weekly and so on;. This service is light-weight and offers an at-least-once scheduling assure. Maestro engine service will deduplicate the triggering requests to attain an exact-once assure when scheduling workflows.

Time-based triggering is well-liked on account of its simplicity and ease of administration. However generally, it isn’t environment friendly. For instance, the day by day workflow ought to course of the information when the information partition is prepared, not at all times at midnight. Due to this fact, on prime of handbook and time-based triggering, we additionally present event-driven triggering.

Sign Service

Maestro helps event-driven triggering over alerts, that are items of messages carrying data akin to parameter values. Sign triggering is environment friendly and correct as a result of we don’t waste sources checking if the workflow is able to run, as a substitute we solely execute the workflow when a situation is met.

Alerts are utilized in two methods:

• A set off to start out new workflow cases
• A gating perform to conditionally begin a step (e.g., information partition readiness)

Sign service targets are to

• Acquire and index alerts
• Register and deal with workflow set off subscriptions
• Register and deal with the step gating features
• Captures the lineage of workflows triggers and steps unblocked by a sign

The maestro sign service consumes all of the alerts from completely different sources, e.g. all of the warehouse desk updates, S3 occasions, a workflow releasing a sign, after which generates the corresponding triggers by correlating a sign with its subscribed workflows. Along with the transformation between exterior alerts and workflow triggers, this service can be liable for step dependencies by trying up the acquired alerts within the historical past. Just like the scheduling service, the sign service along with Maestro engine achieves exactly-once triggering ensures.

Sign service additionally offers the sign lineage, which is beneficial in lots of instances. For instance, a desk up to date by a workflow may result in a sequence of downstream workflow executions. More often than not the workflows are owned by completely different groups, the sign lineage helps the upstream and downstream workflow homeowners to see who will depend on whom.

All providers within the Maestro system are stateless and will be horizontally scaled out. All of the requests are processed through distributed queues for message passing. By having a shared nothing structure, Maestro can horizontally scale to handle the states of thousands and thousands of workflow and step cases on the identical time.

CockroachDB is used for persisting workflow definitions and occasion state. We selected CockroachDB as it’s an open-source distributed SQL database that gives robust consistency ensures that may be scaled horizontally with out a lot operational overhead.

It’s exhausting to assist tremendous giant workflows usually. For instance, a workflow definition can explicitly outline a DAG consisting of thousands and thousands of nodes. With that variety of nodes in a DAG, UI can not render it effectively. We’ve got to implement some constraints and assist legitimate use instances consisting of lots of of hundreds (and even thousands and thousands) of step cases in a workflow occasion.

Primarily based on our findings and person suggestions, we discovered that in apply

• Customers don’t need to manually write the definitions for hundreds of steps in a single workflow definition, which is tough to handle and navigate over UI. When such a use case exists, it’s at all times possible to decompose the workflow into smaller sub workflows.
• Customers anticipate to repeatedly run a sure a part of DAG lots of of hundreds (and even thousands and thousands) occasions with completely different parameter settings in a given workflow occasion. So at runtime, a workflow occasion would possibly embrace thousands and thousands of step cases.

Due to this fact, we implement a workflow DAG dimension restrict (e.g. 1K) and we offer a foreach sample that permits customers to outline a sub DAG inside a foreach block and iterate the sub DAG with a bigger restrict (e.g. 100K). Observe that foreach will be nested by one other foreach. So customers can run thousands and thousands or billions of steps in a single workflow occasion.

In Maestro, foreach itself is a step within the authentic workflow definition. Foreach is internally handled as one other workflow which scales equally as every other Maestro workflow based mostly on the variety of step executions within the foreach loop. The execution of sub DAG inside foreach shall be delegated to a separate workflow occasion. Foreach step will then monitor and gather standing of these foreach workflow cases, every of which manages the execution of 1 iteration.

With this design, foreach sample helps sequential loop and nested loop with excessive scalability. It’s simple to handle and troubleshoot as customers can see the general loop standing on the foreach step or view every iteration individually.

We goal to make Maestro person pleasant and straightforward to study for customers with completely different backgrounds. We made some assumptions about person proficiency in programming languages and so they can carry their enterprise logic in a number of methods, together with however not restricted to, a bash script, a Jupyter notebook, a Java jar, a docker picture, a SQL assertion, or a couple of clicks within the UI utilizing parameterized workflow templates.

Consumer Interfaces

Maestro offers a number of area particular languages (DSLs) together with YAML, Python, and Java, for finish customers to outline their workflows, that are decoupled from their enterprise logic. Customers can even instantly discuss to Maestro API to create workflows utilizing the JSON information mannequin. We discovered that human readable DSL is well-liked and performs an essential function to assist completely different use instances. YAML DSL is the preferred one on account of its simplicity and readability.

Right here is an instance workflow outlined by completely different DSLs.

Moreover, customers can even generate sure forms of workflows on UI or use different libraries, e.g.

• In Pocket book UI, customers can instantly schedule to run the chosen pocket book periodically.
• In Maestro UI, customers can instantly schedule to maneuver information from one supply (e.g. a knowledge desk or a spreadsheet) to a different periodically.
• Customers can use Metaflow library to create workflows in Maestro to execute DAGs consisting of arbitrary Python code.

Parameterized Workflows

Plenty of occasions, customers need to outline a dynamic workflow to adapt to completely different situations. Primarily based on our experiences, a totally dynamic workflow is much less favorable and exhausting to keep up and troubleshooting. As a substitute, Maestro offers three options to help customers to outline a parameterized workflow

• Conditional branching
• Sub-workflow
• Output parameters

As a substitute of dynamically altering the workflow DAG at runtime, customers can outline these adjustments as sub workflows after which invoke the suitable sub workflow at runtime as a result of the sub workflow id is a parameter, which is evaluated at runtime. Moreover, utilizing the output parameter, customers can produce completely different outcomes from the upstream job step after which iterate by way of these throughout the foreach, go it to the sub workflow, or use it within the downstream steps.

Right here is an instance (utilizing YAML DSL) of backfill workflow with 2 steps. In step1, the step computes the backfill ranges and returns the dates (from 20210101 to 20220101) again. Subsequent, foreach step makes use of the dates from step1 to create foreach iterations. Lastly, every of the backfill jobs will get the date from the foreach and backfills the information based mostly on the date.

Workflow:
id: demo.pipeline
FROM_DATE: 20210101 #inclusive
TO_DATE: 20220101   #unique
jobs:
- job:
id: step1
kind: NoOp
'!dates': dateIntsBetween(FROM_DATE, TO_DATE, 1); #SEL expr
- foreach:
id: step2
params:
date: [email protected]  #reference upstream step parameter
jobs:
- job: 
id: backfill
kind: Pocket book
pocket book:
input_path: s3://path/to/pocket book.ipynb
arg1: $date  #go the foreach parameter into pocket book

The parameter system in Maestro is totally dynamic with code injection assist. Customers can write the code in Java syntax because the parameter definition. We developed our personal secured expression language (SEL) to make sure safety. It solely exposes restricted performance and contains extra checks (e.g. the variety of iteration within the loop assertion, and so on.) within the language parser.

Execution Abstractions

Maestro offers a number of ranges of execution abstractions. Customers can select to make use of the supplied step kind and set its parameters. This helps to encapsulate the enterprise logic of generally used operations, making it very simple for customers to create jobs. For instance, for spark step kind, all customers should do is simply specify wanted parameters like spark sql question, reminiscence necessities, and so on, and Maestro will do all behind-the-scenes to create the step. If we’ve got to make a change within the enterprise logic of a sure step, we are able to accomplish that seamlessly for customers of that step kind.

If supplied step sorts usually are not sufficient, customers can even develop their very own enterprise logic in a Jupyter pocket book after which go it to Maestro. Superior customers can develop their very own well-tuned docker picture and let Maestro deal with the scheduling and execution.

Moreover, we summary the widespread features or reusable patterns from numerous use instances and add them to the Maestro in a loosely coupled method by introducing job templates, that are parameterized notebooks. That is completely different from step sorts, as templates present a mix of assorted steps. Superior customers additionally leverage this characteristic to ship widespread patterns for their very own groups. Whereas creating a brand new template, customers can outline the record of required/optionally available parameters with the categories and register the template with Maestro. Maestro validates the parameters and kinds on the push and run time. Sooner or later, we plan to increase this performance to make it very simple for customers to outline templates for his or her groups and for all staff. In some instances, sub-workflows are additionally used to outline widespread sub DAGs to attain multi-step features.

We’re taking Large Knowledge Orchestration to the subsequent stage and continually fixing new issues and challenges, please keep tuned. If you’re motivated to unravel giant scale orchestration issues, please join us as we’re hiring.

Thanks Andrew Seier, Alexandre Bertails, Jess Hester, Xiao Chen, Liang Tian, Romain Cledat, Yun Li, Olek Gorajek, Anoop Panicker, Aravindan Ramkumar, Andrew Leung, and different gorgeous colleagues at Netflix for his or her contributions to the Maestro integration works whereas creating Maestro. We additionally thank Eva Tse, Charles Smith, Charles Zhao, and different leaders of Netflix engineering organizations for his or her constructive suggestions and solutions on the Maestro structure and design.