Orchestrating Data/ML Workflows at Scale With Netflix Maestro
The Netflix TechBlog
OCTOBER 18, 2022
In the data domain, it is common to have a super large number of jobs within a single workflow. For example, a workflow to backfill hourly data for the past five years can lead to 43800 jobs (24 * 365 * 5), each of which processes data for an hour. All the requests are processed via distributed queues for message passing.
Let's personalize your content