Orchestrating Data/ML Workflows at Scale With Netflix Maestro
The Netflix TechBlog
OCTOBER 18, 2022
The scheduler on-call has to closely monitor the system during non-business hours. In the data domain, it is common to have a super large number of jobs within a single workflow. For example, a workflow to backfill hourly data for the past five years can lead to 43800 jobs (24 * 365 * 5), each of which processes data for an hour.
Let's personalize your content