Remove Architecture Remove Java Remove Processing Remove Software Engineering
article thumbnail

Orchestrating Data/ML Workflows at Scale With Netflix Maestro

The Netflix TechBlog

Meson was based on a single leader architecture with high availability. For example, a workflow to backfill hourly data for the past five years can lead to 43800 jobs (24 * 365 * 5), each of which processes data for an hour. Figure 1 shows the high-level architecture. With the high growth of workflows in the past few years?

Java 202
article thumbnail

Achieving observability in async workflows

The Netflix TechBlog

Managing and operating asynchronous workflows can be difficult without the proper tools and architecture that puts observability, debugging, and tracing at the forefront. We are expected to process 1,000 watermarks for a single distribution in a minute, with non-linear latency growth as the number of watermarks increases.

Traffic 160
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

The Show Must Go On: Securing Netflix Studios At Scale

The Netflix TechBlog

Adding to the complexity, many of the checklist items themselves had a variety of different options to fulfill them (“new apps do this , but legacy apps do that”; “Java apps should use this approach , but Ruby apps should try one of these four things”… yes, there were flowcharts inside checklists.

Internet 220
article thumbnail

Sponsored Post: ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

High Scalability

T riplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Application security products and processes, however, have not kept up with advances in software development. Created by former senior-level AWS engineers of 15 years.

Education 115
article thumbnail

Sponsored Post: StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

High Scalability

T riplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Application security products and processes, however, have not kept up with advances in software development. Created by former senior-level AWS engineers of 15 years.

Education 105
article thumbnail

Sponsored Post: StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

High Scalability

T riplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Application security products and processes, however, have not kept up with advances in software development. Created by former senior-level AWS engineers of 15 years.

Education 102
article thumbnail

Sponsored Post: Ipdata, StackHawk, InterviewCamp.io, Educative, Triplebyte, Stream, Fauna

High Scalability

T riplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Application security products and processes, however, have not kept up with advances in software development. Created by former senior-level AWS engineers of 15 years.

Education 105