article thumbnail

Netflix at AWS re:Invent 2019

The Netflix TechBlog

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! Technology advancements in content creation and consumption have also increased its data footprint. We’ve compiled our speaking events below so you know what we’ve been working on.

AWS 100
article thumbnail

Netflix at AWS re:Invent 2019

The Netflix TechBlog

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! Technology advancements in content creation and consumption have also increased its data footprint. We’ve compiled our speaking events below so you know what we’ve been working on.

AWS 100
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Supporting Diverse ML Systems at Netflix

The Netflix TechBlog

Finally, when all matching is done and data is written the new table is committed so it can be read by other jobs. Compute: Titus Whereas open-source users of Metaflow rely on AWS Batch or Kubernetes as the compute backend , we rely on our centralized compute-platform, Titus.

Systems 226
article thumbnail

Netflix at AWS re:Invent 2019

The Netflix TechBlog

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! Technology advancements in content creation and consumption have also increased its data footprint. We’ve compiled our speaking events below so you know what we’ve been working on.

AWS 37
article thumbnail

Sponsored Post: InterviewCamp.io, Scrapinghub, Fauna, Sisu, Educative, PA File Sight, Etleap, Triplebyte, Stream

High Scalability

Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you'd be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years. PA File Sight monitors file access on a server in real-time.

Education 100
article thumbnail

Data Pipelines: The Hammer for Every Nail

Abhishek Tiwari

Airflow provides rich scheduling and execution semantics enabling data engineers to easily define complex pipelines, running at regular intervals. The workflow platform acts as the backbone of Uber's business, enabling seamless coordination, automation, and monitoring of tasks throughout the entire lifecycle of a ride.

article thumbnail

Orchestrating Data/ML Workflows at Scale With Netflix Maestro

The Netflix TechBlog

The scheduler on-call has to closely monitor the system during non-business hours. As the usage increased, we had to vertically scale the system to keep up and were approaching AWS instance type limits. Meson was based on a single leader architecture with high availability. With the high growth of workflows in the past few years?

Java 202