article thumbnail

Supporting Diverse ML Systems at Netflix

The Netflix TechBlog

Explainer flow is event-triggered by an upstream flow, such Model A, B, C flows in the illustration. Since then, open-source Metaflow has gained support for Argo Workflows , a Kubernetes-native orchestrator, as well as support for Airflow which is still widely used by data engineering teams.

Systems 226
article thumbnail

How Data Inspires Building a Scalable, Resilient and Secure Cloud Infrastructure At Netflix

The Netflix TechBlog

While our engineering teams have and continue to build solutions to lighten this cognitive load (better guardrails, improved tooling, …), data and its derived products are critical elements to understanding, optimizing and abstracting our infrastructure. What will be the cost of rolling out the winning cell of an AB test to all users?

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

How HubSpot Uses Apache Kafka Swimlanes for Timely Processing of Workflow Actions

InfoQ

HubSpot adopted routing messages over multiple Kafka topics (called swimlanes) for the same producer to avoid the build-up in the consumer group lag and prioritize the processing of real-time traffic. By Rafal Gancarz

article thumbnail

Netflix at AWS re:Invent 2019

The Netflix TechBlog

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! We’ve compiled our speaking events below so you know what we’ve been working on. Please stop by our “Living Room” for an opportunity to connect or reconnect with Netflixers. Wednesday?—?December

AWS 100
article thumbnail

Netflix at AWS re:Invent 2019

The Netflix TechBlog

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! We’ve compiled our speaking events below so you know what we’ve been working on. Please stop by our “Living Room” for an opportunity to connect or reconnect with Netflixers. Wednesday?—?December

AWS 100
article thumbnail

Netflix at AWS re:Invent 2019

The Netflix TechBlog

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! We’ve compiled our speaking events below so you know what we’ve been working on. Please stop by our “Living Room” for an opportunity to connect or reconnect with Netflixers. Wednesday?—?December

AWS 37
article thumbnail

Sponsored Post: Fauna, Sisu, Educative, PA File Sight, Etleap, PerfOps, Triplebyte, Stream

High Scalability

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.