The Netflix TechBlog

NOVEMBER 14, 2023

This data pipeline monitors the various stages in the customer lifecycle. In the sequential load ETL, we have the following features: Catchup Threshold : This defines the lookback period for the data being read. This helps overwrite data only when required and minimizes unnecessary reprocessing.

Tuning 244

Tuning Processing C++ Efficiency 244

The Netflix TechBlog

MARCH 7, 2024

Since then, open-source Metaflow has gained support for Argo Workflows , a Kubernetes-native orchestrator, as well as support for Airflow which is still widely used by data engineering teams. Internally, we use a production workflow orchestrator called Maestro.

Systems 226

Systems Media Cache Open Source 226

The Netflix TechBlog

NOVEMBER 22, 2019

In this session, we discuss the technologies used to run a global streaming company, growing at scale, billions of metrics, benefits of chaos in production, and how culture affects your velocity and uptime. By watching applications for anomalous actions, security and operations teams can monitor unusual and erroneous behavior.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

In this session, we discuss the technologies used to run a global streaming company, growing at scale, billions of metrics, benefits of chaos in production, and how culture affects your velocity and uptime. By watching applications for anomalous actions, security and operations teams can monitor unusual and erroneous behavior.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

In this session, we discuss the technologies used to run a global streaming company, growing at scale, billions of metrics, benefits of chaos in production, and how culture affects your velocity and uptime. By watching applications for anomalous actions, security and operations teams can monitor unusual and erroneous behavior.

AWS 37

AWS Entertainment Open Source Benchmarking 37

The Netflix TechBlog

NOVEMBER 20, 2023

The need for backfilling could be due to a variety of factors, e.g. (1) upstream data sets got repopulated due to changes in business logic of its data pipeline, (2) business logic was changed in a data pipeline, (3) anew metric was created that needs to be populated for historical time ranges, (4) historical data was found missing, etc.

Processing 210

Processing Big Data Efficiency Engineering 210