article thumbnail

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

The Netflix TechBlog

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can

article thumbnail

Data Pipelines: The Hammer for Every Nail

Abhishek Tiwari

Airflow provides rich scheduling and execution semantics enabling data engineers to easily define complex pipelines, running at regular intervals. From transportation and logistics to e-commerce and food delivery, the core operations of many successful companies can be viewed as workflow problems.