The Netflix TechBlog

OCTOBER 28, 2021

I re-architected our legacy data pipelines and built a new platform on top of Apache Flink and Iceberg. The scale brought some interesting learnings and involved working closely with the Apache Flink and Kafka community to fix core issues. You can learn more about it from my talk at the Flink forward conference.

Data Engineering 173

Data Engineering Engineering Big Data Software Engineering 173

The Netflix TechBlog

OCTOBER 15, 2019

Stream Processing Framework The processing layer of Delta is built on top of Netflix SPaaS platform, which provides Apache Flink integration with the Netflix ecosystem. The platform provides a self-service UI which manages Flink job deployments and Flink cluster orchestration on top of our container management platform Titus.

Transportation 192

Transportation Architecture Processing Storage 192

ScaleOut Software

AUGUST 5, 2020

And how are they different from streaming pipelines like Azure Stream Analytics and Apache Flink/Beam? What Problems Does Streaming Analytics Solve? To understand why we need real-time digital twins for streaming analytics, we first need to look at what problems are tackled by popular streaming platforms.

Analytics 69

Analytics Azure Processing Speed 69

The Netflix TechBlog

OCTOBER 18, 2019

a) Standardized Logging from the treatment services, b) Real-time stream processing over Apache Flink for member activity joins, and c) an Apache Spark client for attribution and reward computation. To be able to support this, the team had to standardize on several infrastructure components.

Infrastructure 126

Infrastructure Metrics Architecture Efficiency 126

The Netflix TechBlog

OCTOBER 18, 2019

a) Standardized Logging from the treatment services, b) Real-time stream processing over Apache Flink for member activity joins, and c) an Apache Spark client for attribution and reward computation. To be able to support this, the team had to standardize on several infrastructure components.

Infrastructure 100

Infrastructure Metrics Architecture Efficiency 100

Abhishek Tiwari

JULY 12, 2018

This includes both data pipeline developed using batch-driven frameworks such as Apache AirFlow, Luigi, etc. as well as data pipelines operating on the data stream and built using frameworks such as Apache Storm, Apache Flink, etc. Depending on frameworks, data processing units (a.k.a

Latency 63

Latency Analytics Scalability Engineering 63

The Netflix TechBlog

NOVEMBER 20, 2023

We will show how we are building a clean and efficient incremental processing solution (IPS) by using Netflix Maestro and Apache Iceberg. Apache Iceberg Iceberg is a high-performance format for huge analytic tables. Here is our solution to achieve incremental change capture built upon Apache Iceberg features.

Processing 210

Processing Big Data Efficiency Engineering 210

InfoQ

APRIL 15, 2024

At the QCon London 2024 conference, Félix GV from LinkedIn discussed the AI/ML platform powering the company’s products. He specifically delved into Venice DB, the NoSQL data store used for feature persistence. The presenter shared the lessons learned from evolving and operating the platform, including cluster management and library versioning.

Artificial Intelligence 106

Artificial Intelligence Big Data Data Engineering Latency 106

The Morning Paper

JULY 2, 2019

Today’s paper choice is authored by a collection of experts from the Apache Beam, Apache Calcite, and Apache Flink projects, outlining their experiences building SQL interfaces for streaming. At the sharp end, they are based on lessons learned working an Apache Flink, Beam, and Calcite. SIGMOD’19.

Efficiency 65

Efficiency Processing Benchmarking Open Source 65

InfoQ

FEBRUARY 11, 2024

Pinterest open-sourced its generic PubSub client library, PSC, which has been heavily used in production for a year and a half. The library helped the engineering teams by increasing developer velocity, and the scalability and stability of services using it. Over 90% of Java applications have migrated to PSC with minimal changes. By Rafal Gancarz

Open Source 99

Open Source Java Scalability Engineering 99

The Netflix TechBlog

APRIL 12, 2022

The schematized events are consumed by Data Mesh processors implemented in the Apache Flink framework. By transacting with a database which is monitored by a CDC connector that creates events, or b. By directly creating events using a Data Mesh client.

Engineering 263

Engineering Architecture Java Infrastructure 263

O'Reilly Software

APRIL 4, 2018

Google was among the pioneers that created “web scale” architectures to analyze the massive data sets that resulted from “crawling” the web that gave birth to Apache Hadoop, MapReduce, and NoSQL databases. Apache Kafka is currently the leading project in this area. In the same year, Apache Flink announced streaming SQL support.

Architecture 80

Architecture Scalability Google Operating System 80

ScaleOut Software

SEPTEMBER 20, 2018

Using digital twins offers three key benefits over more traditional, pipelined stream-processing techniques (such as Apache Storm and Apache Flink ): automatic event correlation by data source, deeper introspection with enhanced state information, and data-parallel analytics to discover aggregate trends for all data sources in real time.

Programming 63

Programming IoT Java Analytics 63

ScaleOut Software

SEPTEMBER 20, 2018

Using digital twins offers three key benefits over more traditional, pipelined stream-processing techniques (such as Apache Storm and Apache Flink ): automatic event correlation by data source, deeper introspection with enhanced state information, and data-parallel analytics to discover aggregate trends for all data sources in real time.

Programming 40

Programming IoT Java Analytics 40

ScaleOut Software

MAY 4, 2018

Traditional stream-processing and complex event processing systems, such as Apache Storm and Software AG’s Apama , have focused on extracting interesting patterns from incoming data with stateless applications.

Processing 40

Processing IoT Programming Software Engineering 40

ScaleOut Software

MAY 4, 2018

Traditional stream-processing and complex event processing systems, such as Apache Storm and Software AG’s Apama , have focused on extracting interesting patterns from incoming data with stateless applications.

Processing 40

Processing IoT Programming Software Engineering 40

Uber Engineering

OCTOBER 9, 2017

Uber facilitates seamless and more enjoyable user experiences by channeling data from a variety of real-time sources.

Open Source 85

Open Source Analytics Traffic Engineering 85

The Netflix TechBlog

NOVEMBER 3, 2023

The Processors themselves are implemented as Flink jobs that use the DataStream API. To do this, they had to use the low-level DataStream API from Flink and the Data Mesh SDK, which came with a steep learning curve. When a user wants to leverage Data Mesh to move and transform data, they start by creating a new Data Mesh pipeline.

Processing 265

Processing Engineering Infrastructure Latency 265