The Netflix TechBlog

OCTOBER 15, 2019

Another thread or process is constantly polling events from the log table and writes them to one or multiple datastores, optionally removing events from the log table after acknowledged by all datastores. CDC (Change-Data-Capture) events are sent by the Delta-Connector to a Keystone Kafka topic. clock drift on the partition leader).

Transportation 192

Transportation Architecture Processing Storage 192

The Netflix TechBlog

NOVEMBER 20, 2023

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Processing 210

Processing Big Data Efficiency Engineering 210

Percona

JANUARY 19, 2023

Compression: Compression is the process of restructuring the data by changing its encoding in order to store it in fewer bytes. There are many compression tools and algorithms for data out there. It was developed for optimizing data storage and access for big data sets. 1 mysql mysql 592K Dec 30 02:48 tb1.ibd

Open Source 87

Open Source Storage Database Big Data 87

The Netflix TechBlog

JULY 26, 2021

Netflix is known for its loosely coupled microservice architecture and with a global studio footprint, surfacing and connecting the data from microservices into a studio data catalog in real time has become more important than ever. With the latest Data Mesh Platform, data movement in Netflix Studio reaches a new stage.

Big Data 253

Big Data Government Analytics Processing 253

SQL Server According to Bob

FEBRUARY 15, 2018

Best practices on Building a Big Data Analytics Solution – Michael Rys. If you want to learn about Azure Data Lake, there is no one better. Friday Sessions: SQL Intelligence excels your tuning and security expertise – Veljko Vasic, Ron Matchoro and Frans Lytzen. I’ve known Michael for a very long time.

Engineering 40

Engineering Azure Best Practices Servers 40

Dynatrace

APRIL 20, 2023

-based financial services group, discussed how the bank uses log monitoring on the Dynatrace platform with an emphasis on observability and security data. To grasp the challenges of multifeatured, cross-team cooperation dealing with observability data, consider the content of the logs generated. Dissolving data silos.

Analytics 187

Analytics Infrastructure Storage Efficiency 187

Dynatrace

FEBRUARY 16, 2023

The goal is to turn more data into insights so the whole organization can make data-driven decisions and automate processes. Grail data lakehouse delivers massively parallel processing for answers at scale Modern cloud-native computing is constantly upping the ante on data volume, variety, and velocity.

Analytics 186

Analytics Innovation Metrics Database 186

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3.

Latency 243

Latency Storage Big Data Tuning 243

Rigor

SEPTEMBER 11, 2019

Reading time 16 min Whether you’re a web performance expert, an evangelist for the culture of performance, a web engineer incorporating performance into your process, or someone new to the web performance entirely, you probably identify as curious, excited about new ideas, and always learning. Maximiliano Firtman. Maximiliano Firtman.

Performance 89

Performance Education Google Website 89

The Netflix TechBlog

JULY 21, 2022

We at Netflix, as a streaming service running on millions of devices, have a tremendous amount of data about device capabilities/characteristics and runtime data in our big data platform. With large data, comes the opportunity to leverage the data for predictive and classification based analysis.

Big Data 179

Big Data Cache Engineering Data Engineering 179

Dynatrace

MARCH 11, 2021

I took a big-data-analysis approach, which started with another problem visualization. I wanted to understand how I could tune Dynatrace’s problem detection, but to do that I needed to understand the situation first. To achieve that I took two approaches: Visualizing historic problem data via a “Swimlane Visualization”.

Tuning 263

Tuning Architecture Monitoring Big Data 263

The Netflix TechBlog

DECEMBER 21, 2020

There are several benefits of such optimizations like saving on storage, faster query time, cheaper downstream processing, and an increase in developer productivity by removing additional ETLs written only for query performance improvement. Some of the optimizations are prerequisites for a high-performance data warehouse.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

MARCH 25, 2019

Netflix’s diverse data landscape made it challenging to capture all the right data and conforming it to a common data model. Spark is the primary big-data compute engine at Netflix and with pretty much every upgrade in Spark, the spark plan changed as well springing continuous and unexpected surprises for us.

Infrastructure 219

Infrastructure Big Data Transportation Architecture 219

The Netflix TechBlog

OCTOBER 18, 2022

by Jun He , Akash Dwivedi , Natallia Dzenisenka , Snehal Chennuru , Praneeth Yenugutala , Pawan Dixit At Netflix, Data and Machine Learning (ML) pipelines are widely used and have become central for the business, representing diverse use cases that go beyond recommendations, predictions and data transformations.

Java 202

Java Scalability Traffic Architecture 202

O'Reilly

SEPTEMBER 19, 2023

Each time, the underlying implementation changed a bit while still staying true to the larger phenomenon of “Analyzing Data for Fun and Profit.” ” They weren’t quite sure what this “data” substance was, but they’d convinced themselves that they had tons of it that they could monetize.

Hardware 99

Hardware Storage Big Data Blockchain 99

All Things Distributed

MARCH 28, 2011

The methods for accessing these objects is also rapidly changing; where in the past you needed a PC or a Laptop to access these objects, now many of our electronic devices have become capable of processing them. Driving down the cost of Big-Data analytics. Introducing the AWS South America (Sao Paulo) Region.

AWS 138

AWS Cloud Storage Internet 138

All Things Distributed

NOVEMBER 30, 2016

There are many more application areas where we use ML extensively: search, autonomous drones, robotics in fulfillment centers, text processing and speech recognition (such as in Alexa) etc. Effectively applying AI involves extensive manual effort to develop and tune many different types of machine learning and deep learning algorithms (e.g.

AWS 165

AWS Lambda Artificial Intelligence Mobile 165

The Netflix TechBlog

NOVEMBER 3, 2023

Democratizing Stream Processing @ Netflix By Guil Pires , Mark Cho , Mingliang Liu , Sujay Jain Data powers much of what we do at Netflix. On the Data Platform team, we build the infrastructure used across the company to process data at scale. Stay tuned for more updates!

Processing 265

Processing Engineering Infrastructure Latency 265

Percona

JANUARY 14, 2019

Processed 4.15 The data is partitioned and sorted by created_utc so queries which include created_utc will be able to using partition pruning: therefore skip the not-needed partitions. Processed 4.15 Processed 8.19 Processed 8.19 Processed 4.15 Processed 3.05 Processed 8.19 count()?? ?

Database 40

Database Analytics Blockchain Healthcare 40