The Netflix TechBlog

FEBRUARY 9, 2021

The Growth Engineering team is responsible for executing growth initiatives that help us anticipate and adapt to this change. For more background on Growth Engineering and the signup funnel, please have a look at our previous blog post that covers the basics. We need to be constantly adapting and innovating as a result of this change.

Engineering 142

Engineering Scalability Architecture Innovation 142

The Netflix TechBlog

NOVEMBER 3, 2023

Additionally, instead of implementing business logic by composing multiple individual Processors together, users could express their logic in a single SQL query, avoiding the additional resource and latency overhead that came from multiple Flink jobs and Kafka topics. Stay tuned for more updates!

Processing 265

Processing Engineering Infrastructure Latency 265

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The data warehouse is not designed to serve point requests from microservices with low latency.

Latency 243

Latency Storage Big Data Tuning 243

The Netflix TechBlog

MARCH 1, 2021

It supports both high throughput services that consume hundreds of thousands of CPUs at a time, and latency-sensitive workloads where humans are waiting for the results of a computation. The subsystems all communicate with each other asynchronously via Timestone, a high-scale, low-latency priority queuing system. Warm capacity.

Serverless 228

Serverless Media Latency Social Media 228

Dynatrace

JULY 18, 2023

Efficient environment configuration at scale One of software engineers’ most significant challenges is managing the numerous tools and technologies required for the software product lifecycle. Stay tuned for more examples and easy-to-adopt automations provided in our public Github project.

Best Practices 198

Best Practices Code Infrastructure Latency 198

The Morning Paper

JULY 7, 2019

Software engineering for machine learning: a case study Amershi et al., More specifically, we’ll be looking at the results of an internal study with over 500 participants designed to figure out how product development and software engineering is changing at Microsoft with the rise of AI and ML. ICSE’19. A general process.

Software Engineering 92

Software Engineering Engineering Software Software 92

The Netflix TechBlog

AUGUST 13, 2020

You’re half awake and wondering, “Is there really a problem or is this just an alert that needs tuning? Over the years we’ve learned from on-call engineers about the pain points of application monitoring: too many alerts, too many dashboards to scroll through, and too much configuration and maintenance. By Andrei U.,

Monitoring 210

Monitoring Tuning Traffic Metrics 210

The Netflix TechBlog

MAY 14, 2021

Prodicle is one of the many applications that is at the exciting intersection of connecting the world of content productions to Netflix Studio Engineering. We are expected to process 1,000 watermarks for a single distribution in a minute, with non-linear latency growth as the number of watermarks increases.

Traffic 160

Traffic Latency Java Google 160

The Netflix TechBlog

AUGUST 29, 2023

Third, we’ve continued to add more functionality to our IPC clients: features such as adaptive concurrency limiting , circuit breaking , hedging, and fault injection have become standard tools that our engineers reach for to make our system more reliable. This is the first in a series of posts on our journey to service mesh, so stay tuned.

Traffic 220

Traffic Latency Cloud C++ 220

Dynatrace

JULY 24, 2023

Imagine having an AI engine that comprehends the complete context of the transaction and intelligently determines whether to send a discount code—and which one to send. Full contextual awareness helps the AI engine make informed decisions. Has the user purchased this product before?

DevOps 228

DevOps Traffic Efficiency Servers 228

The Netflix TechBlog

NOVEMBER 20, 2023

It also improves the engineering productivity by simplifying the existing pipelines and unlocking the new patterns. As our business scales globally, the demand for data is growing and the needs for scalable low latency incremental processing begin to emerge. There are three common issues that the dataset owners usually face.

Processing 210

Processing Big Data Efficiency Engineering 210

The Netflix TechBlog

SEPTEMBER 24, 2021

Uploading and downloading data always come with a penalty, namely latency. Figure 3: Video Processing with Index and Virtual Assembly Using virtual assembly greatly improves the latency performance of the ProRes 422 HQ proxy generation by removing one round trip of cloud downloading and cloud uploading by the physical assembler.

Cloud 237

Cloud Media Storage Cache 237

Percona

APRIL 7, 2023

If we were to select the most important MySQL setting, if we were given a freshly installed MySQL or Percona Server for MySQL and could only tune a single MySQL variable, which one would it be? To be fair, that is also true with PostgreSQL; it hasn’t been tuned either, and it, too, can also perform much better.

Tuning 136

Tuning Cache Servers Benchmarking 136

The Netflix TechBlog

AUGUST 22, 2019

A key responsibility of Netflix engineers is identifying gaps and pain points in the development and operation of services. First, an engineer would launch an EC2 instance and wait for the instance to come online. The canary stage will determine a score based on metrics such as CPU, threads, latency, and GC pauses.

DevOps 207

DevOps AWS Tuning Infrastructure 207

Percona

NOVEMBER 2, 2023

Improved performance : MongoDB continually fine-tunes its database engine, resulting in faster query execution and reduced latency. Regulatory compliance : Upgrading your database is vital for compliance with various legal and regulatory standards, where data management and security play a pivotal role.

Best Practices 87

Best Practices Hardware Tuning Scalability 87

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which We needed to increase engineering productivity via distributed request tracing. That is the first question our engineering teams asked us when integrating the tracer library.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Dynatrace

MAY 5, 2021

At Perform 2021 , Dynatrace’s Kristof Renders, Services Practice Manager for Autonomous Cloud Enablement, joined Sumit Nagal, Principal Engineer at Intuit, to demonstrate how service-level objectives (SLOs) and business-level objectives (BLOs) can “shift left.” For example, improving latency by as little as 0.1

Benchmarking 191

Benchmarking Latency Speed Software 191

The Netflix TechBlog

DECEMBER 21, 2020

Sometimes Data Engineers write downstream ETLs on ingested data to optimize the data/metadata layouts to make other ETL processes cheaper and faster. These principles reduce resource usage by being more efficient and effective while lowering the end-to-end latency in data processing. More processing resources.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

JUNE 13, 2023

By collecting and analyzing key performance metrics of the service over time, we can assess the impact of the new changes and determine if they meet the availability, latency, and performance requirements. They enable us to further fine-tune and configure the system, ensuring the new changes are integrated smoothly and seamlessly.

Traffic 279

Traffic Metrics Systems Strategy 279

The Netflix TechBlog

FEBRUARY 18, 2022

Personalized Experience Refresh Netflix Recommendation engine continuously refreshes recommendations for every member. This separation allows us to tune system configuration and scaling policies independently for different event priorities and traffic patterns. If this is something that excites you, we’d love for you to join us.

Systems 334

Systems Traffic Architecture Mobile 334

John McCalpin

APRIL 25, 2023

To understand what is happening here, we need to understand the way memory bandwidth interacts with memory latency and the concurrency (parallelism) of memory accesses. The NEC Vector Engine processors provide a demonstration of very high single-core bandwidth. Stay tuned! On a VE20B (8 cores, 1.6

Benchmarking 97

Benchmarking Cache Latency Tuning 97

Percona

MARCH 23, 2023

Nowadays, solid-state drives (SSDs) or non-volatile memory express (NVMe) drives are preferred over traditional hard disk drives (HDDs) for database servers due to their faster read and write speeds, lower latency, and improved reliability. If you see concurrency issues, you can tune this variable. I hope this helps! and MariaDB 10.5.4

Performance 125

Performance Hardware Tuning Storage 125

The Netflix TechBlog

JULY 21, 2022

Since memory management is not something one usually associates with classification problems, this blog focuses on formulating the problem as an ML problem and the data engineering that goes along with it. Some nuances while creating this dataset come from the on-field domain knowledge of our engineers. of the time (False Positives).

Big Data 179

Big Data Cache Engineering Data Engineering 179

The Morning Paper

NOVEMBER 10, 2019

You need a lot of software engineers and the willingness to rewrite a lot of software to entertain that idea. Here are the bombshell paragraphs: Our datacenter applications seek ever more CPU-efficient and lower-latency communication, which Pony Express delivers. The desire for CPU efficiency and lower latencies is easy to understand.

Network 92

Network Transportation Latency Entertainment 92

The Netflix TechBlog

NOVEMBER 2, 2020

As engineers at Netflix, we are constantly reevaluating how to redesign traffic management. In Netflix engineering, we’re driven by ensuring Netflix is there when you need it to be. Those two metrics are approximate indicators of failures and latency. Global throttling Another case is when Zuul itself is in trouble.

Traffic 252

Traffic Metrics Infrastructure Architecture 252

The Netflix TechBlog

NOVEMBER 2, 2021

Such coupling problems abound with our Reloaded architecture, and hence the Media Cloud Engineering and Encoding Technologies teams have been working together to develop a solution that addresses many of the concerns with our previous architecture. This enables us to use our scale to increase throughput and reduce latencies.

Media 171

Media Innovation Metrics Latency 171

Dynatrace

APRIL 26, 2022

STM generates traffic that replicates the typical path or behavior of a user on a network to measure performance for example, response times, availability, packet loss, latency, jitter, and other variables). One use case for STM is to model the behavior of a customer in the form of a flow of transactions along the buyer’s journey.

Monitoring 213

Monitoring Social Media IoT Metrics 213

Dynatrace

DECEMBER 2, 2021

SLOs can be a great way for DevOps and infrastructure teams to use data and performance expectations to make decisions, such as whether to release, and where engineers should focus their time. You can set SLOs based on individual indicators, such as batch throughput, request latency, and failures-per-second. Help with decision making.

Metrics 200

Metrics Best Practices DevOps Infrastructure 200

Smashing Magazine

NOVEMBER 8, 2021

As developers, we rightfully obsess about the customer experience, relentlessly working to squeeze every millisecond out of the critical rendering path, optimize input latency, and eliminate jank. Stay tuned for more in 2022! Ilya Grigorik. 2021-11-08T14:30:00+00:00. 2021-11-08T19:34:34+00:00.

Cache 136

Cache Best Practices Strategy Servers 136

Percona

APRIL 17, 2023

The main objective of this post is to share my experience over the past years tuning MongoDB and centralize the diverse sources that I crossed in this journey in a unique place. The CFQ works well for many general use cases but lacks latency guarantees. Spoiler alert: This post focuses on MongoDB 3.6.X tcp_fin_timeout = 30 net.ipv4.tcp_keepalive_intvl

Best Practices 52

Best Practices Design Tuning Database 52

All Things Distributed

NOVEMBER 29, 2017

After this, there is often a long process of training that includes tuning the knobs and levers, called hyperparameters, that control the different aspects of the training algorithm. Built-in, high-performance ML algorithms, re-engineered for greater, speed, accuracy, and data-throughput.

Tuning 94

Tuning AWS Scalability Infrastructure 94

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. I happened to be speaking at a technical confering while still debugging this, and mentioned what I was working on to a processor engineer. A quick check of basic performance statistics showed over 30% higher CPU consumption.

Speed 126

Speed Java AWS Virtualization 126

HammerDB

MAY 5, 2019

However in the Skylake microarchitecture (you can see a list of CPUs here ) the PAUSE instruction changed and in the documentation it says “the latency of the PAUSE instruction in prior generation microarchitectures is about 10 cycles, whereas in Skylake microarchitecture it has been extended to as many as 140 cycles.”

Testing 48

Testing Tuning Latency Storage 48

The Morning Paper

JUNE 13, 2019

Making queries to an inference engine has many of the same throughput, latency, and cost considerations as making queries to a datastore, and more and more applications are coming to depend on such queries. The following figure highlights how just one of these variables, batch size, impacts throughput and latency on ResNet50.

Hardware 58

Hardware Latency Serverless Energy 58

The Morning Paper

MAY 21, 2019

This boils down to a single digit µs latency toleration in the tail for far memory, and in addition to security and privacy concerns, rules out remote memory solutions. Thus we’re fundamentally trading (de)-compression latency at access time for the ability to pack more data in memory. ML-based auto-tuning. Evaluation.

Software 78

Software Software Google Hardware 78

Abhishek Tiwari

JULY 12, 2018

These data pipelines can process data at petabytes scale and to some extent, their success can be attributed to an army of engineers devoted to build and maintain internal data pipelines. Not everyone is operating at Netflix or Spotify scale data engineering function. Generally speaking, data pipelines are leaky by nature.

Latency 63

Latency Analytics Scalability Engineering 63

Brendan Gregg

JUNE 30, 2018

A co-worker introduced me to Craig Hanson and Pat Crain's performance mantras, which neatly summarize much of what we do in performance analysis and tuning. Accurate benchmarking rewards engineering investment that actually improves performance, but, unfortunately, inaccurate benchmarking is more common. Don't do it. Do it less.

Benchmarking 54

Benchmarking Latency Cache Network 54