The Netflix TechBlog

AUGUST 30, 2021

By Benson Ma , Alok Ahuja Introduction At Netflix, hundreds of different device types, from streaming sticks to smart TVs, are tested every day through automation to ensure that new software releases continue to deliver the quality of the Netflix experience that our customers enjoy.

Latency 213

Latency Traffic Transportation Hardware 213

Percona

NOVEMBER 2, 2023

Upgrading to the newest release of MongoDB is the key to unlocking its full potential, but it’s not as simple as clicking a button; it requires meticulous planning, precise execution, and a deep understanding of the upgrade process. We walk you through the essential steps required.

Best Practices 89

Best Practices Hardware Tuning Scalability 89

Adrian Cockcroft

JANUARY 29, 2023

The mean and percentile measurements hide this structure, but the rest of this post will show how the structure can be measured and analyzed so that you can figure out a useful model of your system, understand what is driving the long tail of latencies and come up with better SLAs and measures of capacity.

Lambda 98

Lambda Latency Cache C++ 98

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 110

Performance Latency Cache Virtualization 110

Brendan Gregg

JULY 2, 2021

E.g., to see process execution with timestamps using execsnoop(8): # execsnoop-bpfcc -T. Low frequency events such as process execution should be negligible to capture. Here are the top ten tools you can run and present as a generic BPF observability dashboard, along with suggested visualizations: Tool Shows Visualization.

Latency 145

Latency Cache Energy Systems 145

ACM Sigarch

MARCH 18, 2019

This Part 1 discusses Bottleneck Analysis and Little’s Law, while Part 2 presents the M/M/1 Queue. To this end, we present three simple models that we find useful like Amdahl’s Law, split here into two blog posts. How many buffers are needed to track pending requests as a function of needed bandwidth and expected latency?

Systems 60

Systems Latency Performance Analytics 60

The Netflix TechBlog

NOVEMBER 22, 2019

This entertaining romp through the tech stack serves as an introduction to how we think about and design systems, the Netflix approach to operational challenges, and how other organizations can apply our thought processes and technologies. Technology advancements in content creation and consumption have also increased its data footprint.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

This entertaining romp through the tech stack serves as an introduction to how we think about and design systems, the Netflix approach to operational challenges, and how other organizations can apply our thought processes and technologies. Technology advancements in content creation and consumption have also increased its data footprint.

AWS 100

AWS Entertainment Open Source Benchmarking 100

Dotcom-Montior

DECEMBER 8, 2021

Now that we have talked about what an incident is, incident management is the process by which teams resolve these events and bring systems and services back to normal operation. ITSM is the policies, processes, and structure behind the lifecycle of IT services. Incident Management Lifecyle: Process and Steps. Service Desk.

Social Media 90

Social Media Monitoring Latency DevOps 90

Dynatrace

MAY 21, 2024

Observability data presents executives with new opportunities to achieve this, by creating incremental value for cloud modernization , improved business analytics , and enhanced customer experience. With the latest advances from Dynatrace, this process is instantaneous.

Technology 189

Technology Technology Analytics Storage 189

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

Smashing Magazine

AUGUST 20, 2021

Jamstack CMS: The Past, The Present and The Future. Jamstack CMS: The Past, The Present and The Future. While developers are an essential part of the Jamstack, they’re often heavily involved in the content publishing process. Mike Neumegen. 2021-08-20T08:00:00+00:00. 2021-08-20T09:19:47+00:00. Less Reliance On Developers.

Ecommerce 139

Ecommerce Website Government Internet 139

The Netflix TechBlog

JUNE 13, 2023

Our previous blog post presented replay traffic testing — a crucial instrument in our toolkit that allows us to implement these transformations with precision and reliability. A process that doesn’t just minimize risk, but also facilitates a continuous evaluation of the rollout’s impact.

Traffic 279

Traffic Metrics Systems Strategy 279

Scalegrid

DECEMBER 21, 2023

Identifying key Redis® metrics such as latency, CPU usage, and memory metrics is crucial for effective Redis monitoring. To monitor Redis® instances effectively, collect Redis metrics focusing on cache hit ratio, memory allocated, and latency threshold.

Strategy 130

Strategy Monitoring Latency DevOps 130

Brendan Gregg

JULY 2, 2021

While basic, I've solved many perf issues with this tool alone, including for misconfigured systems where a shell script is launching failing processes in a loop, and when some minor application is crashing and is restarting every few minutes but has not yet been noticed. ## 2. execsnoop New processes (via exec(2)) table 2. acpid [.]

Open Source 40

Open Source Latency Cache Energy 40

The Netflix TechBlog

NOVEMBER 22, 2019

This entertaining romp through the tech stack serves as an introduction to how we think about and design systems, the Netflix approach to operational challenges, and how other organizations can apply our thought processes and technologies. Technology advancements in content creation and consumption have also increased its data footprint.

AWS 37

AWS Entertainment Open Source Benchmarking 37

The Netflix TechBlog

MARCH 1, 2021

It supports both high throughput services that consume hundreds of thousands of CPUs at a time, and latency-sensitive workloads where humans are waiting for the results of a computation. The subsystems all communicate with each other asynchronously via Timestone, a high-scale, low-latency priority queuing system.

Serverless 228

Serverless Media Latency Social Media 228

HammerDB

DECEMBER 5, 2020

A full understanding of why this is important requires some knowledge of the evolution of database hardware and software. For TPC-C this meant enough available spindles to reduce I/O latency and for TPC-H enough bandwidth for data throughput. This was both expensive and time consuming to configure.

C++ 40

C++ Benchmarking Virtualization Open Source 40

The Netflix TechBlog

FEBRUARY 9, 2021

In the process, we changed end-to-end identity propagation within the network of services to use a cryptographically-verifiable token-agnostic identity object. We would need to process authentication tokens (and protocols) further upstream. EAS also covers the read-only processing of tokens to create Passports (more on that later).

Architecture 221

Architecture Latency Servers Website 221

The Netflix TechBlog

NOVEMBER 3, 2022

In the time since it was first presented as an advanced Mesos framework, Titus has transparently evolved from being built on top of Mesos to Kubernetes, handling an ever-increasing volume of containers. This blog post presents how our current iteration of Titus deals with high API call volumes by scaling out horizontally.

Cache 224

Cache Latency Traffic Systems 224

Scalegrid

FEBRUARY 8, 2024

This process effectively duplicates essential parts of information to safeguard against potential loss. Durability Availability Fault tolerance These combined outcomes help minimize latency experienced by clients spread across different geographical regions.

Storage 130

Storage Systems Big Data Azure 130

The Netflix TechBlog

SEPTEMBER 2, 2020

Edgar helps Netflix teams troubleshoot distributed systems efficiently with the help of a summarized presentation of request tracing, logs, analysis, and metadata. When a problem occurs, we put on our detective hats and start our mystery-solving process by gathering evidence. by Elizabeth Carretto Everyone loves Unsolved Mysteries.

Latency 296

Latency Transportation Engineering Traffic 296

The Morning Paper

JULY 7, 2019

Software engineering for machine learning: a case study Amershi et al., Previously on The Morning Paper we’ve looked at the spread of machine learning through Facebook and Google and some of the lessons learned together with processes and tools to address the challenges arising. A general process. ICSE’19.

Software Engineering 92

Software Engineering Engineering Software Software 92

The Netflix TechBlog

SEPTEMBER 3, 2021

When we process a request it is often beneficial to know which fields the caller is interested in and which ones they ignore. Remote calls are never free; they impose extra latency, increase probability of an error, and consume network bandwidth. FieldMask is a protobuf message.

Design 245

Design Java Efficiency Code 245

Sutter's Mill

FEBRUARY 13, 2017

Tue-Thu Apr 25-27: High-Performance and Low-Latency C++ (Stockholm). On April 25-27, I’ll be in Stockholm (Kista) giving a three-day seminar on “High-Performance and Low-Latency C++.” If you’re interested in attending, please check out the links, and I look forward to meeting and re-meeting many of you there.

Latency 51

Latency C++ Hardware Performance 51

Scalegrid

JANUARY 12, 2024

While managing cloud workloads offers numerous benefits, it also presents several challenges such as security risks, compliance issues, and resource optimization, which can be addressed effectively with tools like ScaleGrid, offering features like encryption, disaster recovery, and real-time resource optimization for diverse databases.

Cloud 130

Cloud Virtualization Storage Efficiency 130

The Netflix TechBlog

FEBRUARY 20, 2019

by Jason Koch , with Martin Spier , Brendan Gregg , Ed Hunter Improving the tools available to our engineers to help them diagnose, triage, and work through software performance challenges in the cloud is a key goal for the cloud performance engineering team at Netflix. to the broader community.

Performance 211

Performance Latency Open Source Metrics 211

Dynatrace

DECEMBER 22, 2021

Metrics are measures of critical system values, such as CPU utilization or average write latency to persistent storage. With limited visibility, teams have a narrow understanding of how those decisions impact other software components and vice-versa. Observability is made up of three key pillars: metrics, logs, and traces.

Artificial Intelligence 177

Artificial Intelligence Metrics Architecture DevOps 177

Dynatrace

APRIL 6, 2022

Observability can identify the baseline user experience and allow teams to improve it by optimizing page load times or reducing latency. Cloud environments present IT complexity challenges that don’t exist in on-premises data centers. Why full-stack observability matters. Integrate run-time application security into DevSecOps.

DevOps 230

DevOps Innovation Infrastructure Analytics 230

The Netflix TechBlog

AUGUST 29, 2023

For Inter-Process Communication (IPC) between services, we needed the rich feature set that a mid-tier load balancer typically provides. Eureka and Ribbon presented a simple but powerful interface, which made adopting them easy. There is a downside to fetching this data on-demand: this adds latency to the first request to a cluster.

Traffic 220

Traffic Latency Cloud C++ 220

Percona

MARCH 29, 2023

When this data block is read, it decompresses it in memory and presents it to the incoming request. When data is written to disk, MongoDB compresses it with a specified block compression method and then writes it to disk. Block compression can improve performance by allowing data to be read and written in smaller chunks.

Storage 110

Storage Network Open Source Latency 110

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. What about short-lived processes, like a service restarting in a loop? As a Xen guest, this profile was gathered using perf(1) and the kernel's software cpu-clock soft interrupts, not the hardware NMI. in total.

Speed 126

Speed Java AWS Virtualization 126

The Morning Paper

MAY 12, 2019

An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems Gan et al., Microservices fundamentally change a lot of assumptions current cloud systems are designed with, and present both opportunities and challenges when optimizing for quality of service (QoS) and utilization.

Open Source 61

Open Source Hardware Benchmarking Systems 61

Percona

DECEMBER 26, 2023

The collected data is aggregated and presented in a single view. This simplifies debugging and analysis processes by enabling users to execute the same query. Configuring Grafana For our purposes, the Grafana datasource used in this PoC is also the Postgres data cluster that is generating the data to be monitored.

Benchmarking 124

Benchmarking Metrics C++ Database 124

The Netflix TechBlog

FEBRUARY 9, 2021

Introduction and account creation Highlight our value propositions and begin the account creation process. When we merge these two concepts together and present them to the customer, we have the plan selection page (shown above). Given the flow and mode, the Orchestration Service can then process the request.

Engineering 142

Engineering Scalability Architecture Innovation 142

The Netflix TechBlog

MARCH 13, 2020

Compared to the most recent master version of libaom (AV1 reference software), SVT-AV1 is similar in compression efficiency and at the same time achieves significantly lower encoding latency on multi-core platforms when using its inherent parallelization capabilities.

Open Source 154

Open Source Efficiency C++ Speed 154

The Netflix TechBlog

FEBRUARY 9, 2021

We need to be able to easily determine what imagery is present for a given platform, region, and language. Server-generated assets, since client-side generation would require the retrieval of many individual images, which would increase latency and time-to-render. The imagery needs to be localized.

Engineering 165

Engineering Storage Latency Entertainment 165