The Netflix TechBlog

AUGUST 30, 2021

By Benson Ma , Alok Ahuja Introduction At Netflix, hundreds of different device types, from streaming sticks to smart TVs, are tested every day through automation to ensure that new software releases continue to deliver the quality of the Netflix experience that our customers enjoy.

Latency 213

Latency Traffic Transportation Hardware 213

Dynatrace

APRIL 5, 2021

But with the benefits also come concerns about observability, and how to monitor and manage ever-expanding cloud software stacks. You will likely need to write code to integrate systems and handle complex tasks or incoming network requests. AWS continues to improve how it handles latency issues.

Lambda 186

Lambda AWS Serverless Hardware 186

The Morning Paper

NOVEMBER 15, 2020

An IDS/IPS monitors network flows and matches incoming packets (or more strictly, Protocol Data Units, PDUs) against a set of rules. This makes the whole system latency sensitive. Regular expression matching is well studied, but state of the art hardware algorithms don’t reach the performance and memory targets needed for Pigasus.

Servers 128

Servers Hardware Latency Design 128

Sutter's Mill

FEBRUARY 13, 2017

Tue-Thu Apr 25-27: High-Performance and Low-Latency C++ (Stockholm). On April 25-27, I’ll be in Stockholm (Kista) giving a three-day seminar on “High-Performance and Low-Latency C++.”

Latency 51

Latency C++ Hardware Performance 51

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on. I also wrote about these topics in detail for my recent [Systems Performance 2nd Edition] book.

Performance 140

Performance Latency Hardware Storage 140

Dotcom-Montior

NOVEMBER 24, 2021

There was a time when standing up a website or application was simple and straightforward and not the complex networks they are today. These systems can include physical servers, containers, virtual machines, or even a device, or node, that connects and communicates with the network. The recipe was straightforward. Peer-to-Peer.

Systems 74

Systems Monitoring Hardware Network 74

The Morning Paper

MAY 19, 2019

Last week we learned about the [increased tail-latency sensitivity of microservices based applications with high RPC fan-outs. Seer uses estimates of queue depths to mitigate latency spikes on the order of 10-100ms, in conjunction with a cluster manager. So what we have here is a glimpse of the limits for low-latency RPCs under load.

Latency 80

Latency Hardware Network Architecture 80

Percona

NOVEMBER 9, 2023

If you have any experience working with database software, you have undoubtedly heard the term Kubernetes a lot. Kubernetes manages and orchestrates these containers, handling tasks such as deployment, scaling, load balancing, and networking. Just consider the sheer number of people who stream Netflix every night!

Efficiency 101

Efficiency Cloud Healthcare Open Source 101

Dotcom-Montior

MAY 12, 2020

Network or connection error. Network latency. Hardware resources. Network Latency. With the evolution of cloud technologies, such as Single Page Applications (SPAs), Web APIs, and Model View Controller (MVC), network latency has become a crucial factor to be monitored. Hardware Resources.

Monitoring 63

Monitoring Entertainment Hardware Latency 63

All Things Distributed

MARCH 11, 2016

Almost from day one, we knew that the software we were building would not be the software that would be running a year later. We needed to build such an architecture that we could introduce new software components without taking the service down. Build evolvable systems. Primitives not frameworks. Automation is key.

AWS 144

AWS Hardware Retail Virtualization 144

All Things Distributed

APRIL 4, 2017

As well as AWS Regions, we also have 24 AWS Edge Network Locations in Europe. This enables customers to serve content to their end users with low latency, giving them the best application experience. In 2011, AWS opened a Point of Presence (PoP) in Stockholm to enable customers to serve content to their end users with low latency.

AWS 116

AWS Airlines Latency Games 116

Dynatrace

SEPTEMBER 30, 2021

Cloud migration is the process of transferring some or all your data, software, and operations to a cloud-based computing environment that offers unlimited scale and high availability. They need enough hardware to serve their anticipated volume and keep things running smoothly without buying too much or too little. Reduced cost.

Cloud 172

Cloud Traffic Best Practices Strategy 172

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 110

Performance Latency Cache Virtualization 110

Alex Russell

JANUARY 30, 2024

It's time once again to update our priors regarding the global device and network situation. HTML, CSS, images, and fonts can all be parsed and run at near wire speeds on low-end hardware, but JavaScript is at least three times more expensive, byte-for-byte. What's changed since last year? and 75KiB of JavaScript.

Performance 90

Performance Network Mobile Speed 90

The Netflix TechBlog

FEBRUARY 9, 2021

The homepage needs to load in a reasonable amount of time, even in poor network conditions. Server-generated assets, since client-side generation would require the retrieval of many individual images, which would increase latency and time-to-render. Different assets for different device types and screen sizes.

Engineering 165

Engineering Storage Latency Entertainment 165

The Morning Paper

MARCH 3, 2020

The core algorithms (chain-replication, Paxos-based consensus) aren’t the stars of the show here, instead the paper focuses on how these algorithms are deployed, and the software engineering practices behind the creation of a mission-critical production system employing them. Larger cells have better tolerance of tail latency (e.g.

Database 106

Database AWS Network Design 106

HammerDB

OCTOBER 23, 2023

With a simple example such as this, it would not necessarily be expected for the additional network traffic to be significant between the 2 approaches. However, with more complex application logic this network round trip soon becomes a key focus area for improving performance. On MySQL, we saw a 1.5X performance advantage. client (i.e.

Benchmarking 62

Benchmarking Database Network C++ 62

John McCalpin

AUGUST 1, 2018

According to Dr. Bandwidth, performance analysis has two recurring themes: How fast should this code (or “simple” variations on this code) run on this hardware? The user environment defines the mapping of MPI ranks to hardware resources (cores, sockets, nodes). The MPI runtime library. in ways that are seldom transparent.

Hardware 40

Hardware Transportation Performance Latency 40

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on. I also wrote about these topics in detail for my recent [Systems Performance 2nd Edition] book.

Performance 52

Performance Latency Hardware Storage 52

Smashing Magazine

AUGUST 22, 2021

As we will see, QUIC and HTTP/3 indeed have great web performance potential, but mainly for users on slow networks. If your average visitor is on a fast cabled or cellular network, they probably won’t benefit from the new protocols all that much. Two-way latency is often called round-trip time (RTT). Congestion Control.

Performance 106

Performance Network Latency Servers 106

Brendan Gregg

AUGUST 18, 2019

bpftrace is a new open source tracer for Linux for analyzing production performance problems and troubleshooting software. For example, iostat(1), or a monitoring agent, may tell you your average disk latency, but not the distribution of this latency. software Kernel software-based events.

Latency 68

Latency C++ Cache Programming 68

Percona

JUNE 22, 2023

Replication lag can occur due to various factors such as network latency, system resource limitations, complex transactions, or heavy write loads on the primary/master database. Note that the specific configuration variables and their optimal values may vary depending on the MySQL version, system hardware, workload, and other factors.

Performance 103

Performance Monitoring Traffic Database 103

The Morning Paper

MAY 14, 2019

Last time around we looked at the DeathStarBench suite of microservices-based benchmark applications and learned that microservices systems can be especially latency sensitive, and that hotspots can propagate through a microservices architecture in interesting ways. on end-to-end latency) and less than 0.15% on throughput.

Big Data 45

Big Data Cloud Performance Hardware 45

The Morning Paper

APRIL 14, 2019

The paper sets out what we can do in software given today’s hardware, and along the way also highlights areas where cooperation from hardware will be needed in the future. There are also stateless interconnects including buses and on-chip networks. Requirement 4 : State flushing must be padded to its worst-case latency.

Hardware 44

Hardware Cache Latency Speed 44

CSS Wizardry

OCTOBER 27, 2018

Things always always feel fast when we’re developing because, more often than not, we’re working on high-spec machines on dedicated networks, and also serving from localhost which removes the bulk of the latency and bandwidth issues that a real user would suffer. need to go out of our way to spot the problems.

Performance Testing 172

Performance Testing Testing Performance Strategy 172

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

VoltDB

JUNE 1, 2017

It was – like the hypothetical movie I describe above – more than a little bit odd, as you could leave a session discussing ever more abstract layers of virtualization and walk into one where they emphasized the critical importance of pinning a network interface to a specific VM for optimal performance.

Hardware 56

Hardware Latency Virtualization Cloud 56

Dotcom-Montior

OCTOBER 6, 2021

A site reliability engineer, or SRE, is a role that that encompasses aspects of both software engineering and operations/infrastructure. At that time, the team was made up of software engineers. Site reliability engineers bridge the gap between operations and software developers. What are Some Common SRE Responsibilities?

Engineering 67

Engineering DevOps Monitoring Google 67

The Morning Paper

DECEMBER 8, 2019

Byte-addressable non-volatile memory,) NVM will fundamentally change the way hardware interacts, the way operating systems are designed, and the way applications operate on data. Therefore any programming abstraction must be low latency and the kernel needs to be kept off the path of persistent data access as much as possible.

Hardware 56

Hardware Programming Media Storage 56

The Morning Paper

JULY 4, 2019

degraded hardware, transient networking problem) or, more often, because of some change deployed by Netflix engineers that did not have the intended effect. Two failure modes we focus on are a service becoming slower (increase in response latency) or a service failing outright (returning errors). On error rates.

Latency 77

Latency Engineering Metrics Traffic 77

VoltDB

JUNE 1, 2017

It was – like the hypothetical movie I describe above – more than a little bit odd, as you could leave a session discussing ever more abstract layers of virtualization and walk into one where they emphasized the critical importance of pinning a network interface to a specific VM for optimal performance.

Hardware 40

Hardware Latency Virtualization Cloud 40

Abhishek Tiwari

JULY 12, 2018

Data Pipeline A data pipeline is a software that ingests data from multiple sources, transforms it and finally makes it available to internal or external products. A data pipeline is a software which runs on hardware. The software is error-prone and hardware failures are inevitable.

Latency 63

Latency Analytics Scalability Engineering 63

All Things Distributed

JULY 13, 2010

Customers with complex computational workloads such as tightly coupled, parallel processes, or with applications that are very sensitive to network performance, can now achieve the same high compute and networking performance provided by custom-built infrastructure while benefiting from the elasticity, flexibility and cost advantages of Amazon EC2.

Cloud 118

Cloud AWS Automotive Latency 118

All Things Distributed

JANUARY 23, 2012

VM Import allows our customers to move virtual machine images from their datacenters to the Cloud and Amazon Direct Connect makes the network latencies and bandwidth between on-premises and AWS more predictable. The AWS Storage Gateway is a service connecting an on-premises software appliance with cloud-based storage.

Storage 115

Storage AWS Virtualization Cloud 115

Alex Russell

APRIL 29, 2021

Real-time network protocols for enabling videoconferencing, desktop sharing, and game streaming applications. Modern, asynchronous network APIs that dramatically improve performance in some situations. For heavily latency-sensitive use-cases like WebXR, this is a critical component in delivering a good experience. Gamepad API.

Media 145

Media Games Education Engineering 145

Brendan Gregg

AUGUST 8, 2017

90491 N|rnberg (Germany) Consulting+Networking+Programming+etc'ing 42. I generated this using my offcputime tool from [bcc] (this tool needs eBPF features from Linux 4.8+), and my [flame graph] software: #./bcc/tools/offcputime.py Network I/O? Latency was acceptable and no one complained. Yes, I'd say so. File system I/O?

Latency 111

Latency Metrics C++ Systems 111

Dotcom-Montior

DECEMBER 8, 2021

Software services still require physical devices and hardware for them to function. Knowing when and where an error, downtime, or application latency occurs is a critical factor in limiting the impact to users and customers. Change management is also sometimes grouped with release management. Asset Management.

Social Media 90

Social Media Monitoring Latency DevOps 90