The Netflix TechBlog

MARCH 5, 2024

Reduced tail latencies In both our GRPC and DGS Framework services, GC pauses are a significant source of tail latencies. Each of these errors is a canceled request resulting in a retry so this reduction further reduces overall service traffic by this rate: Errors rates per second.

Latency 228

Latency Java Tuning Efficiency 228

Dynatrace

JUNE 1, 2023

But how do you get started, and what are some service level objective examples? In this post, I’ll lay out five foundational service level objective examples that every DevOps and SRE team should consider. Five example SLOs for faster, more reliable apps 1. Note : you might hear the term latency used instead of response time.

Latency 181

Latency Website Traffic Virtualization 181

The Netflix TechBlog

NOVEMBER 3, 2022

For example, it is OK to send writes through one instance, and do reads from another one with full data read consistency guarantees. For example, a batch workflow orchestration system may create multiple jobs which are part of a single workflow execution. Kubernetes is a good example here.

Cache 224

Cache Latency Traffic Systems 224

Dynatrace

DECEMBER 27, 2022

First, it helps to understand that applications and all the services and infrastructure that support them generate telemetry data based on traffic from real users. In this example, “Reverse proxy” and “Front-end server” are clearly in the critical path. Latency is the time that it takes a request to be served. Reliability.

Software 269

Software Software Benchmarking Latency 269

Dynatrace

MAY 17, 2023

How Dynatrace uses Site Reliability Guardian In each of these Dynatrace examples, insight is made in a production-like environment. These examples can help you define your starting point for establishing DevOps and SRE best practices in your organization. The functionality is implemented via an automated workflow.

DevOps 192

DevOps Traffic Latency Best Practices 192

The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. We expanded pipeline support to serve our studio/content-development use cases, which had different latency and resiliency requirements as compared to the traditional streaming use case. divide the input video into small chunks 2.

Processing 229

Processing Media Latency Innovation 229

Dynatrace

FEBRUARY 15, 2023

SREs use Service-Level Indicators (SLI) to see the complete picture of service availability, latency, performance, and capacity across various systems, especially revenue-critical systems. In addition, the workflow can be easily extended to any custom demands, for example, integration with tools that support your software product lifecycle.

DevOps 224

DevOps Latency Traffic Best Practices 224

Dynatrace

MAY 31, 2023

At the lowest level, SLIs provide a view of service availability, latency, performance, and capacity across systems. For example, a panel of customers may occasionally provide feedback on service quality and performance. 5 SRE best practices Let’s break down SRE best practices into the following five major steps: 1.

Best Practices 227

Best Practices DevOps Latency Metrics 227

Dynatrace

MARCH 16, 2023

Monitors signals The first attribute of a good SLO is the ability to monitor the four “golden signals”: latency, traffic, error rates, and resource saturation. Grabner and Cabrera offer the example of an iOS app experiencing crash issues after a team deploys a new version.

DevOps 220

DevOps Latency Metrics Traffic 220

Dynatrace

JUNE 25, 2020

For example, in a three-node cluster, one node can go down; in a cluster with five or more nodes, two nodes can go down. The network latency between cluster nodes should be around 10 ms or less. Minimized cross-data center network traffic. Automatic recovery for outages for up to 72 hours. Self-contained turnkey solution.

Availability 251

Availability Hardware Latency Traffic 251

Scalegrid

JANUARY 25, 2024

Key Takeaways Critical performance indicators such as latency, CPU usage, memory utilization, hit rate, and number of connected clients/slaves/evictions must be monitored to maintain Redis’s high throughput and low latency capabilities. Similarly, an increased throughput signifies an intensive workload on a server and a larger latency.

Metrics 130

Metrics Monitoring Latency Cache 130

Dynatrace

MAY 19, 2022

Example 1: Architecture boundaries. For example, the mobile device talked to an API gateway, and that team was not responsible for all the details of the back-end systems. In their new dashboard, they added dimensions for load, latency, and open problems for each component. Example 2: Four golden signals. Saturation.

Automotive 274

Automotive Latency Architecture Azure 274

The Netflix TechBlog

AUGUST 16, 2023

For example, a 16-core box connecting to an 800-server origin would have 12,800 connections. That’s a significant amount and certainly more than is necessary relative to the traffic on most clusters. More acutely, if a traffic spike occurs and Zuul instances scale up, it exponentially increases connections open to origins.

Traffic 170

Traffic Servers Google Metrics 170

The Netflix TechBlog

FEBRUARY 18, 2022

For example, a member-triggered event such as “ change in a profile’s maturity level” should have a much higher priority than a “ system diagnostic signal”. We thus assigned a priority to each use case and sharded event traffic by routing to priority-specific queues and the corresponding event processing clusters.

Systems 334

Systems Traffic Architecture Mobile 334

The Netflix TechBlog

NOVEMBER 2, 2020

How viewers are able to watch their favorite show on Netflix while the infrastructure self-recovers from a system failure By Manuel Correa , Arthur Gonigberg , and Daniel West Getting stuck in traffic is one of the most frustrating experiences for drivers around the world. Logs and background requests are examples of this type of traffic.

Traffic 252

Traffic Metrics Infrastructure Architecture 252

Dynatrace

APRIL 25, 2023

For example, to handle traffic spikes and pay only for what they use. Scale automatically based on the demand and traffic patterns. Higher latency and cold start issues due to the initialization time of the functions. The elasticity of serverless services helps organizations scale as needed.

Serverless 213

Serverless Lambda Azure AWS 213

The Netflix TechBlog

SEPTEMBER 8, 2020

As an example, to render the screen shown here, the app sends a query that looks like this: paths: ["videos", 80154610, "detail"] A path starts from a root object , and is followed by a sequence of keys that we want to retrieve the data for. Replay Testing Enter replay testing.

Latency 233

Latency Cache Java Traffic 233

Dynatrace

JULY 24, 2023

For example, consider an e-commerce website that automatically sends personalized discount codes to customers who abandon their shopping carts. Let’s examine a few examples of answer-driven automation using AI and context. DevOps automation example #2: Threat detection and response Let’s build on the previous example.

DevOps 225

DevOps Traffic Efficiency Servers 225

The Netflix TechBlog

MARCH 7, 2024

Example use case: Content Knowledge Graph Our knowledge graph of the entertainment world encodes relationships between titles, actors and other attributes of a film or series, supporting all aspects of business at Netflix. In other cases, it is more convenient to share the results via a low-latency API.

Systems 226

Systems Media Cache Open Source 226

The Morning Paper

NOVEMBER 14, 2019

Taiji: managing global user traffic for large-scale internet services at the edge Xu et al., It’s another networking paper to close out the week (and our coverage of SOSP’19), but whereas Snap looked at traffic routing within the datacenter, Taiji is concerned with routing traffic from the edge to a datacenter. SOSP’19.

Traffic 42

Traffic Internet Internet Latency 42

The Netflix TechBlog

MARCH 10, 2023

Existing data got updated to be backward compatible without impacting the existing running production traffic. Data Sharding strategy in elasticsearch is updated to provide low search latency (as described in blog post) Design of new Cassandra reverse indices to support different sets of queries.

Media 237

Media Traffic Processing Design 237

Dynatrace

JULY 18, 2023

While developers edit files, a simple CLI command applies configurations to Dynatrace and, for example, automates the setup of a quality gate, including workflows and Site Reliability Guardians. Stay tuned for more examples and easy-to-adopt automations provided in our public Github project.

Best Practices 195

Best Practices Code Infrastructure Latency 195

The Netflix TechBlog

SEPTEMBER 2, 2020

Example call graph in Edgar In some cases, the question we’re answering is, “What’s happening right now??” Edgar captures 100% of interesting traces , as opposed to sampling a small fixed percentage of traffic. Searching for key evidence becomes like digging for a needle in a group of haystacks.

Latency 296

Latency Transportation Engineering Traffic 296

Dynatrace

JUNE 27, 2022

Data collected on page load events, for example, can include navigation start (when performance begins to be measured), request start (right before the user makes a request from the server), and speed index metrics (measure page load speed). RUM, however, has some limitations, including the following: RUM requires traffic to be useful.

Best Practices 179

Best Practices Monitoring Wireless Traffic 179

Dynatrace

MAY 17, 2023

Think about items such as general system metrics (for example, CPU utilization, free memory, number of services), the connectivity status, details of our web server, or even more granular in-application tasks like database queries. Let’s take our previous screenshot as an example.

Metrics 175

Metrics Monitoring Database Network 175

Percona

DECEMBER 11, 2023

DLVs are particularly advantageous for databases with large allocated storage, high I/O per second (IOPS) requirements, or latency-sensitive workloads. For write-only traffic, the QPS counters match the performance of standard RDS instances for lower thread counts, though, for higher counters, there is a drastic improvement.

AWS 93

AWS Benchmarking Performance Traffic 93

The Netflix TechBlog

AUGUST 30, 2021

For example, when running tests, the state of the device will change from “available for testing” to “in test.” As such, we can see that the traffic load on the Device Management Platform’s control plane is very dynamic over time. Collect and aggregate information and state updates for all devices attached to the RAEs in the fleet.

Latency 213

Latency Traffic Transportation Hardware 213

Dynatrace

APRIL 26, 2022

STM generates traffic that replicates the typical path or behavior of a user on a network to measure performance for example, response times, availability, packet loss, latency, jitter, and other variables).

Monitoring 209

Monitoring Social Media IoT Metrics 209

Dynatrace

SEPTEMBER 30, 2021

In case of a spike in traffic, you can automatically spin up more resources, often in a matter of seconds. Likewise, you can scale down when your application experiences decreased traffic. For example, as traffic increases, costs will too. Analyze your resource consumption and traffic patterns.

Cloud 169

Cloud Traffic Best Practices Strategy 169

The Netflix TechBlog

AUGUST 13, 2020

We also highlight interesting broader events such as regional traffic evacuations and nearby deployments , information that is vital to understanding health holistically. Regional traffic evacuations. For example, a latency increase is less critical than error rate increase and some error codes are less critical than others.

Monitoring 210

Monitoring Tuning Traffic Metrics 210

IO River

NOVEMBER 2, 2023

If price is your top priority, you'll need to decide how much you're willing to sacrifice in terms of reliability and performance.What are your traffic patterns like? If your traffic is mostly static, you may be able to meet all your needs with a less expensive CDN that provides content distribution services. per one million requests.

Infrastructure 52

Infrastructure Traffic Cache Strategy 52

The Netflix TechBlog

JUNE 4, 2019

Because microprocessors are so fast, computer architecture design has evolved towards adding various levels of caching between compute units and the main memory, in order to hide the latency of bringing the bits to the brains. As an illustrative example, let’s consider a toy instance of 16 hyperthreads.

Cache 251

Cache Latency Airlines Logistics 251

IO River

NOVEMBER 2, 2023

It is particularly beneficial during high-traffic periods or when serving content to a large audience.All of these benefits apply to modern applications that process user thumbnails. For example, BBC and CNN benefit greatly from dynamic content. For example, consider tools like ChatGPT.

Cache 52

Cache Social Media Website Performance Website 52

Adrian Cockcroft

JANUARY 29, 2023

The common way to deal with this is to measure percentiles, and track the 90%, 99% response times for example. For example lets say you have a 99% within 2 seconds SLA, and your current 99%ile measured over one minute is 1 second. There is no way to model how much more traffic you can send to that system before it exceeds it’s SLA.

Lambda 98

Lambda Latency Cache C++ 98

IO River

NOVEMBER 2, 2023

CDNs cache content on edge servers distributed globally, reducing the distance between users and the content they want.‍CDNs use load-balancing techniques to distribute incoming traffic across multiple servers called Points of Presence (PoPs) which distribute content closer to end-users and improve overall performance.

Architecture 52

Architecture Cache Performance Latency 52

Percona

JUNE 22, 2023

Let’s dive in and learn how (and what) to effectively monitor MySQL performance, along with examples from PMM, by understanding the critical KPIs to watch for. This is not an exhaustive list but an example of what we can watch for. That said, it should also be monitored for usage, which will exhibit the traffic pressuring them.

Performance 102

Performance Monitoring Traffic Database 102

IO River

NOVEMBER 2, 2023

If price is your top priority, you'll need to decide how much you're willing to sacrifice in terms of reliability and performance.What are your traffic patterns like? If your traffic is mostly static, you may be able to meet all your needs with a less expensive CDN that provides content distribution services. per one million requests.

Infrastructure 52

Infrastructure Traffic Cache Strategy 52