The Netflix TechBlog

JUNE 13, 2023

Migrating Critical Traffic At Scale with No Downtime — Part 2 Shyam Gala , Javier Fernandez-Ivern , Anup Rokkam Pratap , Devang Shah Picture yourself enthralled by the latest episode of your beloved Netflix series, delighting in an uninterrupted, high-definition streaming experience. This is where large-scale system migrations come into play.

Traffic 279

Traffic Metrics Systems Strategy 279

KeyCDN

NOVEMBER 4, 2021

The POP is strategially located within the country and lowers latency overall. KeyCDN is always on the lookout for ways to minimize latency and accelerate asset delivery worldwide. Traffic from this POP will be billed towards Latin America according to our pricing. Hola Mexico!

Latency 111

Latency Tuning Cache Traffic 111

Dynatrace

JULY 24, 2023

It enables them to adapt to user feedback swiftly, fine-tune feature releases, and deliver exceptional user experiences, all while maintaining control and minimizing disruption. They can also see how the change can affect critical objectives like SLOs and golden signals, such as traffic, latency, saturation, and error rate.

DevOps 222

DevOps Traffic Efficiency Servers 222

IO River

NOVEMBER 2, 2023

‍Just as a well-coordinated airport directs flights to multiple runways based on traffic and weather conditions, a CDN with Multiple Origins Load Balancing ensures that web traffic is distributed across various data centers, optimizing performance and reliability. ‍But how does it decide where to send this traffic?

Traffic 52

Traffic Cache Servers Latency 52

The Netflix TechBlog

AUGUST 30, 2021

Canary Test Workloads In addition to serving the regular message traffic between users and DUTs, the control plane itself is stress-tested at roughly 3-hour intervals, where nearly 3000 ephemeral MQTT clients are created to connect to and generate flash traffic on the MQTT brokers. million elements.

Latency 213

Latency Traffic Transportation Hardware 213

Dynatrace

APRIL 26, 2022

STM generates traffic that replicates the typical path or behavior of a user on a network to measure performance for example, response times, availability, packet loss, latency, jitter, and other variables).

Monitoring 205

Monitoring Social Media IoT Metrics 205

KeyCDN

MARCH 10, 2021

The image below shows a significant drop in latency once we've launched the new point of presence in Israel. In fact, latency has been reduced by almost 50%! With a total of 5 POPs in Oceania, this continent benefits from lower latency with every POP added. So far, traffic from Nigeria has been routed to Europe.

Latency 110

Latency Internet Internet Tuning 110

KeyCDN

NOVEMBER 29, 2022

KeyCDN is always looking for ways to minimize latency and accelerate the delivery of assets worldwide. According to our pricing , traffic from this POP will be billed to Latin America. If you'd like to request a POP in another location or a new feature please let us know and stay tuned for more exciting announcements.

Latency 60

Latency Tuning Cache Traffic 60

Dynatrace

MAY 17, 2023

The other sections on that page (such as Disk analysis) provide further information and charts on topics such as available disk space, latency, dropped network packets, refused connections, and more. This allows us to quickly tell whether the network link may be saturated or the processor is running at its limit.

Metrics 173

Metrics Monitoring Database Network 173

IO River

NOVEMBER 2, 2023

Just as a well-coordinated airport directs flights to multiple runways based on traffic and weather conditions, a CDN with Multiple Origins Load Balancing ensures that web traffic is distributed across various data centers, optimizing performance and reliability. But how does it decide where to send this traffic?

Traffic 40

Traffic Cache Network Servers 40

Percona

MAY 1, 2023

I’ll show you some MySQL settings to tune to get better performance, and cost savings, with AWS RDS. Recently I was engaged in a MySQL Performance Audit for a customer to help troubleshoot performance issues that led to downtime during periods of high traffic on their AWS RDS MySQL instances.

AWS 117

AWS Hardware Storage Tuning 117

The Netflix TechBlog

OCTOBER 19, 2020

If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. Our engineering teams tuned their services for performance after factoring in increased resource utilization due to tracing.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

IO River

NOVEMBER 15, 2023

In technical terms, network-level firewalls regulate access by blocking or permitting traffic based on predefined rules. ‍At its core, WAF operates by adhering to a rulebook—a comprehensive list of conditions that dictate how to handle incoming web traffic. You've put new rules in place. Let's dive deep into these challenges.‍1.

Traffic 52

Traffic Network Logistics Architecture 52

Smashing Magazine

NOVEMBER 8, 2021

As developers, we rightfully obsess about the customer experience, relentlessly working to squeeze every millisecond out of the critical rendering path, optimize input latency, and eliminate jank. Stay tuned for more in 2022! Ilya Grigorik. 2021-11-08T14:30:00+00:00. 2021-11-08T19:34:34+00:00.

Cache 135

Cache Best Practices Strategy Servers 135

Dotcom-Montior

NOVEMBER 11, 2020

Developers don’t have to put in additional time to fine-tuning the system, or rely on other teams for support, as it’s done automatically with the cloud provider. However, when the time comes for resources to be requested, there can be latency in the time it takes to for that code to start back up. Focus on Application Development.

Serverless 57

Serverless Monitoring Lambda Latency 57

IO River

NOVEMBER 2, 2023

In technical terms, network-level firewalls regulate access by blocking or permitting traffic based on predefined rules. At its core, WAF operates by adhering to a rulebook—a comprehensive list of conditions that dictate how to handle incoming web traffic. You've put new rules in place. Let's dive deep into these challenges.‍1.

Traffic 40

Traffic Network Logistics Architecture 40

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. Since instances of both CentOS and Ubuntu were running in parallel, I could collect flame graphs at the same time (same time-of-day traffic mix) and compare them side by side.

Speed 126

Speed Java AWS Virtualization 126

Percona

APRIL 17, 2023

The main objective of this post is to share my experience over the past years tuning MongoDB and centralize the diverse sources that I crossed in this journey in a unique place. The CFQ works well for many general use cases but lacks latency guarantees. Spoiler alert: This post focuses on MongoDB 3.6.X tcp_fin_timeout = 30 net.ipv4.tcp_keepalive_intvl

Best Practices 52

Best Practices Design Tuning Database 52

The Netflix TechBlog

JUNE 14, 2023

The control group’s traffic utilized the legacy Falcor stack, while the experiment population leveraged the new GraphQL client and was directed to the GraphQL Shim. To determine customer impact, we could compare various metrics such as error rates, latencies, and time to render.

Traffic 353

Traffic Latency Cache Metrics 353

KeyCDN

AUGUST 15, 2019

Before you begin tuning your website or application, you must first figure out which metrics matter most to your users and establish some achievable benchmarks. Just because everything works perfectly during production testing doesn’t mean that will be the case when your website is flooded with traffic.

Testing Tools 55

Testing Tools Best Practices Performance Testing Testing 55

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. Since instances of both CentOS and Ubuntu were running in parallel, I could collect flame graphs at the same time (same time-of-day traffic mix) and compare them side by side.

Speed 40

Speed Java AWS Virtualization 40

The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. We expanded pipeline support to serve our studio/content-development use cases, which had different latency and resiliency requirements as compared to the traditional streaming use case. divide the input video into small chunks 2.

Processing 229

Processing Media Latency Innovation 229

The Netflix TechBlog

MAY 14, 2021

Prodicle Distribution Our service is required to be elastic and handle bursty traffic. We are expected to process 1,000 watermarks for a single distribution in a minute, with non-linear latency growth as the number of watermarks increases. Things got hairy. We wanted a scalable service that was near real-time, 2.

Traffic 160

Traffic Latency Java Google 160

The Netflix TechBlog

NOVEMBER 2, 2020

How viewers are able to watch their favorite show on Netflix while the infrastructure self-recovers from a system failure By Manuel Correa , Arthur Gonigberg , and Daniel West Getting stuck in traffic is one of the most frustrating experiences for drivers around the world. Logs and background requests are examples of this type of traffic.

Traffic 252

Traffic Metrics Infrastructure Architecture 252

The Netflix TechBlog

AUGUST 13, 2020

You’re half awake and wondering, “Is there really a problem or is this just an alert that needs tuning? Telltale learns what constitutes typical health for an application, no alert tuning required. Regional traffic evacuations. A regional traffic shift means one region ends up with zero traffic while another region has double.

Monitoring 210

Monitoring Tuning Traffic Metrics 210

The Netflix TechBlog

FEBRUARY 18, 2022

We thus assigned a priority to each use case and sharded event traffic by routing to priority-specific queues and the corresponding event processing clusters. This separation allows us to tune system configuration and scaling policies independently for different event priorities and traffic patterns.

Systems 334

Systems Traffic Architecture Mobile 334

The Netflix TechBlog

AUGUST 29, 2023

In order for a service to talk to another, it needs to know two things: the name of the destination service, and whether or not the traffic should be secure. The ability to run in a degraded but available state during an outage is still a marked improvement over completely stopping traffic flow.

Traffic 220

Traffic Latency Cloud C++ 220

Brendan Gregg

SEPTEMBER 5, 2017

Here's some output from my zfsdist tool, in bcc/BPF, which measures ZFS latency as a histogram on Linux: # zfsdist. Tracing ZFS operation latency. Both Xen and KVM have had many performance and security improvements, and workloads can now be tuned to run at almost bare metal speeds (say, a 3% loss or less). Hit Ctrl-C to end. ^C

Virtualization 76

Virtualization AWS Engineering Hardware 76

The Netflix TechBlog

JULY 26, 2021

Operational Reporting is a reporting paradigm specialized in covering high-resolution, low-latency data sets, serving detailed day-to-day activities¹ and processes of a business domain. Most of the business views created on top of the Iceberg tables can tolerate a few minutes of latency. Please stay tuned! Dehghani, Zhamak.

Big Data 253

Big Data Government Analytics Processing 253

The Netflix TechBlog

NOVEMBER 2, 2021

This enables us to use our scale to increase throughput and reduce latencies. Here, based on the video length, the throughput and latency requirements, available scale etc., Stay tuned for more details on these algorithmic innovations. VQS is called using the measureQuality endpoint. The workflow is initiated.

Media 171

Media Innovation Metrics Latency 171

The Netflix TechBlog

DECEMBER 17, 2019

Nonetheless, we found a number of limitations that could not satisfy our requirements e.g. stalling the processing of log events until a dump is complete, missing ability to trigger dumps on demand, or implementations that block write traffic by using table locks. Blocking write traffic by locking tables. Writing events to any output.

Database 197

Database Traffic Transportation Open Source 197

The Netflix TechBlog

DECEMBER 17, 2019

Nonetheless, we found a number of limitations that could not satisfy our requirements e.g. stalling the processing of log events until a dump is complete, missing ability to trigger dumps on demand, or implementations that block write traffic by using table locks. Blocking write traffic by locking tables. Writing events to any output.

Database 212

Database Traffic Transportation Open Source 212

All Things Distributed

OCTOBER 2, 2017

VPC Endpoints give you the ability to control whether network traffic between your application and DynamoDB traverses the public Internet or stays within your virtual private cloud. Performant – DynamoDB consistently delivers single-digit millisecond latencies even as your traffic volume increases.

Internet 130

Internet Internet AWS Performance 130

Smashing Magazine

AUGUST 22, 2021

Because we are dealing with network protocols here, we will mainly look at network aspects, of which two are most important: latency and bandwidth. Latency can be roughly defined as the time it takes to send a packet from point A (say, the client) to point B (the server). Two-way latency is often called round-trip time (RTT).

Performance 105

Performance Network Latency Servers 105

Smashing Magazine

AUGUST 9, 2021

You’ve probably heard things like: “HTTP/3 is much faster than HTTP/2 when there is packet loss”, or “HTTP/3 connections have less latency and take less time to set up”, and probably “HTTP/3 can send data more quickly and can send more resources in parallel”. TLS, TCP, and QUIC handshake durations ( Large preview ).

Transportation 145

Transportation Internet Internet Network 145

Smashing Magazine

SEPTEMBER 6, 2021

Finally, not inlining resources has an added latency cost because the file needs to be requested. As such, a micro-optimization is, again, how you probably need to fine-tune things on a low level to really benefit from it. Note that there is an Apache Traffic Server implementation, though.). What Does It All Mean?

Network 104

Network Servers Cache Traffic 104