The Netflix TechBlog

JUNE 13, 2023

Migrating Critical Traffic At Scale with No Downtime — Part 2 Shyam Gala , Javier Fernandez-Ivern , Anup Rokkam Pratap , Devang Shah Picture yourself enthralled by the latest episode of your beloved Netflix series, delighting in an uninterrupted, high-definition streaming experience. This is where large-scale system migrations come into play.

Traffic 279

Traffic Metrics Systems Strategy 279

KeyCDN

NOVEMBER 4, 2021

The POP is strategially located within the country and lowers latency overall. KeyCDN is always on the lookout for ways to minimize latency and accelerate asset delivery worldwide. With this new POP, our performance in Latin America has significantly improved and provides a better web experience.

Latency 111

Latency Tuning Cache Traffic 111

Dynatrace

APRIL 26, 2022

Understanding why a user is experiencing transactional or performance issues enables organizations to achieve greater observability that goes beyond metrics, traces and logs. It is proactive monitoring that simulates traffic with established test variables, including location, browser, network, and device type.

Monitoring 200

Monitoring Social Media IoT Metrics 200

The Netflix TechBlog

AUGUST 30, 2021

Canary Test Workloads In addition to serving the regular message traffic between users and DUTs, the control plane itself is stress-tested at roughly 3-hour intervals, where nearly 3000 ephemeral MQTT clients are created to connect to and generate flash traffic on the MQTT brokers. million elements.

Latency 213

Latency Traffic Transportation Hardware 213

Percona

MAY 1, 2023

Default settings can help you get started quickly – but they can also cost you performance and a higher cloud bill at the end of the month. I’ll show you some MySQL settings to tune to get better performance, and cost savings, with AWS RDS. Want to save money on your AWS RDS bill? The settings might not be optimal.

AWS 122

AWS Hardware Storage Tuning 122

KeyCDN

NOVEMBER 29, 2022

KeyCDN is always looking for ways to minimize latency and accelerate the delivery of assets worldwide. With these new POPs, our performance in Latin America has improved significantly, providing a better web experience. Our mission is to improve web performance continuously, and each POP contributes to that. What's next?

Latency 60

Latency Tuning Cache Traffic 60

The Netflix TechBlog

OCTOBER 19, 2020

If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. Our engineering teams tuned their services for performance after factoring in increased resource utilization due to tracing.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Smashing Magazine

NOVEMBER 8, 2021

As developers, we rightfully obsess about the customer experience, relentlessly working to squeeze every millisecond out of the critical rendering path, optimize input latency, and eliminate jank. We work with millions of merchants, and as many companies, we obsess about storefront performance. Stay tuned for more in 2022!

Cache 135

Cache Best Practices Strategy Servers 135

Dotcom-Montior

NOVEMBER 11, 2020

Developers don’t have to put in additional time to fine-tuning the system, or rely on other teams for support, as it’s done automatically with the cloud provider. For workloads that require high performance, organizations would be better suited by purchasing their own servers. Focus on Application Development. Response Times.

Serverless 57

Serverless Monitoring Lambda Latency 57

KeyCDN

AUGUST 15, 2019

Web performance is a broad subject, and you’ll find no shortage of performance testing tips and tutorials all over the web. Before you begin tuning your website or application, you must first figure out which metrics matter most to your users and establish some achievable benchmarks. What is Performance Testing?

Testing Tools 55

Testing Tools Best Practices Performance Testing Testing 55

The Netflix TechBlog

JUNE 14, 2023

This blog post will share broadly-applicable techniques (beyond GraphQL) we used to perform this migration. The control group’s traffic utilized the legacy Falcor stack, while the experiment population leveraged the new GraphQL client and was directed to the GraphQL Shim. The Replay Tester tool samples raw traffic streams from Mantis.

Traffic 353

Traffic Latency Cache Metrics 353

Dynatrace

MAY 17, 2023

Making applications observable—relying on metrics, logs, and traces to understand what software is doing and how it’s performing—has become increasingly important as workloads are shifting to multicloud environments. This will get us straight to the application page, where we get more insight on how our front end actually performs.

Metrics 168

Metrics Monitoring Database Network 168

Dynatrace

JULY 24, 2023

Observability data provides a treasure trove of performance, stability, and user experience metrics encompassing error rates, response times, and user engagement. For instance, in the case of poor performance, you can seamlessly toggle a feature flag and mitigate any detrimental effects.

DevOps 217

DevOps Traffic Efficiency Servers 217

The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. We expanded pipeline support to serve our studio/content-development use cases, which had different latency and resiliency requirements as compared to the traditional streaming use case. divide the input video into small chunks 2.

Processing 229

Processing Media Latency Innovation 229

The Netflix TechBlog

NOVEMBER 2, 2020

How viewers are able to watch their favorite show on Netflix while the infrastructure self-recovers from a system failure By Manuel Correa , Arthur Gonigberg , and Daniel West Getting stuck in traffic is one of the most frustrating experiences for drivers around the world. Logs and background requests are examples of this type of traffic.

Traffic 252

Traffic Metrics Infrastructure Architecture 252

The Netflix TechBlog

MAY 14, 2021

Prodicle Distribution Our service is required to be elastic and handle bursty traffic. We are expected to process 1,000 watermarks for a single distribution in a minute, with non-linear latency growth as the number of watermarks increases. We expect the performance and scaling to continue to get better without much effort on our part.

Traffic 160

Traffic Latency Java Google 160

The Netflix TechBlog

FEBRUARY 18, 2022

By: Ankush Gulati , David Gevorkyan Additional credits: Michael Clark , Gokhan Ozer Intro Netflix has more than 220 million active members who perform a variety of actions throughout each session, ranging from renaming a profile to watching a title. This helps limit the outgoing traffic footprint considerably.

Systems 334

Systems Traffic Architecture Mobile 334

The Netflix TechBlog

JULY 26, 2021

Operational Reporting is a reporting paradigm specialized in covering high-resolution, low-latency data sets, serving detailed day-to-day activities¹ and processes of a business domain. The compaction process is needed to optimize the performance of downstream queries on the business view as well as lower costs of S3 GET OBJECT operations.

Big Data 253

Big Data Government Analytics Processing 253

IO River

NOVEMBER 2, 2023

‍Just as a well-coordinated airport directs flights to multiple runways based on traffic and weather conditions, a CDN with Multiple Origins Load Balancing ensures that web traffic is distributed across various data centers, optimizing performance and reliability. ‍But how does it decide where to send this traffic?

Traffic 52

Traffic Cache Servers Latency 52

The Netflix TechBlog

NOVEMBER 2, 2021

Perceptual quality measurements are used to drive video encoding optimizations , perform video codec comparisons , carry out A/B testing and optimize streaming QoE decisions to mention a few. This enables us to use our scale to increase throughput and reduce latencies. VQS is called using the measureQuality endpoint.

Media 171

Media Innovation Metrics Latency 171

Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you _time_ time?

Speed 126

Speed Java AWS Virtualization 126

Smashing Magazine

AUGUST 22, 2021

HTTP/3: Performance Improvements (Part 2). HTTP/3: Performance Improvements (Part 2). In this second part, we will zoom in on the performance improvements that QUIC and HTTP/3 bring to the table for web-page loading. HTTP/3 performance features ( current article ). Two-way latency is often called round-trip time (RTT).

Performance 105

Performance Network Latency Servers 105

IO River

NOVEMBER 2, 2023

Just as a well-coordinated airport directs flights to multiple runways based on traffic and weather conditions, a CDN with Multiple Origins Load Balancing ensures that web traffic is distributed across various data centers, optimizing performance and reliability. But how does it decide where to send this traffic?

Traffic 40

Traffic Cache Network Servers 40

IO River

NOVEMBER 15, 2023

In technical terms, network-level firewalls regulate access by blocking or permitting traffic based on predefined rules. ‍At its core, WAF operates by adhering to a rulebook—a comprehensive list of conditions that dictate how to handle incoming web traffic. You've put new rules in place. Let's dive deep into these challenges.‍1.

Traffic 52

Traffic Network Logistics Architecture 52

All Things Distributed

OCTOBER 2, 2017

We were pushing the limits of what was a leading commercial database at the time and were unable to sustain the availability, scalability and performance needs that our growing Amazon business demanded. Performant – The service would need to be able to maintain consistent performance in the face of diverse customer workloads.

Internet 128

Internet Internet AWS Performance 128

Percona

APRIL 17, 2023

The main objective of this post is to share my experience over the past years tuning MongoDB and centralize the diverse sources that I crossed in this journey in a unique place. The CFQ works well for many general use cases but lacks latency guarantees. The goal is to reduce memory usage without impacting query performance negatively.

Best Practices 52

Best Practices Design Tuning Database 52

IO River

NOVEMBER 2, 2023

In technical terms, network-level firewalls regulate access by blocking or permitting traffic based on predefined rules. At its core, WAF operates by adhering to a rulebook—a comprehensive list of conditions that dictate how to handle incoming web traffic. You've put new rules in place. Let's dive deep into these challenges.‍1.

Traffic 40

Traffic Network Logistics Architecture 40

KeyCDN

MARCH 10, 2021

It's our mission to continuously improve web performance and each POP contributes to it. The image below shows a significant drop in latency once we've launched the new point of presence in Israel. The image below shows a significant drop in latency once we've launched the new point of presence in Israel. With a population of 2.5

Latency 110

Latency Internet Internet Tuning 110

Brendan Gregg

SEPTEMBER 25, 2021

These strange questions came to the fore back in 2014 when Netflix was switching services from CentOS Linux to Ubuntu, and I helped debug several weird performance issues including one I'll describe here. A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. How would you _time_ time?

Speed 40

Speed Java AWS Virtualization 40

Smashing Magazine

AUGUST 9, 2021

You’ve probably heard things like: “HTTP/3 is much faster than HTTP/2 when there is packet loss”, or “HTTP/3 connections have less latency and take less time to set up”, and probably “HTTP/3 can send data more quickly and can send more resources in parallel”. HTTP/3 performance features ( coming up soon! ).

Transportation 145

Transportation Internet Internet Network 145

Smashing Magazine

SEPTEMBER 6, 2021

If after the previous two parts — HTTP/3 history and core concepts and HTTP/3 performance features — you’re convinced that starting to use the new protocols is a good idea (and you should be!), HTTP/3 performance features. Finally, not inlining resources has an added latency cost because the file needs to be requested.

Network 104

Network Servers Cache Traffic 104

Brendan Gregg

SEPTEMBER 5, 2017

Here's some output from my zfsdist tool, in bcc/BPF, which measures ZFS latency as a histogram on Linux: # zfsdist. Tracing ZFS operation latency. As a performance engineer who does kernel tracing, I find the lack of an ID I can trace in the kernel to be pretty annoying. Hit Ctrl-C to end. ^C People rarely create them manually.

Virtualization 76

Virtualization AWS Engineering Hardware 76