The Netflix TechBlog

AUGUST 30, 2021

System Setup Architecture The following diagram summarizes the architecture description: Figure 1: Event-sourcing architecture of the Device Management Platform. As such, we can see that the traffic load on the Device Management Platform’s control plane is very dynamic over time.

Latency 213

Latency Traffic Transportation Hardware 213

IO River

NOVEMBER 2, 2023

‍Just as a well-coordinated airport directs flights to multiple runways based on traffic and weather conditions, a CDN with Multiple Origins Load Balancing ensures that web traffic is distributed across various data centers, optimizing performance and reliability. ‍But how does it decide where to send this traffic?

Traffic 52

Traffic Cache Servers Latency 52

The Netflix TechBlog

OCTOBER 19, 2020

which is difficult when troubleshooting distributed systems. If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Dynatrace

APRIL 26, 2022

STM generates traffic that replicates the typical path or behavior of a user on a network to measure performance for example, response times, availability, packet loss, latency, jitter, and other variables). Endpoint monitoring (EM). Endpoints can be physical (i.e., PC, smartphone, server) or virtual (virtual machines, cloud gateways).

Monitoring 206

Monitoring Social Media IoT Metrics 206

IO River

NOVEMBER 2, 2023

Just as a well-coordinated airport directs flights to multiple runways based on traffic and weather conditions, a CDN with Multiple Origins Load Balancing ensures that web traffic is distributed across various data centers, optimizing performance and reliability. But how does it decide where to send this traffic?

Traffic 40

Traffic Cache Network Servers 40

Percona

MAY 1, 2023

I’ll show you some MySQL settings to tune to get better performance, and cost savings, with AWS RDS. Recently I was engaged in a MySQL Performance Audit for a customer to help troubleshoot performance issues that led to downtime during periods of high traffic on their AWS RDS MySQL instances.

AWS 124

AWS Hardware Storage Tuning 124

IO River

NOVEMBER 15, 2023

In technical terms, network-level firewalls regulate access by blocking or permitting traffic based on predefined rules. ‍At its core, WAF operates by adhering to a rulebook—a comprehensive list of conditions that dictate how to handle incoming web traffic. You've put new rules in place.

Traffic 52

Traffic Network Logistics Architecture 52

IO River

NOVEMBER 2, 2023

In technical terms, network-level firewalls regulate access by blocking or permitting traffic based on predefined rules. At its core, WAF operates by adhering to a rulebook—a comprehensive list of conditions that dictate how to handle incoming web traffic. You've put new rules in place.

Traffic 40

Traffic Network Logistics Architecture 40

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. CLI tools The Cassandra systems were EC2 virtual machine (Xen) instances. Microbenchmark os::javaTimeMillis() on both systems. Running this on the two systems saw similar results. Try changing the kernel clocksource.

Speed 126

Speed Java AWS Virtualization 126

Dotcom-Montior

NOVEMBER 11, 2020

Developers don’t have to put in additional time to fine-tuning the system, or rely on other teams for support, as it’s done automatically with the cloud provider. However, when the time comes for resources to be requested, there can be latency in the time it takes to for that code to start back up. Security & Privacy.

Serverless 57

Serverless Monitoring Lambda Latency 57

Percona

APRIL 17, 2023

In this blog post, we will discuss the best practices on the MongoDB ecosystem applied at the Operating System (OS) and MongoDB levels. The main objective of this post is to share my experience over the past years tuning MongoDB and centralize the diverse sources that I crossed in this journey in a unique place. 25.84 - Total 21.04

Best Practices 52

Best Practices Design Tuning Database 52

The Netflix TechBlog

FEBRUARY 18, 2022

To this end, we developed a Rapid Event Notification System (RENO) to support use cases that require server initiated communication with devices in a scalable and extensible manner. In this blog post, we will give an overview of the Rapid Event Notification System at Netflix and share some of the learnings we gained along the way.

Systems 334

Systems Traffic Architecture Mobile 334

The Netflix TechBlog

JUNE 14, 2023

The control group’s traffic utilized the legacy Falcor stack, while the experiment population leveraged the new GraphQL client and was directed to the GraphQL Shim. To determine customer impact, we could compare various metrics such as error rates, latencies, and time to render.

Traffic 353

Traffic Latency Cache Metrics 353

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. CLI tools The Cassandra systems were EC2 virtual machine (Xen) instances. Microbenchmark os::javaTimeMillis() on both systems. Running this on the two systems saw similar results. Try changing the kernel clocksource.

Speed 40

Speed Java AWS Virtualization 40

The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Processing 229

Processing Media Latency Innovation 229

The Netflix TechBlog

MAY 14, 2021

However, they are scattered across multiple systems, and there isn’t an easy way to tie related messages together. You’re joining tables, resolving status types, cross-referencing data manually with other systems, and by the end of it all you ask yourself why? Things got hairy. We wanted a scalable service that was near real-time, 2.

Traffic 160

Traffic Latency Java Google 160

The Netflix TechBlog

NOVEMBER 2, 2020

How viewers are able to watch their favorite show on Netflix while the infrastructure self-recovers from a system failure By Manuel Correa , Arthur Gonigberg , and Daniel West Getting stuck in traffic is one of the most frustrating experiences for drivers around the world. CRITICAL : This traffic affects the ability to play.

Traffic 252

Traffic Metrics Infrastructure Architecture 252

The Netflix TechBlog

AUGUST 13, 2020

You’re half awake and wondering, “Is there really a problem or is this just an alert that needs tuning? Our streaming teams need a monitoring system that enables them to quickly diagnose and remediate problems; seconds count! Our Node team needs a system that empowers a small group to operate a large fleet. By Andrei U.,

Monitoring 210

Monitoring Tuning Traffic Metrics 210

Brendan Gregg

SEPTEMBER 5, 2017

What follows are topics that may be of interest to anyone looking to migrate their systems and skillset: scan these to find topics that interest you. ## ZFS ZFS is available for Linux via the [zfsonlinux] and [OpenZFS] projects, and more recently was included in Canonical's Ubuntu Linux distribution: Ubuntu Xenial 16.04 LTS (April 2016).

Virtualization 76

Virtualization AWS Engineering Hardware 76

The Netflix TechBlog

AUGUST 29, 2023

To improve availability, we designed systems where components could fail separately and avoid single points of failure. In order for a service to talk to another, it needs to know two things: the name of the destination service, and whether or not the traffic should be secure. First, we’ve grown the number of different IPC clients.

Traffic 220

Traffic Latency Cloud C++ 220

The Netflix TechBlog

NOVEMBER 2, 2021

The coupling problem Until recently, video quality measurements were generated as part of our Reloaded production system. This system is responsible for processing incoming media files, such as video, audio and subtitles, and making them playable on the streaming service. We call this system Cosmos. The workflow is initiated.

Media 171

Media Innovation Metrics Latency 171

The Netflix TechBlog

JULY 26, 2021

This happens at an unprecedented scale and introduces many interesting challenges; one of the challenges is how to provide visibility of Studio data across multiple phases and systems to facilitate operational excellence and empower decision making. Please stay tuned! We will have follow up blog posts on these topics in future.

Big Data 253

Big Data Government Analytics Processing 253

The Netflix TechBlog

DECEMBER 17, 2019

Nonetheless, we found a number of limitations that could not satisfy our requirements e.g. stalling the processing of log events until a dump is complete, missing ability to trigger dumps on demand, or implementations that block write traffic by using table locks. Blocking write traffic by locking tables. Writing events to any output.

Database 212

Database Traffic Transportation Open Source 212

The Netflix TechBlog

DECEMBER 17, 2019

Nonetheless, we found a number of limitations that could not satisfy our requirements e.g. stalling the processing of log events until a dump is complete, missing ability to trigger dumps on demand, or implementations that block write traffic by using table locks. Blocking write traffic by locking tables. Writing events to any output.

Database 197

Database Traffic Transportation Open Source 197

All Things Distributed

OCTOBER 2, 2017

With these requirements in mind, and a willingness to question the status quo, a small group of distributed systems experts came together and designed a horizontally scalable distributed database that would scale out for both reads and writes to meet the long-term needs of our business. This was the genesis of the Amazon Dynamo database.

Internet 128

Internet Internet AWS Performance 128

Smashing Magazine

AUGUST 22, 2021

Because we are dealing with network protocols here, we will mainly look at network aspects, of which two are most important: latency and bandwidth. Latency can be roughly defined as the time it takes to send a packet from point A (say, the client) to point B (the server). Two-way latency is often called round-trip time (RTT).

Performance 105

Performance Network Latency Servers 105

Smashing Magazine

SEPTEMBER 6, 2021

Finally, not inlining resources has an added latency cost because the file needs to be requested. As discussed in part 2 , in HTTP/2, this multiplexing is steered using its prioritization system. As also explained in part 2 , HTTP/3 fundamentally changes the internals of this prioritization system. What Does It All Mean?

Network 104

Network Servers Cache Traffic 104