The Netflix TechBlog

NOVEMBER 22, 2019

In 2019, Netflix moved thousands of container hosts to bare metal. Netflix runs dozens of stateful services on AWS under strict sub-millisecond tail-latency requirements, which brings unique challenges. It launches more than four million containers per week across thousands of underlying hosts.

AWS 37

AWS Entertainment Open Source Benchmarking 37

The Morning Paper

OCTOBER 4, 2020

We are standing on the eve of the 5G era… 5G, as a monumental shift in cellular communication technology, holds tremendous potential for spurring innovations across many vertical industries, with its promised multi-Gbps speed, sub-10 ms low latency, and massive connectivity. Throughput and latency. km university campus.

Energy 130

Energy Latency Performance Network 130

Scalegrid

JUNE 18, 2019

The easiest way to induce failover is to run the rs.stepDown() command: RS-example-0:PRIMARY> rs.stepDown() 2019-04-18T19:44:42.257+0530 E QUERY [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host 'SG-example-1.servers.mongodirector.com:27017' 27017 (sg-example-17026.servers.mongodirector.com:27017,

C++ 100

C++ Servers Database Network 100

Brendan Gregg

JULY 4, 2021

## References I've reproduced the talk references below, so you can click on links: - [Gregg 08] Brendan Gregg, “ZFS L2ARC,” [link] Jul 2008 - [Gregg 10] Brendan Gregg, “Visualizations for Performance Analysis (and More),” [link] 2010 - [Greenberg 11] Marc Greenberg, “DDR4: Double the speed, double the latency? Ford, et al., “TCP

Performance 52

Performance Latency Hardware Storage 52

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

The Netflix TechBlog

OCTOBER 19, 2020

If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. Using simple lookup indices in Cassandra gives us the ability to maintain acceptable read latencies while doing heavy writes.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

John McCalpin

JANUARY 9, 2019

As part of my attempt to become organized in 2019, I found several draft blog entries that had never been completed and made public.

Latency 40

Latency Architecture Servers Performance 40

High Scalability

MARCH 1, 2019

It was made possible by using a low latency of 0.1 seconds, the lower the latency, the more responsive the robot. They'll learn a lot and love you forever. AWSonAir : @McDonalds uses Amazon ECS to scale to support 20,000 orders per second. antoniogm : Know why the European startup scene sucks?

Internet 33

Internet Internet Scalability Blockchain 33

High Scalability

MAY 10, 2019

Quotable Stuff: @mjpt777 : APIs to IO need to be asynchronous and support batching otherwise the latency of calls dominate throughput and latency profile under burst conditions. . $84.4 : average yearly Facebook ad revenue per user in North America.

Internet 31

Internet Internet Scalability Energy 31

KeyCDN

JANUARY 31, 2019

The next closest active POP location to Bucharest was Istanbul which was still almost 900km away; this distance adds up in terms of latency. Before the implementation of our Bucharest POP, Romanian users were delivered content from our surrounding KeyCDN POPs.

Internet 48

Internet Internet Ecommerce Speed 48

All Things Distributed

SEPTEMBER 25, 2017

The Region will be in the heart of Gulf Cooperation Council (GCC) countries, and we're aiming to have it ready by early 2019. This Region will consist of three Availability Zones at launch, and it will provide even lower latency to users across the Middle East. This news marks the 22nd AWS Region we have announced globally.

Cloud 152

Cloud Education Energy Government 152

Brendan Gregg

AUGUST 29, 2021

biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. As I mentioned in my LSFMMBPF 2019 keynote in Puerto Rico, where the Linux mm kernel engineers were present, I think the cachestat statistics are so commonly needed that they should be in /proc and not need my experimental tool. Tracing block device I/O.

Cache 105

Cache C++ AWS Latency 105

Smashing Magazine

APRIL 17, 2019

2019-04-17T12:30:16+02:00. 2019-04-29T18:34:58+00:00. This typically happens once per server and takes up valuable time — especially if the server is very distant from the browser and network latency is high. Optimizing Performance With Resource Hints. Optimizing Performance With Resource Hints. Drew McLellan.

Performance 106

Performance Games Servers Cache 106

O'Reilly

DECEMBER 2, 2019

For applications like communication between AVs, latency–how long it takes to get a response–is more likely to be a bigger limitation than raw bandwidth, and is subject to limits imposed by physics. There are impressive estimates for latency for 5G, but reality has a tendency to be harsh on such predictions. Upcoming events.

Wireless 53

Wireless Serverless IoT Artificial Intelligence 53

Adrian Cockcroft

MAY 6, 2023

I don’t advocate “Serverless Only”, and I recommended that if you need sustained high traffic, low latency and higher efficiency, then you should re-implement your rapid prototype as a continuously running autoscaled container, as part of a larger serverless event driven architecture, which is what they did.

Serverless 145

Serverless Lambda Best Practices Traffic 145

Rigor

JULY 2, 2019

By adding the need for additional JavaScript resources to your page, you increase the latency caused by the need to first download the webpage and then parse and execute the JavaScript before the browser can execute the redirect. Originally published September 2016, updated July 2019. REQUEST A FREE TRIAL OF RIGOR.

Mobile 60

Mobile Traffic Google Latency 60

O'Reilly

NOVEMBER 12, 2019

latency, startup, mocking, etc.) The results in Figure 12 reflect what we know of the cloud market and mirror what we found in our cloud native survey from earlier in 2019. 1] The serverless adoption survey ran in June 2019. “Integration/testing is harder” ranked as the third biggest worry, noted by 30% of respondents.

Serverless 88

Serverless Architecture FinTech Infrastructure 88

The Netflix TechBlog

MARCH 1, 2021

It supports both high throughput services that consume hundreds of thousands of CPUs at a time, and latency-sensitive workloads where humans are waiting for the results of a computation. The subsystems all communicate with each other asynchronously via Timestone, a high-scale, low-latency priority queuing system. Warm capacity.

Serverless 228

Serverless Media Latency Social Media 228

Brendan Gregg

AUGUST 29, 2021

biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. As I mentioned in my LSFMMBPF 2019 keynote in Puerto Rico, where the Linux mm kernel engineers were present, I think the cachestat statistics are so commonly needed that they should be in /proc and not need my experimental tool.

Cache 40

Cache C++ AWS Systems 40

Scalegrid

NOVEMBER 25, 2019

ScyllaDB offers significantly lower latency which allows you to process a high volume of data with minimal delay. percentile latency is up to 11X better than Cassandra on AWS EC2 bare metal. This number is more inline with our recent 2019 Open Source Database Trends Report where 56.9% Databases Most Commonly Used with ScyllaDB.

Big Data 187

Big Data Database Open Source Azure 187

Tasktop

JUNE 18, 2019

In one week’s time, thousands of IT and business professionals will descend on London for the latest iteration of DevOps Enterprise Summit London 2019 (June 25-27 – InterContinental O2, London, UK). Here are four tips to get the most out of DOES London 2019: Tip #1 – Develop a plan of attack. The countdown is on.

DevOps 74

DevOps Network Software Software 74

KeyCDN

MAY 14, 2019

This allows for global processing, which means no matter where your users are located they will receive processed images with low latency. Our image processing is advantageous because it combines high performance image transformation and optimization with our global CDN. This is achieved by defining the applicable image processing parameters.

Processing 13

Processing Cache Latency Network 13

KeyCDN

SEPTEMBER 12, 2019

Well, according to HTTP Archive , as of June 1, 2019 the average desktop page is 1,896.8 The main reason is because it decreases the latency to the user where they are located by serving your images from a POP physically closest to them. Image Compression Just how much of a website is made up of images?

Social Media 106

Social Media Media Google Website 106

The Netflix TechBlog

MAY 14, 2021

We are expected to process 1,000 watermarks for a single distribution in a minute, with non-linear latency growth as the number of watermarks increases. The goal is to process these documents as fast as possible and reliably deliver them to recipients while offering strong observability to both our users and internal teams.

Traffic 160

Traffic Latency Java Google 160

The Netflix TechBlog

NOVEMBER 2, 2020

Those two metrics are approximate indicators of failures and latency. Netflix experienced a similar issue with the same potential impact as the outage seen in 2019. Service throttling Zuul can sense when a back-end service is in trouble by monitoring the error rates and concurrent requests to that service.

Traffic 252

Traffic Metrics Infrastructure Architecture 252

CSS Wizardry

MARCH 3, 2019

— Harry Roberts (@csswizardry) 3 March, 2019. If, however, there wasn’t a new file on the server, we’ll bring back a 304 header, no new file, but an entire roundtrip of latency. We can completely cut out the overhead of a roundtrip of latency. On high latency connections, this saving could be tangible.

Cache 264

Cache Latency Strategy Servers 264

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. The change was obvious in the production graphs, showing a drop in write latencies: Once tested more broadly, it showed the write latencies dropped by 43%, delivering slightly better performance than on CentOS.

Speed 126

Speed Java AWS Virtualization 126

High Scalability

DECEMBER 21, 2018

swardley: X : What's going to happen in cloud in 2019? Tim Bray : How to talk about [Serverless Latency] · To start with, don’t just say “I need 120ms.” And if you know someone with hearing problems they might find Live CC useful. 202,157 flights tracked! Me : Nothing special. 3) Serverless will rocket.

Internet 137

Internet Internet Scalability Serverless 137

The Morning Paper

NOVEMBER 8, 2020

The initial version of Delos went into production after eight months using a ZooKeeper-backed Loglet implementation, and then four months later it was swapped out for a new custom-built NativeLoglet that gave a 10x improvement in end-to-end latency. For Facebook’s Delos, reconfiguration latencies of 10s of ms are ok.

Virtualization 126

Virtualization Latency Storage Systems 126

CSS - Tricks

NOVEMBER 1, 2019

The FCP distribution for the 10th, 50th and 90th percentile values as reported on August 1, 2019. TTI distribution for the 10th, 50th and 90th percentile values as reported on August 1, 2019. TTFB mobile speed distribution (CrUX, July 2019). FCP mobile speed distribution (CrUX, July 2019). First Contentful Paint.

Speed 46

Speed Mobile Metrics Scalability 46

Dynatrace

JUNE 25, 2021

Already in the 2000s, service-oriented architectures (SOA) became popular, and operations teams discovered the need to understand how transactions traverse through all tiers and how these tiers contributed to the execution time and latency. In 2019, the OpenCensus and OpenTracing projects merged into what we now know as OpenTelemetry.

Open Source 197

Open Source Monitoring Google Systems 197

CSS - Tricks

MARCH 16, 2020

Estimated input latency - Estimated Input Latency is an estimate of how long your app takes to respond to user input, in milliseconds, during the busiest 5s window of page load. If your latency is higher than 50ms, users may perceive your app as laggy. How should the metric comparison be output to the console?

Google 82

Google Latency Website Metrics 82

Alex Russell

DECEMBER 18, 2022

Without beating around the bush, our ASP 2019 device was an Android that cost between $300-$350, new and unlocked. We've been tracking the mobile device landscape more carefully over the years and, as with desktop, ASP s today are tomorrow's performance destiny. But this also gives rise to the critique: OK, but does it work?

Performance 113

Performance Network Mobile Latency 113

The Morning Paper

JULY 4, 2019

Two failure modes we focus on are a service becoming slower (increase in response latency) or a service failing outright (returning errors). The criticality score is combined with a safety score and experiment weight (failure experiments, then latency, than failure inducing latency) to produce the final prioritization score.

Latency 77

Latency Engineering Metrics Traffic 77

The Netflix TechBlog

DECEMBER 17, 2019

Passive instances across regions are also possible, though it is recommended to operate in the same region as the database host in order to keep the change capture latencies low. Production usage DBLog is the foundation of the MySQL and PostgreSQL Connectors at Netflix, which are used in Delta. Beresford, and Boerge Svingen.

Database 197

Database Traffic Transportation Open Source 197

The Netflix TechBlog

DECEMBER 17, 2019

Passive instances across regions are also possible, though it is recommended to operate in the same region as the database host in order to keep the change capture latencies low. Production usage DBLog is the foundation of the MySQL and PostgreSQL Connectors at Netflix, which are used in Delta. Beresford, and Boerge Svingen.

Database 212

Database Traffic Transportation Open Source 212

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. The change was obvious in the production graphs, showing a drop in write latencies: Once tested more broadly, it showed the write latencies dropped by 43%, delivering slightly better performance than on CentOS.

Speed 52

Speed Java AWS Virtualization 52