The Netflix TechBlog

NOVEMBER 22, 2019

4:45pm-5:45pm NFX 202 A day in the life of a Netflix Engineer Dave Hahn , SRE Engineering Manager Abstract : Netflix is a large, ever-changing ecosystem serving millions of customers across the globe through cloud-based systems and a globally distributed CDN. In 2019, Netflix moved thousands of container hosts to bare metal.

AWS 37

AWS Entertainment Open Source Benchmarking 37

Brendan Gregg

JULY 4, 2021

I also wrote about these topics in detail for my recent [Systems Performance 2nd Edition] book. TCP Extensions for Multipath Operation with Multiple Addresses,” [link] Mar 2020 - [Gregg 20] Brendan Gregg, “Systems Performance: Enterprise and the Cloud, Second Edition,” Addison-Wesley, 2020 - [Hruska 20] Joel Hruska, “Intel Demos PCIe 5.0

Performance 52

Performance Latency Hardware Storage 52

Brendan Gregg

FEBRUARY 28, 2023

This talk originated from my updates to Systems Performance 2nd Edition , and this was the first time I've given this talk in person! CXL in a way allows a custom memory controller to be added to a system, to increase memory capacity, bandwidth, and overall performance. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

Brendan Gregg

AUGUST 29, 2021

1072-aws (xxx) 12/18/2018 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.03 avg-cpu: %user %nice %system %iowait %steal %idle 14.81 avg-cpu: %user %nice %system %iowait %steal %idle 14.86 avg-cpu: %user %nice %system %iowait %steal %idle 14.95 avg-cpu: %user %nice %system %iowait %steal %idle 14.54

Cache 105

Cache C++ AWS Latency 105

All Things Distributed

SEPTEMBER 25, 2017

The Region will be in the heart of Gulf Cooperation Council (GCC) countries, and we're aiming to have it ready by early 2019. This Region will consist of three Availability Zones at launch, and it will provide even lower latency to users across the Middle East. This news marks the 22nd AWS Region we have announced globally.

Cloud 152

Cloud Education Energy Government 152

Brendan Gregg

AUGUST 29, 2021

1072-aws (xxx) 12/18/2018 _x86_64_ (16 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 5.03 avg-cpu: %user %nice %system %iowait %steal %idle 14.81 avg-cpu: %user %nice %system %iowait %steal %idle 14.86 avg-cpu: %user %nice %system %iowait %steal %idle 14.95 avg-cpu: %user %nice %system %iowait %steal %idle 14.54

Cache 40

Cache C++ AWS Systems 40

O'Reilly

DECEMBER 2, 2019

I don’t need more bandwidth for video conferences or movies, but I would like to be able to download operating system updates and other large items in seconds rather than minutes. There are impressive estimates for latency for 5G, but reality has a tendency to be harsh on such predictions. Upcoming events.

Wireless 53

Wireless Serverless IoT Artificial Intelligence 53

Adrian Cockcroft

MAY 6, 2023

I don’t advocate “Serverless Only”, and I recommended that if you need sustained high traffic, low latency and higher efficiency, then you should re-implement your rapid prototype as a continuously running autoscaled container, as part of a larger serverless event driven architecture, which is what they did.

Serverless 145

Serverless Lambda Best Practices Traffic 145

The Netflix TechBlog

MARCH 1, 2021

It supports both high throughput services that consume hundreds of thousands of CPUs at a time, and latency-sensitive workloads where humans are waiting for the results of a computation. The first generation of this system went live with the streaming launch in 2007. Delivery?—?A

Serverless 228

Serverless Media Latency Social Media 228

KeyCDN

SEPTEMBER 12, 2019

This can be a problem in a lot of content management systems such as WordPress or Magento, because theme developers tend to scale images down in responsive themes using CSS. Well, according to HTTP Archive , as of June 1, 2019 the average desktop page is 1,896.8 It is usually recommended that you upload images at scale.

Social Media 106

Social Media Media Google Website 106

The Netflix TechBlog

MAY 14, 2021

However, they are scattered across multiple systems, and there isn’t an easy way to tie related messages together. You’re joining tables, resolving status types, cross-referencing data manually with other systems, and by the end of it all you ask yourself why? Things got hairy.

Traffic 160

Traffic Latency Java Google 160

O'Reilly

NOVEMBER 12, 2019

These are not system administrators whose job it is to worry about keeping software patched, secured, and managed. latency, startup, mocking, etc.) The results in Figure 12 reflect what we know of the cloud market and mirror what we found in our cloud native survey from earlier in 2019.

Serverless 87

Serverless Architecture FinTech Infrastructure 87

Tasktop

JUNE 18, 2019

In one week’s time, thousands of IT and business professionals will descend on London for the latest iteration of DevOps Enterprise Summit London 2019 (June 25-27 – InterContinental O2, London, UK). Here are four tips to get the most out of DOES London 2019: Tip #1 – Develop a plan of attack. The countdown is on.

DevOps 74

DevOps Network Software Software 74

The Netflix TechBlog

NOVEMBER 2, 2020

How viewers are able to watch their favorite show on Netflix while the infrastructure self-recovers from a system failure By Manuel Correa , Arthur Gonigberg , and Daniel West Getting stuck in traffic is one of the most frustrating experiences for drivers around the world. Those two metrics are approximate indicators of failures and latency.

Traffic 252

Traffic Metrics Infrastructure Architecture 252

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. CLI tools The Cassandra systems were EC2 virtual machine (Xen) instances. Microbenchmark os::javaTimeMillis() on both systems. Running this on the two systems saw similar results. Try changing the kernel clocksource.

Speed 126

Speed Java AWS Virtualization 126

The Morning Paper

NOVEMBER 8, 2020

While ultimately this new system should be able to take advantage of the latest advances in consensus for improved performance, that’s not realistic given a 6-9 month in-production target. It’s such a powerful idea that I can imagine distributed systems implementers everywhere adopting it from now on. What does the VirtualLog give us?

Virtualization 126

Virtualization Latency Storage Systems 126

Dynatrace

JUNE 25, 2021

Distributed tracing describes the act of following a transaction through all participating applications (tiers) and sub-systems, such as databases. All systems that support distributed tracing use some identifiers, the trace context, that is passed along with the transaction. W3C Trace Context. But how is this relationship represented?

Open Source 196

Open Source Monitoring Google Systems 196

High Scalability

DECEMBER 21, 2018

@swardley: X : What's going to happen in cloud in 2019? odrotbohm : I’ve seen microservice based systems more deranged after 2 years than any 1,5 decades old monolith could ever have been. Tim Bray : How to talk about [Serverless Latency] · To start with, don’t just say “I need 120ms.”

Internet 137

Internet Internet Scalability Serverless 137

The Netflix TechBlog

DECEMBER 17, 2019

In order to be supported, a database is required to fulfill a set of features that are commonly available in systems like MySQL, PostgreSQL, MariaDB, and others. We want to support these systems as a source so that they can provide their data for further consumption. Some of DBLog’s features are: Processes captured log events in-order.

Database 197

Database Traffic Transportation Open Source 197

The Netflix TechBlog

DECEMBER 17, 2019

In order to be supported, a database is required to fulfill a set of features that are commonly available in systems like MySQL, PostgreSQL, MariaDB, and others. We want to support these systems as a source so that they can provide their data for further consumption. Some of DBLog’s features are: Processes captured log events in-order.

Database 212

Database Traffic Transportation Open Source 212

The Morning Paper

JULY 4, 2019

Are you ready to take your system assurance programme to the next level? In all cases we need to be able to carefully monitor the impact on the system, and back out if things start going badly wrong. Netflix’s system is deployed on the public cloud as complex set of interacting microservices.

Latency 77

Latency Engineering Metrics Traffic 77

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. CLI tools The Cassandra systems were EC2 virtual machine (Xen) instances. Microbenchmark os::javaTimeMillis() on both systems. Running this on the two systems saw similar results. Try changing the kernel clocksource.

Speed 52

Speed Java AWS Virtualization 52

CSS - Tricks

NOVEMBER 1, 2019

The FCP distribution for the 10th, 50th and 90th percentile values as reported on August 1, 2019. TTI distribution for the 10th, 50th and 90th percentile values as reported on August 1, 2019. TTFB mobile speed distribution (CrUX, July 2019). FCP mobile speed distribution (CrUX, July 2019). First Contentful Paint.

Speed 46

Speed Mobile Metrics Scalability 46

CSS - Tricks

MARCH 16, 2020

module called fs (short for "file system"). We just need to remember to swap any colons ( : ) for underscores ( _ ) so it plays nice with the Windows file system. If your latency is higher than 50ms, users may perceive your app as laggy. == "/") { dirName = dirName + urlObj.pathname.replace(///g, "_"); }. replace(/:/g, "_")}.json`,

Google 82

Google Latency Website Metrics 82

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. CLI tools The Cassandra systems were EC2 virtual machine (Xen) instances. Microbenchmark os::javaTimeMillis() on both systems. Running this on the two systems saw similar results. Try changing the kernel clocksource.

Speed 40

Speed Java AWS Virtualization 40

The Morning Paper

JUNE 30, 2019

Today we’re jumping from HotOS topics of 2019, to hot topics of 1977! Here’s the set-up as relayed to me by Pat (with permission): At work, I am part of a good sized team working on a large system implementation. One of the very senior engineers with 25+ years experience mentioned a problem with the system.

Traffic 75

Traffic Latency Programming Scalability 75

Dynatrace

JANUARY 15, 2020

Having released this functionality in an Preview Release back in September 2019, we’re now happy to announce the General Availability of our Citrix monitoring extension. Citrix platform performance—optimize your Citrix landscape with insights into user load and screen latency per server. Dynatrace news. Citrix VDA. SAP server.

Latency 35

Latency Performance Virtualization Infrastructure 35

The Morning Paper

DECEMBER 10, 2019

HotStorage 2019. The global object space contains all objects (potentially across multiple systems), allowing persistent pointers to refer to data with long lifetimes and giving software the ability to operate directly on persistent structures. A tale of two abstractions: the case for object space , Bittman et al.,

Hardware 52

Hardware Virtualization Operating System Programming 52

High Scalability

JANUARY 24, 2019

The Smalltalk group is astonishingly insular, almost childlike, but is just now opening up, looking at other systems with open-eyed curiosity and fascination. George Dyson : The next revolution will be the rise of analog systems that can no longer be mastered by digital programming.

Internet 29

Internet Internet Scalability Games 29

All Things Distributed

DECEMBER 12, 2018

They can run applications in Sweden, serve end users across the Nordics with lower latency, and leverage advanced technologies such as containers, serverless computing, and more. For VR, this is a journey that is already one-third complete and expected to be finished by the end of 2019.

AWS 124

AWS Cloud Games Serverless 124

HammerDB

OCTOBER 18, 2019

So lets take an Ubuntu system with Platinum 8280 CPUs with the following Ubuntu OS, reboot and check the CPU configuration before running any tests. rc3-custom #1 SMP Mon Aug 12 14:07:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Welcome to Ubuntu 19.04 (GNU/Linux 5.3.0-rc3-custom So by default, the system boots into powersave.

Benchmarking 63

Benchmarking Performance Hardware C++ 63

The Morning Paper

JANUARY 26, 2020

This week we’ll be looking at a selection of papers from the 2019 edition of the ACM Symposium of Cloud Computing ( SoCC ). Reverb: speculative debugging for web applications , Netravali & Mickens, SOCC’19. candidate bug-fixes) during replay. candidate bug-fixes) during replay.

Programming 86

Programming Servers Network Latency 86

Alex Russell

MARCH 6, 2021

India has been the epicentre of smartphone growth in recent years, owing to the sheer size of its market and an accelerating shift away from feature phones which made up the majority of Indian mobile devices until as late as 2019. So what did $150USD fetch in 2019? The smooth, dulcet tones of 2019's Moto E6. " package.

Performance 124

Performance Network Cache Metrics 124

O'Reilly

JULY 20, 2021

In 2019, YouTube had to settle with the FTC for a $170 million fine for selling ads targeting children. There may be alarm systems. There have been cases of harassment, intimidation, and domestic abuse by people whose access should have been revoked: for example, an ex-partner turning off the heating system.

Google 121

Google Games Technology Technology 121

Enprowess

SEPTEMBER 7, 2021

Real-time software system – Collaboration tools used for video/audio conferencing, document writing, Chat applications, etc. It helps isolated bugs quickly and reduces system downtime. with its low latency I/O operations, gives the benefit of ‘No buffering’ to developers. Use cases of Node.js Micro-services. Key Features.

Open Source 52

Open Source Virtualization Programming Servers 52

SQL Performance

APRIL 8, 2019

They feature low latency, local NVMe storage that can directly leverage the 128 PCIe 3.0 On March 25, 2019, Microsoft announced the availability of higher performance, larger capacity managed disks for Azure VMs. Having multiple low latency local NVMe SSDs connected to those PCIe lanes will give you excellent storage performance.

Azure 42

Azure Virtualization Storage Benchmarking 42

The Morning Paper

SEPTEMBER 12, 2019

The Apache MADLib project is still going strong, and the recent (July 2019) 1.16 SQL provides a declarative programming interface, below which the system itself can figure out the most effective execution plans based on data size and statistics, layout, compute hardware etc. VLDB’19. Why would anyone want to do that???

Network 44

Network Database Programming Hardware 44