The Netflix TechBlog

MARCH 4, 2024

The rule-based classifier classifies job errors based on a set of predefined rules and provides insights for schedulers to decide whether to retry the job and for engineers to diagnose and remediate the job failure. Rule Execution Engine is responsible for matching the collected logs against a set of predefined rules.

Tuning 210

Tuning Efficiency Big Data Engineering 210

The Netflix TechBlog

SEPTEMBER 8, 2020

This allowed Android engineers to have much more control and observability over how we get our data. For each route we migrated, we wanted to make sure we were not introducing any regressions: either in the form of missing (or worse, wrong) data, or by increasing the latency of each endpoint. This meant that data that was static (e.g.

Latency 233

Latency Cache Java Traffic 233

Alex Russell

FEBRUARY 28, 2022

The chief effect of the architectural difference is to shift the distribution of latency within the loop. Herein lies the source of our collective anxiety about front-end architectures: traversing networks is always fraught, but the costs to deliver client-side logic to cushion users from variable network latency remain stubbornly high.

Performance 104

Performance Latency Architecture Network 104

The Netflix TechBlog

JUNE 4, 2019

Because microprocessors are so fast, computer architecture design has evolved towards adding various levels of caching between compute units and the main memory, in order to hide the latency of bringing the bits to the brains. This avoids thrashing caches too much for B and evens out the pressure on the L3 caches of the machine.

Cache 251

Cache Latency Airlines Logistics 251

Dynatrace

AUGUST 27, 2019

Welcome back to the blog series in which we share how you can easily solve three common problem scenarios by using Dynatrace and xMatters Flow Designer. This is where xMatters Flow Designer comes into play, by automating remediation steps at the touch of a button. Step 5 – xMatters triggers a runbook in Ansible to fix the disk latency.

Systems 176

Systems DevOps Latency Azure 176

Brendan Gregg

JULY 4, 2021

I summarized these topics and more as a plenary conference talk, including my own predictions (as a senior performance engineer) for the future of computing performance, with a focus on back-end servers. Ford, et al., “TCP

Performance 140

Performance Latency Hardware Storage 140

Dynatrace

SEPTEMBER 13, 2023

This is a set of best practices and guidelines that help you design and operate reliable, secure, efficient, cost-effective, and sustainable systems in the cloud. These workflows also utilize Davis® , the Dynatrace causal AI engine, and all your observability and security data across all platforms, in context, at scale, and in real-time.

AWS 252

AWS Efficiency Azure Cloud 252

Randon ASCII

JANUARY 12, 2022

That meant I started having regular meetings with the hardware engineers who were working with IBM on the CPU which gave me even more expertise on this CPU, which was critical in helping me discover a design flaw in one of its instructions , and in helping game developers master this finicky beast. So, anyway. Standard stuff.

Cache 126

Cache Latency Benchmarking Hardware 126

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which Now let’s look at how we designed the tracing infrastructure that powers Edgar. Now let’s look at how we designed the tracing infrastructure that powers Edgar.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 110

Performance Latency Cache Virtualization 110

Dynatrace

DECEMBER 15, 2022

ITOps refers to the process of acquiring, designing, deploying, configuring, and maintaining equipment and services that support an organization’s desired business outcomes. This includes response time, accuracy, speed, throughput, uptime, CPU utilization, and latency. Performance. What does IT operations do?

Artificial Intelligence 208

Artificial Intelligence DevOps Hardware Virtualization 208

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! In this talk, we share how Netflix deploys systems to meet its demands, Ceph’s design for high availability, and results from our benchmarking. We look forward to seeing you there!

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! In this talk, we share how Netflix deploys systems to meet its demands, Ceph’s design for high availability, and results from our benchmarking. We look forward to seeing you there!

AWS 100

AWS Entertainment Open Source Benchmarking 100

Dynatrace

APRIL 5, 2021

You can eliminate the latency issues caused by cold starts — an increase in normal response time when a new instance receives its first request — by using edge-optimized functions that run code closer to users and other projects. AWS continues to improve how it handles latency issues. It helps SRE teams automate responses.

Lambda 183

Lambda AWS Serverless Hardware 183

CSS Wizardry

MAY 19, 2020

This is fast, incredibly well suited to the device in question, and has almost zero engineering overhead. On a high-latency connection, this spells bad news. However, the execution of this header is bound by the response’s TTFB, which on high-latency networks can be very, very high. To hell with it! Let’s have a web font!

Google 364

Google Media Latency Metrics 364

Dynatrace

DECEMBER 2, 2021

SLOs can be a great way for DevOps and infrastructure teams to use data and performance expectations to make decisions, such as whether to release, and where engineers should focus their time. You can set SLOs based on individual indicators, such as batch throughput, request latency, and failures-per-second. Help with decision making.

Metrics 196

Metrics Best Practices DevOps Infrastructure 196

The Morning Paper

JANUARY 30, 2020

Edge servers are the middle ground – more compute power than a mobile device, but with latency of just a few ms. physics engine that simulates 3D cubes falling from the air. Why would we want to live migrate web workers? The kind of edge server envisaged here might, for example, be integrated with your WiFi access point.

Mobile 104

Mobile Cloud Latency Games 104

Dotcom-Montior

NOVEMBER 16, 2021

In this article, we will take a deeper dive into the various SRE principles and guidelines that a site reliability engineer practices in their role. Google was the first company to create, embrace, and put support behind the role of site reliability engineering. Release Engineering. Release engineering.

Monitoring 91

Monitoring Google DevOps Engineering 91

InfoQ

JULY 3, 2023

The new solution achieved over 99% hit rate, helped reduce tail latencies by more than 60% and costs by 10% annually. LinkedIn introduced Couchbase as a centralized caching tier for scaling member profile reads to handle increasing traffic that has outgrown their existing database cluster. By Rafal Gancarz

Cache 82

Cache Latency Traffic Database 82

Dynatrace

APRIL 26, 2022

STM generates traffic that replicates the typical path or behavior of a user on a network to measure performance for example, response times, availability, packet loss, latency, jitter, and other variables). One use case for STM is to model the behavior of a customer in the form of a flow of transactions along the buyer’s journey.

Monitoring 209

Monitoring Social Media IoT Metrics 209

InfoQ

AUGUST 10, 2020

After analyzing the performance of several common lossless compression algorithms, Dropbox engineers modified slightly Google's Brotli encoder to improve their engine sync performance. This reduced median latency and data transfer by more than 30%, Dropbox engineers Rishabh Jain and Daniel Reiter Horn maintain.

Performance 65

Performance Latency Google Engineering 65

The Morning Paper

OCTOBER 6, 2019

Prediction serving latency matters. Lesson 4: prediction serving latency matters. In a experiment introducing synthetic latency, Booking.com found that an increase of about 30% in latency cost about 0.5% Even mathematically simple models have the potential of introducing relevant latency.

Latency 102

Latency Metrics Cache Design 102

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

Percona

MAY 2, 2023

At the same time that I see database engineers relying on the tool, sites such as StackOverflow are banning ChatGPT. Also, the debate with other engineers allows you to gather other points of view, which improves the solution to a point that is nowhere near ChatGPT can reach. Let’s make it harder now. The answer could be better.

Social Media 119

Social Media Database Servers Cache 119

Brendan Gregg

JULY 4, 2021

I summarized these topics and more as a plenary conference talk, including my own predictions (as a senior performance engineer) for the future of computing performance, with a focus on back-end servers. Ford, et al., “TCP

Performance 52

Performance Latency Hardware Storage 52

The Morning Paper

OCTOBER 11, 2020

Orbital Edge Computing (OEC) is designed to do just that. The simplified general perturbation model (SGP4) is used as the orbital mechanics engine. Close-spaced constellation have much lower effective bandwidth, but also much lower latency. Introducing Orbital Edge Computing (OEC). SPG4 is the GPS of space!

Systems 125

Systems Latency Architecture Energy 125

High Scalability

AUGUST 17, 2018

12 million requests / hour with sub-second latency, ~300GB of throughput / day. allspaw : engineer: “Unless you’re familiar with Lamport, Brewer, Fox, Armstrong, Stonebraker, Parker, Shapiro.(and They'll love you even more. million : lost in ATM malware hack; $1.5 Its running in 3 nodes that cost us about $150 / month.

Internet 105

Internet Internet Scalability Games 105

Brendan Gregg

AUGUST 18, 2019

For example, iostat(1), or a monitoring agent, may tell you your average disk latency, but not the distribution of this latency. For smaller environments, it can be of more use helping eliminate latency outliers. bpftrace uses BPF (Berkeley Packet Filter), an in-kernel execution engine that processes a virtual instruction set.

Latency 68

Latency C++ Cache Programming 68

The Netflix TechBlog

SEPTEMBER 3, 2021

Remote calls are never free; they impose extra latency, increase probability of an error, and consume network bandwidth. How can we achieve a similar functionality when designing our gRPC APIs? The solution we use within the Netflix Studio Engineering is protobuf FieldMask.

Design 245

Design Java Efficiency Code 245

Smashing Magazine

AUGUST 3, 2021

Fortunately, we as engineers can avoid, or at least mitigate the impact of breakages in the web apps we build. On design systems, CSS/JS and UX. Engineers need visibility on the root cause behind a degraded experience. Even errors not surfaced to end-users (secondary errors) must propagate to engineers. More after jump!

Network 145

Network Engineering Availability Monitoring 145

Dotcom-Montior

DECEMBER 8, 2021

In the world of a site reliability engineer (SRE) , failure is not only an option, but also expected. In a different article, we talked about chaos engineering and how SRE teams proactively seek out and test for failures to prevent the worst from happening. ITSM defines how teams design, create, and deliver their services.

Social Media 90

Social Media Monitoring Latency DevOps 90

The Morning Paper

JUNE 13, 2019

Making queries to an inference engine has many of the same throughput, latency, and cost considerations as making queries to a datastore, and more and more applications are coming to depend on such queries. The following figure highlights how just one of these variables, batch size, impacts throughput and latency on ResNet50.

Hardware 58

Hardware Latency Serverless Energy 58

Smashing Magazine

NOVEMBER 8, 2021

As developers, we rightfully obsess about the customer experience, relentlessly working to squeeze every millisecond out of the critical rendering path, optimize input latency, and eliminate jank. Continue reading below ↓ Meet Smart Interface Design Patterns , with Vitaly Friedman. Ilya Grigorik. 2021-11-08T14:30:00+00:00.

Cache 135

Cache Best Practices Strategy Servers 135

Abhishek Tiwari

JULY 12, 2018

These data pipelines can process data at petabytes scale and to some extent, their success can be attributed to an army of engineers devoted to build and maintain internal data pipelines. Not everyone is operating at Netflix or Spotify scale data engineering function.

Latency 63

Latency Analytics Scalability Engineering 63

Dynatrace

SEPTEMBER 30, 2021

This can dramatically decrease network latency and its effect on the end-user experience. By establishing these, you can work backward to ensure every step of the process is designed to serve these outcomes. As a result, organizations are seeing improved availability and performance. Establish a clear set of goals and outcomes.

Cloud 169

Cloud Traffic Best Practices Strategy 169

All Things Distributed

SEPTEMBER 25, 2017

This Region will consist of three Availability Zones at launch, and it will provide even lower latency to users across the Middle East. For the new AWS infrastructure Region we will also be hiring datacenter engineers, support engineers, engineering operations managers, security specialists, and many more.

Cloud 154

Cloud Education Energy Government 154

The Netflix TechBlog

FEBRUARY 9, 2021

The Growth Engineering team is responsible for executing growth initiatives that help us anticipate and adapt to this change. In particular, it’s our job to design and build the systems and protocols that enable customers from all over the world to sign up for Netflix with the plan features and incentives that best suit their needs.

Engineering 142

Engineering Scalability Architecture Innovation 142