The Netflix TechBlog

MARCH 4, 2024

Operational automation–including but not limited to, auto diagnosis, auto remediation, auto configuration, auto tuning, auto scaling, auto debugging, and auto testing–is key to the success of modern data platforms. We have also noted a great potential for further improvement by model tuning (see the section of Rollout in Production).

Tuning 210

Tuning Efficiency Big Data Engineering 210

Percona

NOVEMBER 2, 2023

Improved performance : MongoDB continually fine-tunes its database engine, resulting in faster query execution and reduced latency. Safeguarding your data, testing in a controlled environment, and having rollback plans are all part of this stage. We walk you through the essential steps required.

Best Practices 86

Best Practices Hardware Tuning Scalability 86

Dynatrace

MAY 5, 2021

Keptn closes the loop of planning, testing, deployment, and analysis in Agile-like environments with the help of quality gates defined by service- and business-level indicators. For example, improving latency by as little as 0.1 latency is the number one reason consumers abandon mobile sites. Meanwhile, in the U.S.,

Benchmarking 183

Benchmarking Latency Speed Software 183

Scalegrid

DECEMBER 14, 2023

Having too high of a value here could lead to your application not performing optimally because you are having the system wastefully wait for an unsuccessful attempt at establishing contact again and again which is why testing different values until arriving at one suitable for what your app requires is strongly suggested. </p>

Java 130

Java Network Servers Database 130

KeyCDN

NOVEMBER 4, 2021

The POP is strategially located within the country and lowers latency overall. KeyCDN is always on the lookout for ways to minimize latency and accelerate asset delivery worldwide. Hola Mexico! We've launched our new point of presence (POP) in Mexico City. In this case, the POP's identifier is mxmc.

Latency 111

Latency Tuning Cache Traffic 111

The Netflix TechBlog

AUGUST 30, 2021

By Benson Ma , Alok Ahuja Introduction At Netflix, hundreds of different device types, from streaming sticks to smart TVs, are tested every day through automation to ensure that new software releases continue to deliver the quality of the Netflix experience that our customers enjoy. In this blog post, we will focus on the latter feature set.

Latency 213

Latency Traffic Transportation Hardware 213

HammerDB

MAY 5, 2019

Fortunately the HammerDB TPC-C/OLTP workload intentionally has a great deal of contention between threads and is therefore ideal for testing spin-locks. ” and “as the PAUSE latency has been increased significantly, workloads that are sensitive to PAUSE latency will suffer some performance loss.” linux-glibc2.12-x86_64/data

Testing 48

Testing Tuning Latency Storage 48

The Netflix TechBlog

OCTOBER 19, 2020

If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. Our engineering teams tuned their services for performance after factoring in increased resource utilization due to tracing.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Dynatrace

APRIL 26, 2022

STM generates traffic that replicates the typical path or behavior of a user on a network to measure performance for example, response times, availability, packet loss, latency, jitter, and other variables). One use case for STM is to model the behavior of a customer in the form of a flow of transactions along the buyer’s journey.

Monitoring 203

Monitoring Social Media IoT Metrics 203

Dynatrace

DECEMBER 2, 2021

Stable, well-calibrated SLOs pave the way for teams to automate more processes and testing throughout the software delivery life cycle (SDLC). You can set SLOs based on individual indicators, such as batch throughput, request latency, and failures-per-second. Promote automation. SLO best practices.

Metrics 191

Metrics Best Practices DevOps Infrastructure 191

Smashing Magazine

NOVEMBER 8, 2021

As developers, we rightfully obsess about the customer experience, relentlessly working to squeeze every millisecond out of the critical rendering path, optimize input latency, and eliminate jank. Stay tuned for more in 2022! Ilya Grigorik. 2021-11-08T14:30:00+00:00. 2021-11-08T19:34:34+00:00. Large preview ).

Cache 136

Cache Best Practices Strategy Servers 136

KeyCDN

AUGUST 15, 2019

Web performance is a broad subject, and you’ll find no shortage of performance testing tips and tutorials all over the web. Before you begin tuning your website or application, you must first figure out which metrics matter most to your users and establish some achievable benchmarks. What is Performance Testing?

Testing Tools 55

Testing Tools Best Practices Performance Testing Testing 55

Abhishek Tiwari

JULY 12, 2018

Here are 8 fallacies of data pipeline The pipeline is reliable Topology is stateless Pipeline is infinitely scalable Processing latency is minimum Everything is observable There is no domino effect Pipeline is cost­-effective Data is homogeneous The pipeline is reliable The inconvenient truth is that pipeline is not reliable.

Latency 63

Latency Analytics Scalability Engineering 63

The Netflix TechBlog

JUNE 14, 2023

The three strategies we will discuss today are AB Testing , Replay Testing, and Sticky Canaries. To launch Phase 1 safely, we used AB Testing. To launch Phase 2 safely, we used Replay Testing and Sticky Canaries. We knew we could test the same query with the same inputs and consistently expect the same results.

Traffic 353

Traffic Latency Cache Metrics 353

KeyCDN

JANUARY 31, 2019

The next closest active POP location to Bucharest was Istanbul which was still almost 900km away; this distance adds up in terms of latency. Alternatively, If you would like to request a POP in a certain location or a new feature please let us know on our feature requests community page and stay tuned for more exciting announcements.

Internet 48

Internet Internet Ecommerce Speed 48

Brendan Gregg

OCTOBER 8, 2018

Screenshot: tracing read latency for PID 181: # bpftrace -e 'kprobe:vfs_read /pid == 30153/ { @start[tid] = nsecs; } kretprobe:vfs_read /@start[tid]/ { @ns = hist(nsecs - @start[tid]); delete(@start[tid]); }'. Back then I could already tell if disks were seeking by interpreting iostat(1) output: seeing high disk latency but small I/O.

C++ 110

C++ Virtualization Programming Latency 110

Dynatrace

JANUARY 14, 2022

Tuning thousands of parameters has become an impossible task to achieve via a manual and time-consuming approach. The following figure shows the high-level architecture where any load testing solution (e.g. SREcon21 – Automating Performance Tuning with Machine Learning. The Akamas approach. lower than 2%.).

Latency 204

Latency Tuning Efficiency AWS 204

KeyCDN

OCTOBER 24, 2018

Although both countries are relatively close to one another, they are separated by a distance of approximately 500km, which adds up in terms of latency. Before the implementation of our Helsinki POP, Finnish users were delivered content from our POP in Stockholm, Sweden. Happy content delivery!

Internet 48

Internet Internet Speed Network 48

The Netflix TechBlog

JUNE 13, 2023

Our previous blog post presented replay traffic testing — a crucial instrument in our toolkit that allows us to implement these transformations with precision and reliability. Compared to replay testing, canaries allow us to extend the validation scope beyond the service level.

Traffic 279

Traffic Metrics Systems Strategy 279

O'Reilly

JANUARY 16, 2024

These long solo drives were the perfect opportunity to test out ChatGPT’s voice feature because I couldn’t interact with the app using my hands for safety reasons. Most notably, ChatGPT is tuned to be overly-friendly and overly-agreeable (sounding like a customer service agent) so it will simply go along with whatever you assert.

Entertainment 73

Entertainment Internet Internet Design 73

Scalegrid

JUNE 4, 2020

Compare Latency. lower latency compared to DigitalOcean for PostgreSQL. PostgreSQL DigitalOcean Performance Test. Now, let’s take a look at the throughput and latency performance of our comparison. Next, we are going to test and compare the latency performance between ScaleGrid and DigitalOcean for PostgreSQL.

Database 230

Database Latency Benchmarking Performance 230

The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. We expanded pipeline support to serve our studio/content-development use cases, which had different latency and resiliency requirements as compared to the traditional streaming use case. This testing stage took about two weeks.

Processing 229

Processing Media Latency Innovation 229

The Netflix TechBlog

MARCH 1, 2021

It supports both high throughput services that consume hundreds of thousands of CPUs at a time, and latency-sensitive workloads where humans are waiting for the results of a computation. Local development tools including specialized test runners, code generators, and a command line interface. Productivity?—?Local Delivery?—?A

Serverless 228

Serverless Media Latency Social Media 228

Percona

APRIL 13, 2023

In PostgreSQL, replication lag can occur due to various reasons such as network latency, slow disk I/O, long-running transactions, etc. Replication lag can occur due to various reasons, such as: Network latency: Network latency is the delay caused by the time it takes for data to travel between the primary and standby databases.

Latency 80

Latency Tuning Open Source Network 80

Percona

APRIL 7, 2023

If we were to select the most important MySQL setting, if we were given a freshly installed MySQL or Percona Server for MySQL and could only tune a single MySQL variable, which one would it be? To be fair, that is also true with PostgreSQL; it hasn’t been tuned either, and it, too, can also perform much better.

Tuning 135

Tuning Cache Servers Benchmarking 135

Brendan Gregg

OCTOBER 7, 2018

Screenshot: tracing read latency for PID 181: # bpftrace -e 'kprobe:vfs_read /pid == 30153/ { @start[tid] = nsecs; } kretprobe:vfs_read /@start[tid]/ { @ns = hist(nsecs - @start[tid]); delete(@start[tid]); }'. Back then I could already tell if disks were seeking by interpreting iostat(1) output: seeing high disk latency but small I/O.

C++ 40

C++ Virtualization Programming Latency 40

Percona

MARCH 23, 2023

Nowadays, solid-state drives (SSDs) or non-volatile memory express (NVMe) drives are preferred over traditional hard disk drives (HDDs) for database servers due to their faster read and write speeds, lower latency, and improved reliability. The optimal value can be decided after testing multiple settings, starting from eight is a good choice.

Performance 124

Performance Hardware Tuning Storage 124

The Netflix TechBlog

AUGUST 22, 2019

Artisan Crafted Images In the Netflix full cycle DevOps culture the team responsible for building a service is also responsible for deploying, testing, infrastructure, and operation of that service. Now each change in the infrastructure is tested, canaried, and deployed like any other code change.

DevOps 207

DevOps AWS Tuning Infrastructure 207

The Netflix TechBlog

MAY 14, 2021

We are expected to process 1,000 watermarks for a single distribution in a minute, with non-linear latency growth as the number of watermarks increases. Even though Cosmos was developed for asynchronous media processing, we worked with them to expand to generic file processing and tune their workflow platform for our near real-time use case.

Traffic 160

Traffic Latency Java Google 160

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. There's also a test and println() in the loop to, hopefully, convince the compiler not to optimize-out an otherwise empty loop. This will slow this test a little.) Trying it out: centos$ time java TimeBench.

Speed 126

Speed Java AWS Virtualization 126

The Netflix TechBlog

AUGUST 29, 2023

Keeping feature parity between all of these implementations and ensuring that they all behave the same way is challenging: what we want is a single, well-tested implementation of all of this functionality, so we can make changes and fix bugs in one place. This is the first in a series of posts on our journey to service mesh, so stay tuned.

Traffic 220

Traffic Latency Cloud C++ 220

The Netflix TechBlog

NOVEMBER 2, 2020

High level playback architecture with priority throttling and chaos testing Building a request taxonomy We decided to focus on three dimensions in order to categorize request traffic: throughput, functionality, and criticality. Those two metrics are approximate indicators of failures and latency.

Traffic 252

Traffic Metrics Infrastructure Architecture 252

The Netflix TechBlog

NOVEMBER 2, 2021

Perceptual quality measurements are used to drive video encoding optimizations , perform video codec comparisons , carry out A/B testing and optimize streaming QoE decisions to mention a few. This enables us to use our scale to increase throughput and reduce latencies. Stay tuned for more details on these algorithmic innovations.

Media 171

Media Innovation Metrics Latency 171

Scalegrid

OCTOBER 17, 2019

While there is plenty of well-documented benefits to using a connection pooler, there are some arguments to be made against using one: Introducing a middleware in the communication inevitably introduces some latency. Our tests show that even a small number of clients can significantly benefit from using a connection pooler.

Architecture 200

Architecture Database Latency Servers 200

The Netflix TechBlog

NOVEMBER 20, 2023

Whether in analyzing A/B tests, optimizing studio production, training algorithms, investing in content acquisition, detecting security breaches, or optimizing payments, well structured and accurate data is foundational. Introduction Netflix relies on data to power its business in all phases.

Processing 210

Processing Big Data Efficiency Engineering 210

The Netflix TechBlog

JULY 21, 2022

Dealing with ambiguities and missing data : Sometimes the entries in BDP are contaminated with testing entries and NULL values, along with ambiguous values that have no meaning or just simply contradictory values due to unreal test environments. Restricting Testing and Analysis to one day and device at a time.

Big Data 179

Big Data Cache Engineering Data Engineering 179

Percona

APRIL 17, 2023

The main objective of this post is to share my experience over the past years tuning MongoDB and centralize the diverse sources that I crossed in this journey in a unique place. The CFQ works well for many general use cases but lacks latency guarantees. This parameter needs to be extensively tested!

Best Practices 52

Best Practices Design Tuning Database 52