The Netflix TechBlog

MARCH 1, 2021

It supports both high throughput services that consume hundreds of thousands of CPUs at a time, and latency-sensitive workloads where humans are waiting for the results of a computation. The subsystems all communicate with each other asynchronously via Timestone, a high-scale, low-latency priority queuing system. Warm capacity.

Serverless 228

Serverless Media Latency Social Media 228

Dynatrace

APRIL 25, 2023

Observability is essential to ensure the reliability, security and quality of any software system. Higher latency and cold start issues due to the initialization time of the functions. Understanding cold-start behavior is essential to tune your cloud applications cost or performance to meet your operational needs.

Serverless 213

Serverless Lambda Azure AWS 213

Dynatrace

MAY 5, 2021

As more organizations respond to the pressure to release better software faster, there is an increasing need to build quality gates into every stage of BizDevOps processes , from early development to deployment. For example, improving latency by as little as 0.1 For example, improving latency by as little as 0.1 Dynatrace news.

Benchmarking 188

Benchmarking Latency Speed Software 188

The Morning Paper

JULY 7, 2019

Software engineering for machine learning: a case study Amershi et al., More specifically, we’ll be looking at the results of an internal study with over 500 participants designed to figure out how product development and software engineering is changing at Microsoft with the rise of AI and ML. ICSE’19. A general process.

Software Engineering 92

Software Engineering Engineering Software Software 92

Scalegrid

DECEMBER 14, 2023

Faster detection of a bad connection is accomplished when there’s a lower value, while latency or intermittent network issues may be better served by increasing it slightly – although overly long timeouts can result in application hang-ups with little chance for an established link. Careful consideration must be given before making changes.

Java 130

Java Network Servers Database 130

Dynatrace

JULY 24, 2023

DevOps automation example #3: Progressive delivery In software development and delivery, if an organization uses feature flags to control feature releases, the marriage of observability data and answer-driven automation becomes a formidable force. But it doesn’t stop there. They intend to roll it out to the production environment.

DevOps 225

DevOps Traffic Efficiency Servers 225

The Netflix TechBlog

AUGUST 22, 2019

Once the instance was available, the engineer would use a remote administration tool like RDP to login to the instance to install software and customize settings. We now have the software and instance configuration as code. The last piece of the puzzle was finding a way to package our software for installation on Windows.

DevOps 207

DevOps AWS Tuning Infrastructure 207

Percona

NOVEMBER 2, 2023

Improved performance : MongoDB continually fine-tunes its database engine, resulting in faster query execution and reduced latency. Protection against data loss : Older MongoDB versions pose a risk of data loss due to unsupported software.

Best Practices 84

Best Practices Hardware Tuning Scalability 84

The Morning Paper

MAY 21, 2019

Software-defined far memory in warehouse-scale computers Lagar-Cavilla et al., This boils down to a single digit µs latency toleration in the tail for far memory, and in addition to security and privacy concerns, rules out remote memory solutions. ML-based auto-tuning. ASPLOS’19. Evaluation.

Software 78

Software Software Google Hardware 78

The Netflix TechBlog

AUGUST 29, 2023

Second, we’ve moved from a Java-only environment to a Polyglot one: we now also support node.js , Python , and a variety of OSS and off the shelf software. There is a downside to fetching this data on-demand: this adds latency to the first request to a cluster. Our internal IPC traffic is now a mix of plain REST, GraphQL , and gRPC.

Traffic 220

Traffic Latency Cloud C++ 220

Dynatrace

JANUARY 26, 2021

When an application is triggered, it can cause latency as the application starts. Performing updates, installing software, and resolving hardware issues requires up to 17 hours of developer time every week. This creates latency when they need to restart. How does serverless computing tackle inefficiencies?

Serverless 245

Serverless Efficiency Lambda Azure 245

The Netflix TechBlog

JUNE 13, 2023

By collecting and analyzing key performance metrics of the service over time, we can assess the impact of the new changes and determine if they meet the availability, latency, and performance requirements. They enable us to further fine-tune and configure the system, ensuring the new changes are integrated smoothly and seamlessly.

Traffic 279

Traffic Metrics Systems Strategy 279

Dynatrace

SEPTEMBER 23, 2020

This is why the Dynatrace Software Intelligence Platform is recognized as a market leader not only for monitoring coverage, but also, very importantly, for providing the shortest time-to-value. Storage mount points in a system might be larger or smaller, local or remote, with high or low latency, and various speeds.

Storage 150

Storage Latency Operating System Network 150

Dynatrace

APRIL 26, 2022

STM generates traffic that replicates the typical path or behavior of a user on a network to measure performance for example, response times, availability, packet loss, latency, jitter, and other variables). One use case for STM is to model the behavior of a customer in the form of a flow of transactions along the buyer’s journey.

Monitoring 209

Monitoring Social Media IoT Metrics 209

Dynatrace

DECEMBER 2, 2021

Error budgets allow development teams to make informed decisions between new development vs operations and polishing existing software. Generally, SLOs are important because they: Improve software quality. It is inevitable that software can break. For example, if your SLO guarantees 99.5% What are SLIs? Watch webinar now!

Metrics 196

Metrics Best Practices DevOps Infrastructure 196

The Netflix TechBlog

AUGUST 30, 2021

By Benson Ma , Alok Ahuja Introduction At Netflix, hundreds of different device types, from streaming sticks to smart TVs, are tested every day through automation to ensure that new software releases continue to deliver the quality of the Netflix experience that our customers enjoy. In this blog post, we will focus on the latter feature set.

Latency 213

Latency Traffic Transportation Hardware 213

The Netflix TechBlog

MAY 14, 2021

We are expected to process 1,000 watermarks for a single distribution in a minute, with non-linear latency growth as the number of watermarks increases. Even though Cosmos was developed for asynchronous media processing, we worked with them to expand to generic file processing and tune their workflow platform for our near real-time use case.

Traffic 160

Traffic Latency Java Google 160

The Morning Paper

NOVEMBER 10, 2019

It’s been clear for a while that software designed explicitly for the data center environment will increasingly want/need to make different design trade-offs to e.g. general-purpose systems software that you might install on your own machines. The desire for CPU efficiency and lower latencies is easy to understand. Enter Google!

Network 92

Network Transportation Latency Entertainment 92

Dynatrace

MAY 17, 2023

Making applications observable—relying on metrics, logs, and traces to understand what software is doing and how it’s performing—has become increasingly important as workloads are shifting to multicloud environments. This allows us to quickly tell whether the network link may be saturated or the processor is running at its limit.

Metrics 175

Metrics Monitoring Database Network 175

The Netflix TechBlog

OCTOBER 19, 2020

If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. Our engineering teams tuned their services for performance after factoring in increased resource utilization due to tracing.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

The Netflix TechBlog

NOVEMBER 20, 2023

As our business scales globally, the demand for data is growing and the needs for scalable low latency incremental processing begin to emerge. It serves thousands of users, including data scientists, data engineers, machine learning engineers, software engineers, content producers, and business analysts, in various use cases.

Processing 210

Processing Big Data Efficiency Engineering 210

Scalegrid

MARCH 22, 2024

They can also bolster uptime and limit latency issues or potential downtimes. It’s important to ensure the bells and whistles of any software-as-a-service (SaaS) they offer can support where you aim to take your business, keeping your strategy tight and on track.

Strategy 130

Strategy Cloud Government Innovation 130

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. As a Xen guest, this profile was gathered using perf(1) and the kernel's software cpu-clock soft interrupts, not the hardware NMI. Software-based clocksources could fix those issues and provide accurate monotonically-increasing time.

Speed 126

Speed Java AWS Virtualization 126

The Netflix TechBlog

JULY 21, 2022

for us at Netflix, this is a combination of the device type, app session ID and software development kit version (SDK version). Stay tuned for further posts on memory management and the use of ML modeling to deal with systemic and low latency data collected at the device level.

Big Data 179

Big Data Cache Engineering Data Engineering 179

Percona

APRIL 17, 2023

The main objective of this post is to share my experience over the past years tuning MongoDB and centralize the diverse sources that I crossed in this journey in a unique place. The CFQ works well for many general use cases but lacks latency guarantees. Spoiler alert: This post focuses on MongoDB 3.6.X tcp_fin_timeout = 30 net.ipv4.tcp_keepalive_intvl

Best Practices 52

Best Practices Design Tuning Database 52

Brendan Gregg

JUNE 30, 2018

A co-worker introduced me to Craig Hanson and Pat Crain's performance mantras, which neatly summarize much of what we do in performance analysis and tuning. But I see this all the time in tech: benchmarks that spin some software component at unrealistic rates, in order to claim a "win" over a competitor. Don't do it. Do it less.

Benchmarking 54

Benchmarking Latency Cache Network 54

HammerDB

OCTOBER 18, 2019

cpupower frequency-info analyzing CPU 0: driver: intel_pstate CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: Cannot determine or is not supported. hardware limits: 1000 MHz - 4.00 bin/pgbench -c 1 -S -T 60 pgbench starting vacuum.end.

Benchmarking 63

Benchmarking Performance Hardware C++ 63

Brendan Gregg

JUNE 29, 2018

A co-worker introduced me to Craig Hanson and Pat Crain's performance mantras, which neatly summarize much of what we do in performance analysis and tuning. But I see this all the time in tech: benchmarks that spin some software component at unrealistic rates, in order to claim a "win" over a competitor. Don't do it. Do it less.

Benchmarking 40

Benchmarking Latency Cache Network 40

The Netflix TechBlog

FEBRUARY 9, 2021

Stay tuned for more details on this, as well as more details on the internals of the new SKU Platform in one of our upcoming blog posts. Lower latency as a result of fewer service calls, which means fewer errors for our visitors. The next step is to invest further into self-service and support rule changes via a SKU UI.

Engineering 142

Engineering Scalability Architecture Innovation 142

Abhishek Tiwari

JULY 12, 2018

Data Pipeline A data pipeline is a software that ingests data from multiple sources, transforms it and finally makes it available to internal or external products. A data pipeline is a software which runs on hardware. The software is error-prone and hardware failures are inevitable.

Latency 63

Latency Analytics Scalability Engineering 63

IO River

NOVEMBER 15, 2023

Two of them are particularly gnarly: fine-tuning rules to perfection and managing a WAF over a multi-CDN architecture. Attackers discover a previously unknown vulnerability in a software or system. Benchmarks should be established to measure the latency introduced by the WAF, ensuring it stays within acceptable limits.‍Conclusion‍In

Traffic 52

Traffic Network Logistics Architecture 52

O'Reilly

JANUARY 16, 2024

For years I had an aspirational goal of unifying all of my backups into one central directory tree, akin to the concept of a monorepo in software development. software” rather than “hardware” in our brains). So I figured that ChatGPT could help me brainstorm during one of my long drives.

Entertainment 73

Entertainment Internet Internet Design 73

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. As a Xen guest, this profile was gathered using perf(1) and the kernel's software cpu-clock soft interrupts, not the hardware NMI. Software-based clocksources could fix those issues and provide accurate monotonically-increasing time.

Speed 40

Speed Java AWS Virtualization 40

O'Reilly Software

APRIL 4, 2018

Here are five considerations every software architect and developer needs to take into account when setting the architectural foundations for a fast data platform. The data shape will dictate capacity planning, tuning of the backbone, and scalability analysis for individual components. Determine requirements first. At least once?

Architecture 80

Architecture Scalability Google Operating System 80

Brendan Gregg

OCTOBER 8, 2018

eBPF was created by Alexei Starovoitov while at PLUMgrid (he's now at Facebook) as a generic in-kernel virtual machine, with software defined networks as the primary use case. Back then I could already tell if disks were seeking by interpreting iostat(1) output: seeing high disk latency but small I/O. eBPF does more.

C++ 110

C++ Virtualization Programming Latency 110

IO River

NOVEMBER 2, 2023

Two of them are particularly gnarly: fine-tuning rules to perfection and managing a WAF over a multi-CDN architecture. They exploit unknown vulnerabilities, giving developers zero days to fix the issue—hence the name.‍Attackers discover a previously unknown vulnerability in a software or system. Let's dive deep into these challenges.‍1.

Traffic 40

Traffic Network Logistics Architecture 40

John McCalpin

JANUARY 22, 2018

Each of the two vector units can issue one FMA instruction per cycle, assuming that there are enough independent accumulators to tolerate the 6-cycle dependent-operation latency. This is an uninspiring fraction of peak performance that would normally suggest significant inefficiencies in either the hardware or software.

Latency 40

Latency Hardware Code Testing 40