Dynatrace

DECEMBER 2, 2021

Cloud-native software and its supporting tools and infrastructure generate a diversity of metrics and data points every second that indicate a system’s state and performance. For example, an objective for system availability can be: 90% – one 9. How SLOs work. 99% – two 9s. 99.9% – three 9s. SLO best practices.

Metrics 193

Metrics Best Practices DevOps Infrastructure 193

The Netflix TechBlog

FEBRUARY 18, 2022

To this end, we developed a Rapid Event Notification System (RENO) to support use cases that require server initiated communication with devices in a scalable and extensible manner. In this blog post, we will give an overview of the Rapid Event Notification System at Netflix and share some of the learnings we gained along the way.

Systems 334

Systems Traffic Architecture Mobile 334

The Netflix TechBlog

JUNE 13, 2023

This is where large-scale system migrations come into play. By collecting and analyzing key performance metrics of the service over time, we can assess the impact of the new changes and determine if they meet the availability, latency, and performance requirements. But what happens when this machinery needs a transformation?

Traffic 279

Traffic Metrics Systems Strategy 279

Brendan Gregg

OCTOBER 8, 2018

Created by Alastair Robertson , bpftrace is an open source high-level tracing front-end that lets you analyze systems in custom ways. eBPF (extended Berkeley Packet Filter) is in the Linux kernel and is the new hotness in systems engineering. It makes off-CPU analysis effective, which is the counterpart to CPU analysis.

C++ 110

C++ Virtualization Programming Latency 110

Dynatrace

MAY 17, 2023

Think about items such as general system metrics (for example, CPU utilization, free memory, number of services), the connectivity status, details of our web server, or even more granular in-application tasks like database queries. Let’s take a look at what kind of additional telemetry data we will have at our fingertips with OneAgent.

Metrics 173

Metrics Monitoring Database Network 173

The Morning Paper

MARCH 10, 2020

So if you want to design a system that will offer good real-world performance, it’s really useful to have benchmarks that accurately represent real-world workloads. This benchmark can synthetically generate more precise key-value queries that represent the reads and writes of key-value stores to the underlying storage system.

Benchmarking 59

Benchmarking Storage Cache Open Source 59

The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Processing 229

Processing Media Latency Innovation 229

Dynatrace

JULY 24, 2023

It enables them to adapt to user feedback swiftly, fine-tune feature releases, and deliver exceptional user experiences, all while maintaining control and minimizing disruption. Consider an event-driven automation system designed for incident management. But it doesn’t stop there. All these actions aim to avert future incidents.

DevOps 222

DevOps Traffic Efficiency Servers 222

Brendan Gregg

OCTOBER 7, 2018

Created by Alastair Robertson , bpftrace is an open source high-level tracing front-end that lets you analyze systems in custom ways. eBPF (extended Berkeley Packet Filter) is in the Linux kernel and is the new hotness in systems engineering. It makes off-CPU analysis effective, which is the counterpart to CPU analysis.

C++ 40

C++ Virtualization Programming Latency 40

The Netflix TechBlog

JULY 21, 2022

With large data, comes the opportunity to leverage the data for predictive and classification based analysis. We also highlight steps and guidelines for exploratory analysis and prediction to understand Out of Memory kills on a sample set of devices. It also prevents the system running out of memory during execution of the query.

Big Data 179

Big Data Cache Engineering Data Engineering 179

The Netflix TechBlog

SEPTEMBER 24, 2021

Lastly, the packager kicks in, adding a system layer to the asset, making it ready to be consumed by the clients. Uploading and downloading data always come with a penalty, namely latency. There are existing distributed file systems for the cloud as well as off-the-shelf FUSE modules for S3.

Cloud 237

Cloud Media Storage Cache 237

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. CLI tools The Cassandra systems were EC2 virtual machine (Xen) instances. This was a weird problem to think about: Time itself had now become a resource and target of performance analysis. Try changing the kernel clocksource.

Speed 126

Speed Java AWS Virtualization 126

The Netflix TechBlog

AUGUST 13, 2020

You’re half awake and wondering, “Is there really a problem or is this just an alert that needs tuning? Our streaming teams need a monitoring system that enables them to quickly diagnose and remediate problems; seconds count! Our Node team needs a system that empowers a small group to operate a large fleet. By Andrei U.,

Monitoring 210

Monitoring Tuning Traffic Metrics 210

The Netflix TechBlog

NOVEMBER 2, 2021

The coupling problem Until recently, video quality measurements were generated as part of our Reloaded production system. This system is responsible for processing incoming media files, such as video, audio and subtitles, and making them playable on the streaming service. We call this system Cosmos. The workflow is initiated.

Media 171

Media Innovation Metrics Latency 171

The Netflix TechBlog

JULY 26, 2021

This happens at an unprecedented scale and introduces many interesting challenges; one of the challenges is how to provide visibility of Studio data across multiple phases and systems to facilitate operational excellence and empower decision making. Please stay tuned! We will have follow up blog posts on these topics in future.

Big Data 253

Big Data Government Analytics Processing 253

IO River

NOVEMBER 15, 2023

This involves:Threat Intelligence: Keeping an eye on the latest types of attacks, whether it's a novel form of SQL injection or a new phishing tactic.Internal Audits: Regularly scanning your own system for vulnerabilities. Two of them are particularly gnarly: fine-tuning rules to perfection and managing a WAF over a multi-CDN architecture.

Traffic 52

Traffic Network Logistics Architecture 52

The Morning Paper

JULY 7, 2019

In addition to availability, our respondents focus most heavily on supporting the following data attributes: “accessibility, accuracy, authoritativeness, freshness, latency, structuredness, ontological typing, connectedness, and semantic joinability.” Learnings and emerging best practices.

Software Engineering 92

Software Engineering Engineering Software Software 92

Brendan Gregg

JUNE 30, 2018

A co-worker introduced me to Craig Hanson and Pat Crain's performance mantras, which neatly summarize much of what we do in performance analysis and tuning. Answering this usually requires analysis using other observability tools while the benchmark is running (what I call "[active benchmarking]"). ### 2. Don't do it.

Benchmarking 54

Benchmarking Latency Cache Network 54

The Morning Paper

MAY 21, 2019

” This paper describes a “far memory” system that has been in production deployment at Google since 2016. This boils down to a single digit µs latency toleration in the tail for far memory, and in addition to security and privacy concerns, rules out remote memory solutions.

Software 78

Software Software Google Hardware 78

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. CLI tools The Cassandra systems were EC2 virtual machine (Xen) instances. This was a weird problem to think about: Time itself had now become a resource and target of performance analysis. Try changing the kernel clocksource.

Speed 40

Speed Java AWS Virtualization 40

IO River

NOVEMBER 2, 2023

This involves:Threat Intelligence: Keeping an eye on the latest types of attacks, whether it's a novel form of SQL injection or a new phishing tactic.Internal Audits: Regularly scanning your own system for vulnerabilities. Two of them are particularly gnarly: fine-tuning rules to perfection and managing a WAF over a multi-CDN architecture.

Traffic 40

Traffic Network Logistics Architecture 40

Brendan Gregg

JUNE 29, 2018

A co-worker introduced me to Craig Hanson and Pat Crain's performance mantras, which neatly summarize much of what we do in performance analysis and tuning. Answering this usually requires analysis using other observability tools while the benchmark is running (what I call "[active benchmarking]"). ### 2. Don't do it.

Benchmarking 40

Benchmarking Latency Cache Network 40

All Things Distributed

APRIL 9, 2015

The Amazon ML console and API provide data and model visualization tools, as well as wizards to guide you through the process of creating machine learning models, measuring their quality and fine-tuning the predictions to match your application requirements. The Amazon Elastic File System. Details on the AWS Blog. for a while already.

Lambda 123

Lambda Cloud IoT AWS 123

Brendan Gregg

SEPTEMBER 5, 2017

What follows are topics that may be of interest to anyone looking to migrate their systems and skillset: scan these to find topics that interest you. ## ZFS ZFS is available for Linux via the [zfsonlinux] and [OpenZFS] projects, and more recently was included in Canonical's Ubuntu Linux distribution: Ubuntu Xenial 16.04 LTS (April 2016).

Virtualization 76

Virtualization AWS Engineering Hardware 76

O'Reilly Software

APRIL 4, 2018

Those were the days when “connected” meant having some web presence, “interactions” were measured in number of clicks, and the analysis happened in batch overnight processes. Are there inherent time relationships in the messages that need to be preserved as they travel across the system? At least once? At most once? Exactly once?

Architecture 80

Architecture Scalability Google Operating System 80

John McCalpin

JANUARY 22, 2018

Each of the two vector units can issue one FMA instruction per cycle, assuming that there are enough independent accumulators to tolerate the 6-cycle dependent-operation latency. of the “adjusted peak performance”, there is no longer a significant upside to performance tuning. not FMAs). cmpl $1000000000, %eax.

Latency 40

Latency Hardware Code Testing 40