The Netflix TechBlog

AUGUST 29, 2023

A brief history of IPC at Netflix Netflix was early to the cloud, particularly for large-scale companies: we began the migration in 2008, and by 2010, Netflix streaming was fully run on AWS. Today we have a wealth of tools, both OSS and commercial, all designed for cloud-native environments.

Traffic 220

Traffic Latency Cloud C++ 220

The Netflix TechBlog

SEPTEMBER 8, 2020

How we migrated our Android endpoints out of a monolith into a new microservice by Rohan Dhruva , Ed Ballot As Android developers, we usually have the luxury of treating our backends as magic boxes running in the cloud, faithfully returning us JSON. Java…Script? It was a Node.js

Latency 233

Latency Cache Java Traffic 233

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. There's no Java stack—there should be a tower of green Java methods—instead there's only a single green frame or two. This is how Java flame graphs looked at the time. 30.14% in the middle of the flame graph.

Speed 126

Speed Java AWS Virtualization 126

Dynatrace

JULY 10, 2019

OneAgent & cloud metrics. Virtualization can be a key player in your process’ performance, and Dynatrace has built-in integrations to bring metrics about the Cloud Infrastructure into your Dynatrace environment. Dynatrace provides out-of-the-box support for VMware, AWS, Azure, Pivotal Cloud Foundry, and Kubernetes.

Java 140

Java Best Practices Metrics Azure 140

The Netflix TechBlog

OCTOBER 19, 2020

If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. We chose Open-Zipkin because it had better integrations with our Spring Boot based Java runtime environment.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Dynatrace

MAY 17, 2023

It is available for the major OS and cloud platforms (for example, Windows, Linux, Solaris, AWS, Azure, and more) and only requires the deployment of a single service to monitor its environment. Garbage collection count Garbage collection is JVM related and indicates how often the Java GC ran.

Metrics 176

Metrics Monitoring Database Network 176

High Scalability

JULY 20, 2018

And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. crabbone : This is the prism through which Java programmers view the world. It would mean a great deal to me. They'll love you even more.

Internet 121

Internet Internet Scalability Automotive 121

The Morning Paper

MAY 12, 2019

An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems Gan et al., Microservices fundamentally change a lot of assumptions current cloud systems are designed with, and present both opportunities and challenges when optimizing for quality of service (QoS) and utilization.

Open Source 61

Open Source Hardware Benchmarking Systems 61

All Things Distributed

FEBRUARY 22, 2012

Expanding the Cloud â?? Amazon SWF makes it very easy for developers to architect and implement these tasks, run them in the cloud or on premise and coordinate their flow. They must deal with the increased latency and unreliability inherent in remote communication. All Things Distributed. The Amazon Simple Workflow Service.

Cloud 130

Cloud AWS Java Scalability 130

Dynatrace

JANUARY 18, 2023

This methodology aims to improve software system reliability using several key categories such as availability, performance, latency, efficiency, capacity, and incident response. These capabilities prevent malicious actors from executing commands on certain Java processes that are accessible to the outside world.

DevOps 203

DevOps Best Practices Innovation Strategy 203

The Netflix TechBlog

FEBRUARY 9, 2021

As we migrated to EdgePaaS, front-end services were moved from the Java-based API to a BFF (backend for frontend), aka NodeQuark, as shown: This model enables front-end engineers to own and operate their services outside of the core API framework. The following examples of these gains are from the primary API service.

Architecture 221

Architecture Latency Servers Website 221

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. There's no Java stack—there should be a tower of green Java methods—instead there's only a single green frame or two. This is how Java flame graphs looked at the time. This will slow this test a little.)

Speed 52

Speed Java AWS Virtualization 52

The Netflix TechBlog

AUGUST 22, 2019

Though the majority of our services run on Linux Amazon Machine Images (AMIs), there are still many services critical to the Netflix Playback Experience running on Windows Elastic Compute Cloud (EC2) instances at scale. The canary stage will determine a score based on metrics such as CPU, threads, latency, and GC pauses.

DevOps 207

DevOps AWS Tuning Infrastructure 207

All Things Distributed

APRIL 9, 2015

Amazon EFS is a fully-managed service that makes it easy to set up and scale shared file storage in the AWS Cloud. Synchronous requests allow mobile and IoT apps to move data transformations and analysis to the cloud and make it easy for any application or web service to use Lambda to create back-end functionality.

Lambda 122

Lambda Cloud IoT AWS 122

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. There's no Java stack—there should be a tower of green Java methods—instead there's only a single green frame or two. This is how Java flame graphs looked at the time. 30.14% in the middle of the flame graph.

Speed 40

Speed Java AWS Virtualization 40

All Things Distributed

NOVEMBER 8, 2012

Improving the Cloud - More Efficient Queuing with SQS. For example, AWS customers use SQS for asynchronous communication pipelines, buffer queues for databases, asynchronous work queues, and moving latency out of highly responsive requests paths. All Things Distributed. By Werner Vogels on 08 November 2012 03:00 PM. Comments ().

Efficiency 104

Efficiency Cloud Games Scalability 104

ScaleOut Software

JULY 14, 2021

The Microsoft Azure IoT ecosystem offers a rich set of capabilities for processing IoT telemetry, from its arrival in the cloud through its storage in databases and data lakes. Acting as a switchboard for incoming and outgoing messages, Azure IoT Hub forms the core of these capabilities. Azure digital twins serve a different purpose.

IoT 52

IoT Azure Analytics Serverless 52

All Things Distributed

JUNE 21, 2017

As we said at the time, DynamoDB was a result of 15 years of learning in the area of large scale non-relational databases and cloud services. These high-throughput, low-latency requirements need caching, not as a consideration, but as a best practice. DynamoDB was the first service at AWS to use SSD storage. " Canon INC.

Speed 121

Speed Cache Latency AWS 121

The Morning Paper

MARCH 3, 2020

This work is latency critical, because volume IO is blocked until it is complete. Larger cells have better tolerance of tail latency (e.g. Few studies consider a factor that is especially important to cloud customers: large-scale correlated failures which affect many cloud resources at the same time. Before and After.

Database 106

Database AWS Network Design 106

Brendan Gregg

AUGUST 29, 2021

A cloud-wide monitoring tool, Atlas, showed a high rate of paging for the larger file uploads: The blue is pageins (page ins). biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. The problem was that large files, such as 100 Gbytes, seemed to take forever to upload. Tracing block device I/O. . ## 4.

Cache 105

Cache C++ AWS Latency 105

All Things Distributed

JULY 14, 2015

No matter which mechanism you choose to use, we make the stream data available to you instantly (latency in milliseconds) and how fast you want to apply the changes is up to you. Their cloud architecture has two main components: a point-of-sales system and a merchandising system. DynamoDB Cross-region Replication.

Database 167

Database Lambda AWS IoT 167

Brendan Gregg

MAY 16, 2017

We're on the EC2 cloud, which has great scalability, and our own cloud architecture of microservices is also designed for scalability. Getting frame pointer support in Java was another project I did a while ago. Java core dump analysis for a crashing JVM. - A latency outlier issue that happened every 15 minutes.

Java 75

Java Entertainment Engineering Scalability 75

The Netflix TechBlog

DECEMBER 17, 2019

One which: interleaves log with dump events so that both can make progress allows to trigger dumps at any time does not use table locks uses standardized database features DBLog Framework DBLog is a Java-based framework, able to capture changes in real-time and to take dumps. Proceedings of the Third ACM Symposium on Cloud Computing.

Database 197

Database Traffic Transportation Open Source 197

The Netflix TechBlog

DECEMBER 17, 2019

One which: interleaves log with dump events so that both can make progress allows to trigger dumps at any time does not use table locks uses commonly available database features DBLog Framework DBLog is a Java-based framework, able to capture changes in real-time and to take dumps. Proceedings of the Third ACM Symposium on Cloud Computing.

Database 212

Database Traffic Transportation Open Source 212

Brendan Gregg

AUGUST 29, 2021

A cloud-wide monitoring tool, Atlas, showed a high rate of paging for the larger file uploads: The blue is pageins (page ins). biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. The problem was that large files, such as 100 Gbytes, seemed to take forever to upload. Hit Ctrl-C to end. ^C

Cache 40

Cache C++ AWS Systems 40

O'Reilly Software

MARCH 7, 2018

Created in 2007, and described as "a versatile open source integration framework based on known enterprise integration patterns," it is a very popular Java library for system integration, offering implementations of most (if not all) of the standard enterprise integration patterns (EIP). AWS, Kafka, Google Cloud, Spring, ElasticSearch).

Transportation 72

Transportation Java Programming Architecture 72

The Morning Paper

SEPTEMBER 12, 2019

Though some of this discrepancy was due to the fact that we implemented our ideas on top of a research prototype, high-latency Java/Hadoop system, reducing that gap is an attractive target for future work. VLDB’19. The RDBMS was slower than TensorFlow for GPU-based implementations of neural networks, however.

Network 44

Network Database Programming Hardware 44