The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203

Storage Latency Efficiency Data Engineering 203

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

Percona

DECEMBER 11, 2023

A quick configuration change may do the trick in improving the performance of your AWS RDS for MySQL instance. A Dedicated Log Volume (DLV) is a specialized storage volume designed to house database transaction logs separately from the volume containing the database tables. Who can benefit from DLV? and later v13 versions, 14.7

AWS 99

AWS Benchmarking Performance Traffic 99

Dotcom-Montior

OCTOBER 6, 2021

Site reliability engineers are focused with creating scalable and reliable software systems, so this also includes ensuring that development work is efficient and reliable, so when the finished product is ready for production, there are no surprises. that are required to keep the software deployments live are running efficiently.

Engineering 67

Engineering DevOps Monitoring Google 67

Adrian Cockcroft

APRIL 18, 2018

This is true for CPU, network, memory and storage, and also for bare metal, virtual machines, containers, processes, functions and threads. For efficiency it is common to co-locate a monitoring system with the infrastructure, in the same datacenter or cloud region, but that sets up common dependencies that could cause both to fail together.

Monitoring 40

Monitoring Systems Virtualization Metrics 40

Scalegrid

MARCH 14, 2024

In practice, a hybrid cloud operates by melding resources and services from multiple computing environments, which necessitates effective coordination, orchestration, and integration to work efficiently. Tailoring resource allocation efficiently ensures faster application performance in alignment with organizational demands.

Strategy 130

Strategy Cloud Artificial Intelligence Infrastructure 130

The Netflix TechBlog

SEPTEMBER 24, 2021

Figure 1: A Simplified Video Processing Pipeline With this architecture, chunk encoding is very efficient and processed in distributed cloud computing instances. From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step.

Cloud 237

Cloud Media Storage Cache 237

Percona

JULY 1, 2023

Now that Database-as-a-service (DBaaS) is in high demand, there are multiple questions regarding AWS services that cannot always be answered easily: When should I use Aurora and when should I use RDS MySQL ? Amazon Aurora is a proprietary, cloud-native, fully managed relational database service developed by Amazon Web Services (AWS).

AWS 52

AWS Database Serverless Storage 52

Scalegrid

JANUARY 12, 2024

This article analyzes cloud workloads, delving into their forms, functions, and how they influence the cost and efficiency of your cloud infrastructure. Storage is a critical aspect to consider when working with cloud workloads. This opens up possibilities not only difficult but almost impossible to attain conventionally!

Cloud 130

Cloud Virtualization Storage Efficiency 130

The Netflix TechBlog

OCTOBER 27, 2020

The processed data is typically stored as data warehouse tables in AWS S3. The data warehouse is not designed to serve point requests from microservices with low latency. Therefore, we must efficiently move data from the data warehouse to a global, low-latency and highly-reliable key-value store.

Latency 243

Latency Storage Big Data Tuning 243

Dynatrace

OCTOBER 4, 2022

While data lakes and data warehousing architectures are commonly used modes for storing and analyzing data, a data lakehouse is an efficient third way to store and analyze data that unifies the two architectures while preserving the benefits of both. Unlike data warehouses, however, data is not transformed before landing in storage.

Artificial Intelligence 205

Artificial Intelligence Analytics Storage Government 205

All Things Distributed

DECEMBER 12, 2018

In April 2017, Amazon Web Services announced that it would launch a new AWS infrastructure region Region in Sweden. Today, I'm happy to announce that the AWS Europe (Stockholm) Region, our 20th Region globally, is now generally available for use by customers. Public sector.

AWS 124

AWS Cloud Games Serverless 124

High Scalability

SEPTEMBER 8, 2018

Anna is not only incredibly fast, it’s incredibly efficient and elastic too: an autoscaling, multi-tier, selectively-replicating cloud service. No, I don’t think that is because AWS is earning a 355x margin on DynamoDB! Each storage server collects statistics about the requests it serves, the data it stores, etc.

Storage 199

Storage Performance AWS Efficiency 199

The Netflix TechBlog

APRIL 26, 2022

The first version of our logger library optimized for storage by deduplicating facts and optimized for network i/o using different compression methods for each fact. Since we were optimizing at the logging level for storage and performance, we had less data and metadata to play with to optimize the query performance.

Storage 187

Storage Design Scalability Latency 187

The Morning Paper

FEBRUARY 6, 2020

Given the simplicity and economic appeal of FaaS, it is interesting to explore designs that preserve the autoscaling and operational benefits of current offerings, while adding performant, cost-efficient and consistent shared state and communication. Updates should be allowed at any function invocation site.

Lambda 98

Lambda Serverless Cache Latency 98

All Things Distributed

MARCH 11, 2016

The epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Given that AWS is a pioneer in building and operating these services world-wide, these lessons have been of crucial importance to our business. AWS helps its customers do this too.

AWS 144

AWS Hardware Retail Virtualization 144

All Things Distributed

JUNE 21, 2018

In fact, this is been proven by our customers as Amazon Aurora remains the fastest growing service in AWS history. Use cases such as gaming, ad tech, and IoT lend themselves particularly well to the key-value data model where the access patterns require low-latency Gets/Puts for known key values. The opposite is true.

Database 167

Database AWS Games Latency 167

All Things Distributed

AUGUST 20, 2015

Today, we are releasing a plugin that allows customers to use the Titan graph engine with Amazon DynamoDB as the backend storage layer. It opens up the possibility to enjoy the value that graph databases bring to relationship-centric use cases, without worrying about managing the underlying storage. The importance of relationships.

Database 134

Database Logistics Availability Social Media 134

All Things Distributed

JANUARY 18, 2012

Amazon DynamoDB offers low, predictable latencies at any scale. Amazon DynamoDB stores data on Solid State Drives (SSDs) and replicates it synchronously across multiple AWS Availability Zones in an AWS Region to provide built-in high availability and data durability. s read latency, particularly as dataset sizes grow.

Scalability 167

Scalability Database Ecommerce Latency 167

All Things Distributed

OCTOBER 12, 2016

Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis , a fully managed in-memory data store that operates at sub-millisecond latency. ElastiCache for Redis Multi-AZ capability is built to handle any failover case for Redis Cluster with robustness and efficiency.

Scalability 130

Scalability Analytics Cache AWS 130

All Things Distributed

NOVEMBER 15, 2010

For example, the most fundamental abstraction trade-off has always been latency versus throughput. Modern CPUs strongly favor lower latency of operations with clock cycles in the nanoseconds and we have built general purpose software architectures that can exploit these low latencies very well.Â

AWS 136

AWS Latency Programming Architecture 136

The Morning Paper

JUNE 23, 2019

Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. The network latency of fetching data over the network, even considering fast data center networks. Who knew! ;).

Cache 79

Cache Latency Google Lambda 79

Adrian Cockcroft

APRIL 5, 2023

This proposal seeks to define a standard for real-time carbon and energy data as time-series data that would be accessed alongside and synchronized with the existing throughput, utilization and latency metrics that are provided for the components and applications in computing environments.

Energy 52

Energy Metrics Cloud Operating System 52

Percona

APRIL 17, 2023

The CFQ works well for many general use cases but lacks latency guarantees. The deadline excels at latency-sensitive use cases ( like databases ), and noop is closer to no schedule at all. By default, most Linux installs use the CFQ ( Completely-Fair Queue ) scheduler. Two other schedulers are deadline and noop.

Best Practices 52

Best Practices Design Tuning Database 52

The Morning Paper

MAY 28, 2019

We don’t talk about energy as often as we probably should on this blog, but it’s certainly true that our data centres and various IT systems consume an awful lot of it. One efficient way of doing that in analog hardware is the use of current-starved inverters. Boosted race trees for low energy classification Tzimpragos et al.,

Energy 52

Energy Hardware Efficiency Architecture 52

Brendan Gregg

SEPTEMBER 5, 2017

. - **eBPF**: tracing features completed in 2016, this provides efficient programmatic tracing to existing kernel frameworks. Here's some output from my zfsdist tool, in bcc/BPF, which measures ZFS latency as a histogram on Linux: # zfsdist. Tracing ZFS operation latency. Hit Ctrl-C to end. ^C

Virtualization 76

Virtualization AWS Engineering Hardware 76

The Netflix TechBlog

MARCH 6, 2019

Mounting object storage in Netflix’s media processing platform By Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. Our object storage service splits objects into many parts and stores them in S3.

Media 214

Media Storage Processing Cache 214

Dynatrace

AUGUST 5, 2019

In this role, I am leading a global team that works closely with our strategic partners such as AWS, Microsoft, Google, Pivotal, Red Hat and others. Remember: This is a critical aspect as you do not want to migrate a service and suddenly introduce high latency or costs to a system that you forgot about having a dependency with!

Cloud 31

Cloud Traffic Database Network 31

Smashing Magazine

JANUARY 7, 2019

Estimated Input Latency tells us if we are hitting that threshold, and ideally, it should be below 50ms. In exchange, your team gains maintainability and developer efficiency, of course. But this consideration needs to be deliberate. Still, it’s not that difficult , and its support is coming, e.g. it’s available since Apache 2.4.26.

Performance 103

Performance Cache Metrics Network 103

Smashing Magazine

JANUARY 6, 2020

Estimated Input Latency tells us if we are hitting that threshold, and ideally, it should be below 50ms. Designed for the modern web, it responds to actual congestion, rather than packet loss like TCP does, it is significantly faster , with higher throughput and lower latency — and the algorithm works differently.

Performance 145

Performance Cache Network Metrics 145

Smashing Magazine

JANUARY 11, 2021

Estimated Input Latency tells us if we are hitting that threshold, and ideally, it should be below 50ms. Designed for the modern web, it responds to actual congestion, rather than packet loss like TCP does, it is significantly faster , with higher throughput and lower latency — and the algorithm works differently.

Performance 145

Performance Cache Media Metrics 145