The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows.

AWS 100

AWS Entertainment Open Source Benchmarking 100

All Things Distributed

OCTOBER 25, 2018

Expanding the AWS Cloud—An AWS Region is coming to South Africa! Today, I am excited to announce our plans to open a new AWS Region in South Africa! AWS is committed to South Africa's transformation. This news marks the 23rd AWS Region that we have announced globally. We have a long history in South Africa.

AWS 167

AWS Cloud Education Programming 167

The Netflix TechBlog

MARCH 6, 2019

—?Mounting object storage in Netflix’s media processing platform By Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. Encoding is not a one-time process?—?large We have one file?—?the

Media 214

Media Storage Processing Cache 214

Scalegrid

NOVEMBER 25, 2019

Released just four years ago in 2015, Scylla has averaged over 220% year-over-year growth in popularity according to DB-Engines. While Cassandra is still the most popular, ScyllaDB is gaining fast as the 7th most popular wide column store according to DB-Engines. of cloud deployments were reported running on AWS. Google Cloud.

Big Data 187

Big Data Database Open Source Azure 187

High Scalability

FEBRUARY 19, 2021

This means a system that is not merely available but is also engineered with extensive redundant measures to continue to work as its users expect. reliability situations, where continuity of service is essential, with redundant elements continuously in-service, such as with airplane engines. This ensures reliability.

Engineering 215

Engineering Systems Scalability Availability 215

The Netflix TechBlog

NOVEMBER 3, 2022

We introduce a caching mechanism in the API gateway layer, allowing us to offload processing from singleton leader elected controllers without giving up strict data consistency and guarantees clients observe. cell): Titus Job Coordinator is a leader elected process managing the active state of the system.

Cache 224

Cache Latency Traffic Systems 224

Percona

JULY 1, 2023

Now that Database-as-a-service (DBaaS) is in high demand, there are multiple questions regarding AWS services that cannot always be answered easily: When should I use Aurora and when should I use RDS MySQL ? What we should really compare is the MySQL and Aurora database engines provided by Amazon RDS. How do I choose which one to use?

AWS 52

AWS Database Serverless Storage 52

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which The process started with manual pull of member account information that was part of the session. We needed to increase engineering productivity via distributed request tracing.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3.

Latency 243

Latency Storage Big Data Tuning 243

The Netflix TechBlog

SEPTEMBER 24, 2021

By Xiaomei Liu , Rosanna Lee , Cyril Concolato Introduction Behind the scenes of the beloved Netflix streaming service and content, there are many technology innovations in media processing. Packaging has always been an important step in media processing. Uploading and downloading data always come with a penalty, namely latency.

Cloud 237

Cloud Media Storage Cache 237

Scalegrid

MARCH 14, 2024

Defining Hybrid Cloud Strategy The decision-making process about where to situate data and applications is vital to any hybrid cloud solution. Enterprise cloud architects and engineers require significant knowledge to ensure that private and public clouds work together compatibly.

Strategy 130

Strategy Cloud Artificial Intelligence Infrastructure 130

The Netflix TechBlog

FEBRUARY 20, 2019

by Jason Koch , with Martin Spier , Brendan Gregg , Ed Hunter Improving the tools available to our engineers to help them diagnose, triage, and work through software performance challenges in the cloud is a key goal for the cloud performance engineering team at Netflix. Vector is open source and in use by multiple companies.

Performance 211

Performance Latency Open Source Metrics 211

The Netflix TechBlog

AUGUST 22, 2019

Baking Windows with Packer By Justin Phelps and Manuel Correa Customizing Windows images at Netflix was a manual, error-prone, and time consuming process. A key responsibility of Netflix engineers is identifying gaps and pain points in the development and operation of services. Last year, we decided to improve the AMI baking process.

DevOps 207

DevOps AWS Tuning Infrastructure 207

The Netflix TechBlog

FEBRUARY 18, 2022

Personalized Experience Refresh Netflix Recommendation engine continuously refreshes recommendations for every member. Event Prioritization Considering the use cases were wide ranging both in terms of their sources and their importance, we built segmentation into the event processing.

Systems 334

Systems Traffic Architecture Mobile 334

The Netflix TechBlog

AUGUST 29, 2023

A brief history of IPC at Netflix Netflix was early to the cloud, particularly for large-scale companies: we began the migration in 2008, and by 2010, Netflix streaming was fully run on AWS. For Inter-Process Communication (IPC) between services, we needed the rich feature set that a mid-tier load balancer typically provides.

Traffic 220

Traffic Latency Cloud C++ 220

Adrian Cockcroft

NOVEMBER 18, 2019

AWS Developer Relations on how the shift from Robot Operating System (ROS) 1 to ROS 2 will change the landscape for all robot lovers. OPN205-R Contributing to the AWS Construct Library (repeats) Using and loving the AWS Cloud Development Kit and want to help make it better? Alejandra Olvera-Novack?—?AWS

Open Source 40

Open Source AWS Lambda Serverless 40

Adrian Cockcroft

NOVEMBER 18, 2019

AWS Developer Relations on how the shift from Robot Operating System (ROS) 1 to ROS 2 will change the landscape for all robot lovers. OPN205-R Contributing to the AWS Construct Library (repeats) Using and loving the AWS Cloud Development Kit and want to help make it better? Alejandra Olvera-Novack?—?AWS

Open Source 40

Open Source AWS Lambda Serverless 40

Dynatrace

AUGUST 27, 2019

In Part 1 we explored how DevOps teams can prevent a process crash from taking down services across an organization in five easy steps. The Dynatrace all-in-one software intelligence platform gives your team real-time visibility into your underlying infrastructure —be it on bare metal, VMware, OpenStack, AWS, Azure, or a hybrid solution.

Systems 177

Systems DevOps Latency Azure 177

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. On the other hand, these optimizations themselves need to be sufficiently inexpensive to justify their own processing cost over the gains they bring.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows.

AWS 37

AWS Entertainment Open Source Benchmarking 37

All Things Distributed

JUNE 21, 2018

In fact, this is been proven by our customers as Amazon Aurora remains the fastest growing service in AWS history. Use cases such as gaming, ad tech, and IoT lend themselves particularly well to the key-value data model where the access patterns require low-latency Gets/Puts for known key values. The opposite is true.

Database 167

Database AWS Games Latency 167

Adrian Cockcroft

JANUARY 20, 2023

switches to connect processing nodes, pooled memory and I/O resources into very large coherent fabrics within a rack, and use Ethernet between racks. I have always been particularly interested in the interconnects and protocols used to create clusters, and the latency and bandwidth of the various offerings that are available.

Architecture 52

Architecture Latency Benchmarking AWS 52

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. What about short-lived processes, like a service restarting in a loop? I happened to be speaking at a technical confering while still debugging this, and mentioned what I was working on to a processor engineer.

Speed 126

Speed Java AWS Virtualization 126

The Morning Paper

FEBRUARY 6, 2020

On the Cloudburst design teams’ wish list: A running function’s ‘hot’ data should be kept physically nearby for low-latency access. A low-latency autoscaling KVS can serve as both global storage and a DHT-like overlay network. Oh, and there’s a scheduler too of course to keep all the plates spinning.

Lambda 98

Lambda Serverless Cache Latency 98

Smashing Magazine

AUGUST 3, 2023

These pages serve as a pivotal tool in our digital marketing strategy, not only providing valuable information about our services but also designed to be easily discoverable through search engines. Large preview ) We’ve known for a long time that fast page performance influences search engine rankings. SEO is key to our success.

Performance 74

Performance Cache Traffic Metrics 74

The Morning Paper

MARCH 3, 2020

It takes you through the thinking processes and engineering practices behind the design of a key part of the control plane for AWS Elastic Block Storage (EBS): the Physalia database that stores configuration information. Engineering decisions involve making lots of trade-offs. Millions of tiny databases , Brooker et al.,

Database 106

Database AWS Network Design 106

All Things Distributed

SEPTEMBER 25, 2017

Today, I am excited to announce plans for Amazon Web Services (AWS) to bring an infrastructure Region to the Middle East! Based in Bahrain, this will be the first Region for AWS in the Middle East. This Region will consist of three Availability Zones at launch, and it will provide even lower latency to users across the Middle East.

Cloud 152

Cloud Education Energy Government 152

Scalegrid

JANUARY 12, 2024

These vendors serve data center players and offer advanced options, such as ScaleGrid’s engine, which ensures that different elements work well together automatically, eliminating the need for manual effort in managing heterogeneous environments. Ultimately improving efficiency while minimizing errors.

Cloud 130

Cloud Virtualization Storage Efficiency 130

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 110

Performance Latency Cache Virtualization 110

All Things Distributed

MARCH 11, 2016

The epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Given that AWS is a pioneer in building and operating these services world-wide, these lessons have been of crucial importance to our business. AWS helps its customers do this too.

AWS 144

AWS Hardware Retail Virtualization 144

The Netflix TechBlog

APRIL 26, 2022

Fig 2: Internal components of Axion Axion’s fact logging client logs facts to the keystone real-time stream processing platform , which outputs data to an Iceberg table. We use Keystone as it is easy to use, reliable, scalable, and provides aggregation of facts from different cloud regions into a single AWS region.

Storage 187

Storage Design Scalability Latency 187

Dynatrace

SEPTEMBER 30, 2021

Cloud services platforms like AWS, Azure, and GCP are reshaping how organizations deliver value to their customers, making cloud migration an increasingly attractive option for running applications. A key step in digital transformation is migrating from traditional on-prem IT processes to adopting cloud services.

Cloud 169

Cloud Traffic Best Practices Strategy 169

IO River

NOVEMBER 2, 2023

‍Think of a situation where you're asked to build a service in AWS that distributes static content to your users. Your first option is to use its native CDN - Amazon CloudFront, as it seamlessly integrates with all of your other AWS Services that you're using to build your service.‍For What is vendor lock-in?‍Think

Lambda 52

Lambda AWS Cache Website 52

The Netflix TechBlog

AUGUST 13, 2020

Over the years we’ve learned from on-call engineers about the pain points of application monitoring: too many alerts, too many dashboards to scroll through, and too much configuration and maintenance. It usually has dependencies, talks to other services, and lives in different AWS regions. Client metrics and QoE changes.

Monitoring 210

Monitoring Tuning Traffic Metrics 210

All Things Distributed

OCTOBER 12, 2016

Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis , a fully managed in-memory data store that operates at sub-millisecond latency. This allows for faster failover times while minimizing latency. Amazon’s enhancements address many day-to-day challenges with running Redis.

Scalability 130

Scalability Analytics Cache AWS 130

All Things Distributed

JUNE 21, 2017

You can add DAX to your existing DynamoDB applications with just a few clicks in the AWS Management Console – no application rewrites required. DynamoDB was the first service at AWS to use SSD storage. These high-throughput, low-latency requirements need caching, not as a consideration, but as a best practice.

Speed 121

Speed Cache Latency AWS 121

All Things Distributed

NOVEMBER 26, 2013

As I discussed in my re:Invent keynote earlier this month, I am now happy to announce the immediate availability of Amazon RDS Cross Region Read Replicas , which is another important enhancement for our customers using or planning to use multiple AWS Regions to deploy their applications. Cross Region Read Replicas are available for MySQL 5.6

Cloud 145

Cloud AWS Traffic Latency 145