The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

SEPTEMBER 24, 2021

Lastly, the packager kicks in, adding a system layer to the asset, making it ready to be consumed by the clients. From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step.

Cloud 237

Cloud Media Storage Cache 237

All Things Distributed

JANUARY 23, 2012

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - The AWS Storage Gateway. Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customersâ?? s storage infrastructure.

Storage 116

Storage AWS Virtualization Cloud 116

The Netflix TechBlog

NOVEMBER 3, 2022

As the number of Titus users increased over the years, the load and pressure on the system increased substantially. cell): Titus Job Coordinator is a leader elected process managing the active state of the system. When a new leader is elected it loads all data from external storage.

Cache 224

Cache Latency Traffic Systems 224

All Things Distributed

JUNE 7, 2017

Unlocking the value of data is a primary goal that AWS helps our customers to pursue. Some applications – medical equipment, industrial machinery, and building automation are just a few – can't rely exclusively on the cloud for control, and require some form of local storage and execution. Law of the Land.

AWS 120

AWS IoT Lambda Storage 120

The Netflix TechBlog

OCTOBER 19, 2020

which is difficult when troubleshooting distributed systems. If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Scalegrid

MARCH 14, 2024

Understanding Hybrid Cloud Strategy A hybrid cloud merges the capabilities of public and private clouds into a singular, coherent system. Managing Costs Lower storage expenses in the public cloud and the ability to scale resources as needed enhance the cost-effectiveness of a hybrid cloud environment compared to on-premises infrastructure.

Strategy 130

Strategy Cloud Artificial Intelligence Infrastructure 130

Percona

JULY 1, 2023

Now that Database-as-a-service (DBaaS) is in high demand, there are multiple questions regarding AWS services that cannot always be answered easily: When should I use Aurora and when should I use RDS MySQL ? Amazon Aurora is a proprietary, cloud-native, fully managed relational database service developed by Amazon Web Services (AWS).

AWS 52

AWS Database Serverless Storage 52

All Things Distributed

DECEMBER 12, 2018

In April 2017, Amazon Web Services announced that it would launch a new AWS infrastructure region Region in Sweden. Today, I'm happy to announce that the AWS Europe (Stockholm) Region, our 20th Region globally, is now generally available for use by customers. Public sector.

AWS 124

AWS Cloud Games Serverless 124

High Scalability

SEPTEMBER 8, 2018

No, I don’t think that is because AWS is earning a 355x margin on DynamoDB! The issue is that Anna is now orders of magnitude more efficient than competing systems, in addition to being orders of magnitude faster. Each storage server collects statistics about the requests it serves, the data it stores, etc.

Storage 199

Storage Performance AWS Efficiency 199

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on. Note that my predictions in this talk may be wrong, but they should be thought provoking.

Performance 140

Performance Latency Hardware Storage 140

The Netflix TechBlog

OCTOBER 27, 2020

The processed data is typically stored as data warehouse tables in AWS S3. The data warehouse is not designed to serve point requests from microservices with low latency. Therefore, we must efficiently move data from the data warehouse to a global, low-latency and highly-reliable key-value store.

Latency 243

Latency Storage Big Data Tuning 243

Scalegrid

JANUARY 12, 2024

Simply put, it’s the set of computational tasks that cloud systems perform, such as hosting databases, enabling collaboration tools, or running compute-intensive algorithms. Such demanding use cases place a great value on systems capable of fast and reliable execution, a need that spans across various industry segments.

Cloud 130

Cloud Virtualization Storage Efficiency 130

The Netflix TechBlog

APRIL 26, 2022

The first version of our logger library optimized for storage by deduplicating facts and optimized for network i/o using different compression methods for each fact. Since we were optimizing at the logging level for storage and performance, we had less data and metadata to play with to optimize the query performance.

Storage 187

Storage Design Scalability Latency 187

Percona

NOVEMBER 3, 2023

Rather than listing the concepts, function calls, etc, available in Citus, which frankly is a bit boring, I’m going to explore scaling out a database system starting with a single host. I won’t cover all the features but show just enough that you’ll want to see more of what you can learn to accomplish for yourself.

Database 102

Database Benchmarking Latency C++ 102

The Morning Paper

FEBRUARY 6, 2020

On the Cloudburst design teams’ wish list: A running function’s ‘hot’ data should be kept physically nearby for low-latency access. The canononical cloud platform architecture decouples storage and compute services so that each can be scaled and operated independently, i.e., they are disaggregated. Consistency.

Lambda 98

Lambda Serverless Cache Latency 98

All Things Distributed

NOVEMBER 13, 2014

When AWS launched, it changed how developers thought about IT services: What used to take weeks or months of purchasing and provisioning turned into minutes with Amazon EC2. Capital-intensive storage solutions became as simple as PUTting and GETting objects in Amazon S3. Our answer is a new compute service called AWS Lambda.

Lambda 136

Lambda AWS Cloud Scalability 136

Brendan Gregg

FEBRUARY 28, 2023

This talk originated from my updates to [Systems Performance 2nd Edition], and this was the first time I've given this talk in person! CXL in a way allows a custom memory controller to be added to a system, to increase memory capacity, bandwidth, and overall performance. Ford, et al., “TCP

Performance 110

Performance Latency Cache Virtualization 110

All Things Distributed

DECEMBER 14, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Introducing the AWS South America (Sao Paulo) Region. Today, Amazon Web Services is expanding its worldwide coverage with the launch of a new AWS Region in Sao Paulo, Brazil. RAMA , a Brazilian financial services firm and AWS customer : â??The

AWS 115

AWS Latency Storage Big Data 115

Scalegrid

JANUARY 8, 2024

Artificial intelligence can automate tasks ranging from: data analysis resource provisioning system maintenance decision-making natural language processing This not only improves accuracy and reliability but also frees up valuable time for IT teams to focus on strategic tasks, such as resource management on platforms like Google Cloud.

Artificial Intelligence 147

Artificial Intelligence Cloud Scalability Analytics 147

All Things Distributed

MARCH 2, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - Introducing the AWS Asia Pacific (Tokyo) Region. Today Amazon Web Services is expanding its world-wide coverage with the launch of a new AWS Region located in Tokyo, Japan. For more details on AWS in Japan see [link].

AWS 114

AWS Cloud Games Latency 114

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows.

AWS 37

AWS Entertainment Open Source Benchmarking 37

The Netflix TechBlog

FEBRUARY 9, 2021

Server-generated assets, since client-side generation would require the retrieval of many individual images, which would increase latency and time-to-render. To reduce latency, assets should be generated in an offline fashion and not in real time. This requires an asset storage solution.

Engineering 165

Engineering Storage Latency Entertainment 165

All Things Distributed

MARCH 11, 2016

The epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Given that AWS is a pioneer in building and operating these services world-wide, these lessons have been of crucial importance to our business. Build evolvable systems. Systems need to keep running even if the “house is on fire.”

AWS 147

AWS Hardware Retail Virtualization 147

All Things Distributed

AUGUST 20, 2015

Today, we are releasing a plugin that allows customers to use the Titan graph engine with Amazon DynamoDB as the backend storage layer. It opens up the possibility to enjoy the value that graph databases bring to relationship-centric use cases, without worrying about managing the underlying storage. The importance of relationships.

Database 134

Database Logistics Availability Social Media 134

The Morning Paper

AUGUST 29, 2019

If you’re moving an OLAP workload to the cloud (AWS in the context of this paper), what DBMS setup should you go with? We focused on OLAP-oriented parallel data warehouse products available for AWS and restricted our attention to commercially available systems. Choosing a cloud DBMS: architectures and tradeoffs Tan et al.,

Architecture 76

Architecture Cloud Storage Serverless 76

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on. Note that my predictions in this talk may be wrong, but they should be thought-provoking.

Performance 52

Performance Latency Hardware Storage 52

Adrian Cockcroft

APRIL 18, 2018

In reality, in any non-trivial installation, there are multiple tools collecting, storing and displaying overlapping sets of metrics from many types of systems and different levels of abstraction. This is true for CPU, network, memory and storage, and also for bare metal, virtual machines, containers, processes, functions and threads.

Monitoring 40

Monitoring Systems Virtualization Metrics 40

High Scalability

DECEMBER 21, 2018

2) Cloud will decentralise in terms of provision not power i.e. Amazon will "invade" more of those holdouts with AWS Outpost. odrotbohm : I’ve seen microservice based systems more deranged after 2 years than any 1,5 decades old monolith could ever have been. 202,157 flights tracked! Me : Nothing special. If you plan for it.

Internet 137

Internet Internet Scalability Serverless 137

High Scalability

JULY 20, 2018

The third wing of the architecture piece is the “domain specific system-on-chip.” tef : You can use a message broker to glue systems together, but never use one to cut systems apart. CompSciFact : 'Fancy algorithms are slow when n is small, and n is usually small.' -- Rob Pike. There are a few more quotes.

Internet 121

Internet Internet Scalability Automotive 121

All Things Distributed

OCTOBER 2, 2017

With these requirements in mind, and a willingness to question the status quo, a small group of distributed systems experts came together and designed a horizontally scalable distributed database that would scale out for both reads and writes to meet the long-term needs of our business. This was the genesis of the Amazon Dynamo database.

Internet 130

Internet Internet AWS Performance 130

All Things Distributed

AUGUST 16, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - The AWS GovCloud (US) Region. Today AWS announced the launch of the AWS GovCloud (US) Region. The concept of regions gives AWS customers control over the placement of their resources and services. All Things Distributed.

AWS 113

AWS Government Big Data Cloud 113

All Things Distributed

APRIL 9, 2015

Details on the AWS Blog. The Amazon Elastic File System. AWS has been offering a range of storage solutions: objects, block storage, databases, archiving, etc. Amazon EFS is a fully-managed service that makes it easy to set up and scale shared file storage in the AWS Cloud. Details on the AWS Blog.

Lambda 123

Lambda Cloud IoT AWS 123

The Morning Paper

MARCH 3, 2020

It takes you through the thinking processes and engineering practices behind the design of a key part of the control plane for AWS Elastic Block Storage (EBS): the Physalia database that stores configuration information. For Physalia, and for AWS more generally, the guiding principle is minimise the blast radius.

Database 106

Database AWS Network Design 106

Brendan Gregg

FEBRUARY 28, 2023

This talk originated from my updates to Systems Performance 2nd Edition , and this was the first time I've given this talk in person! CXL in a way allows a custom memory controller to be added to a system, to increase memory capacity, bandwidth, and overall performance. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

All Things Distributed

JANUARY 18, 2012

Werner Vogels weblog on building scalable and robust distributed systems. The original Dynamo design was based on a core set of strong distributed systems principles resulting in an ultra-scalable and highly reliable database system. Amazon DynamoDB offers low, predictable latencies at any scale. All Things Distributed.

Scalability 167

Scalability Database Ecommerce Latency 167