The Netflix TechBlog

MARCH 7, 2024

Compute: Titus Whereas open-source users of Metaflow rely on AWS Batch or Kubernetes as the compute backend , we rely on our centralized compute-platform, Titus. We have talked about the importance of a production-grade workflow orchestrator in the context of Metaflow when we released support for AWS Step Functions years ago.

Systems 226

Systems Media Cache Open Source 226

Dynatrace

OCTOBER 10, 2022

Many organizations today rely on cloud-native applications for their scalability and agility, among other benefits. Popular examples include AWS Lambda and Microsoft Azure Functions , but new providers are constantly emerging as this model becomes more mainstream. Serverless benefits include the following: Dynamic scalability.

Serverless 163

Serverless Infrastructure Lambda Latency 163

All Things Distributed

NOVEMBER 13, 2014

When AWS launched, it changed how developers thought about IT services: What used to take weeks or months of purchasing and provisioning turned into minutes with Amazon EC2. Our answer is a new compute service called AWS Lambda. You can go from code to service in three clicks and then let AWS Lambda take care of the rest.

Lambda 136

Lambda AWS Cloud Scalability 136

The Netflix TechBlog

OCTOBER 19, 2020

If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. The next challenge was to stream large amounts of traces via a scalable data processing platform.

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

All Things Distributed

JANUARY 23, 2012

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - The AWS Storage Gateway. Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customersâ?? Once the AWS Storage Gatewayâ??s Comments ().

Storage 115

Storage AWS Virtualization Cloud 115

All Things Distributed

DECEMBER 14, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Introducing the AWS South America (Sao Paulo) Region. Today, Amazon Web Services is expanding its worldwide coverage with the launch of a new AWS Region in Sao Paulo, Brazil. RAMA , a Brazilian financial services firm and AWS customer : â??The

AWS 114

AWS Latency Storage Big Data 114

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows. Wednesday?—?December

AWS 37

AWS Entertainment Open Source Benchmarking 37

Dynatrace

SEPTEMBER 30, 2021

Cloud services platforms like AWS, Azure, and GCP are reshaping how organizations deliver value to their customers, making cloud migration an increasingly attractive option for running applications. Increased scalability. But what does it take to migrate your existing applications to the cloud? What is cloud migration? Reduced cost.

Cloud 163

Cloud Traffic Best Practices Strategy 163

ScaleOut Software

APRIL 6, 2021

With the ScaleOut Digital Twin Streaming Service , an Azure-hosted cloud service, ScaleOut Software introduced breakthrough capabilities for streaming analytics using the real-time digital twin concept. Scaleout StreamServer® DT was created to meet this need.

IoT 52

IoT Azure Analytics Healthcare 52

All Things Distributed

NOVEMBER 12, 2012

Werner Vogels weblog on building scalable and robust distributed systems. Today, Amazon Web Services has greater worldwide coverage with the launch of a new AWS Region in Sydney, Australia. Many young businesses as well as established enterprises are already using AWS, many of them targeting customers globally. Comments ().

Cloud 117

Cloud AWS Ecommerce Latency 117

All Things Distributed

APRIL 28, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - Opening the AWS Asia Pacific (Singapore) Region. Today Amazon Web Services has taken another important step in serving customers worldwide: the AWS Asia Pacific (Singapore) Region is now launched. All Things Distributed. Comments ().

AWS 61

AWS Cloud Latency Storage 61

Dotcom-Montior

OCTOBER 6, 2021

A site reliability engineer, or SRE, is a role that that encompasses aspects of both software engineering and operations/infrastructure. At that time, the team was made up of software engineers. Site reliability engineers bridge the gap between operations and software developers. What are Some Common SRE Responsibilities?

Engineering 67

Engineering DevOps Monitoring Google 67

The Morning Paper

AUGUST 29, 2019

If you’re moving an OLAP workload to the cloud (AWS in the context of this paper), what DBMS setup should you go with? We focused on OLAP-oriented parallel data warehouse products available for AWS and restricted our attention to commercially available systems. Scalability. VLDB’19. Key findings. Query restrictions.

Architecture 76

Architecture Cloud Storage Serverless 76

All Things Distributed

JULY 13, 2010

Werner Vogels weblog on building scalable and robust distributed systems. During my academic career, I spent many years working on HPC technologies such as user-level networking interfaces, large scale high-speed interconnects, HPC software stacks, etc. All Things Distributed. By Werner Vogels on 12 July 2010 05:00 PM. Comments ().

Cloud 118

Cloud AWS Automotive Latency 118

cdemi

DECEMBER 4, 2016

Modern software usually involves running tasks that invoke services. Maximizing Scalability : Number of Queues and Services can be varied to meet Demand. Azure Service Bus - The Go-To choice if you're already on Azure, High Throughput, Predictable Performance, Predictable Pricing, Secure, Scalable on Demand.

Design 47

Design Azure Scalability Latency 47

All Things Distributed

DECEMBER 5, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Route 53 has the business properties that you have come to expect from an AWS service: fully self-service and programmable, with transparent pay-as-you-go pricing and no minimum usage commitments. All Things Distributed. Comments (). Some fundamentals on Naming.

Cloud 117

Cloud Internet Internet AWS 117

All Things Distributed

APRIL 4, 2017

Today, I am very excited to announce our plans to open a new AWS Region in the Nordics! The new region will give Nordic-based businesses, government organisations, non-profits, and global companies with customers in the Nordics, the ability to leverage the AWS technology infrastructure from data centers in Sweden.

AWS 116

AWS Airlines Latency Games 116

VoltDB

JUNE 1, 2017

While the ultimate goal is still to save money, it’s human FTE hours as opposed to hardware and software costs. Latency Optimizers” – need support for very large federated deployments. They want to compete with AWS or Azure, and want to use OpenStack as a foundation for a commercial offering. The “Public Private Cloud” folks.

Hardware 56

Hardware Latency Virtualization Cloud 56

High Scalability

JULY 20, 2018

Investing tons of efforts into IT, building complicated deployment and clustering software etc. During our testing using the storage optimized EC2 instances (I3.2xlarge) we noticed that we were able to perform over 200K IOPS of 1K byte items thus meeting our throughput goals with latency rarely exceeding 1 millisecond.

Internet 121

Internet Internet Scalability Automotive 121

Percona

JULY 1, 2023

Now that Database-as-a-service (DBaaS) is in high demand, there are multiple questions regarding AWS services that cannot always be answered easily: When should I use Aurora and when should I use RDS MySQL ? These may be performance, high availability, operational cost, management, capacity planning, scalability, security, monitoring, etc.

AWS 52

AWS Database Serverless Storage 52

VoltDB

JUNE 1, 2017

While the ultimate goal is still to save money, it’s human FTE hours as opposed to hardware and software costs. Latency Optimizers” – need support for very large federated deployments. They want to compete with AWS or Azure, and want to use OpenStack as a foundation for a commercial offering. The “Public Private Cloud” folks.

Hardware 40

Hardware Latency Virtualization Cloud 40

The Netflix TechBlog

NOVEMBER 3, 2022

In that scenario, the system would need to deal with the data propagation latency directly, for example, by use of timeouts or client-originated update tracking mechanisms. We started seeing increased response latencies and leader servers running at dangerously high utilization.

Cache 224

Cache Latency Traffic Systems 224

All Things Distributed

JANUARY 18, 2012

Werner Vogels weblog on building scalable and robust distributed systems. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. The original Dynamo design was based on a core set of strong distributed systems principles resulting in an ultra-scalable and highly reliable database system.

Scalability 167

Scalability Database Ecommerce Latency 167

All Things Distributed

OCTOBER 12, 2016

Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis , a fully managed in-memory data store that operates at sub-millisecond latency. This allows for faster failover times while minimizing latency. Amazon’s enhancements address many day-to-day challenges with running Redis.

Scalability 130

Scalability Analytics Cache AWS 130

Dynatrace

MAY 17, 2023

Making applications observable—relying on metrics, logs, and traces to understand what software is doing and how it’s performing—has become increasingly important as workloads are shifting to multicloud environments. OneAgent is the native telemetry data collector and monitoring solution of Dynatrace.

Metrics 168

Metrics Monitoring Database Network 168

Scalegrid

JANUARY 12, 2024

Strategic allocation of these resources plays a crucial role in achieving scalability, cost savings, improved performance, and staying ahead of advancements in the field. This also aids scalability down the line. This includes zero-day vulnerabilities and software weaknesses that are not yet known and can be exploited without warning.

Cloud 130

Cloud Virtualization Storage Efficiency 130

All Things Distributed

JULY 14, 2015

DynamoDB Streams is the enabling technology behind two other features announced today: cross-region replication maintains identical copies of DynamoDB tables across AWS regions with push-button ease, and triggers execute AWS Lambda functions on streams, allowing you to respond to changing data conditions. DynamoDB Streams.

Database 167

Database Lambda AWS IoT 167

Dynatrace

OCTOBER 1, 2021

In these modern environments, every hardware, software, and cloud infrastructure component and every container, open-source tool, and microservice generates records of every activity. The architects and developers who create the software must design it to be observed.

Metrics 363

Metrics Open Source Monitoring Infrastructure 363

The Netflix TechBlog

APRIL 26, 2022

We use Keystone as it is easy to use, reliable, scalable, and provides aggregation of facts from different cloud regions into a single AWS region. We plan to split these Keystone streams into multiple streams for horizontal scalability. We needed scalability testing and performance testing as well.

Storage 187

Storage Design Scalability Latency 187

All Things Distributed

MARCH 11, 2016

The epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Looking back over the past 10 years, there are hundreds of lessons that we’ve learned about building and operating services that need to be secure, reliable, scalable, with predictable performance at the lowest possible cost. Automation is key.

AWS 144

AWS Hardware Retail Virtualization 144

All Things Distributed

APRIL 9, 2015

Amazon ML is highly scalable and can generate billions of predictions, and serve those predictions in real-time and at high throughput. Details on the AWS Blog. AWS has been offering a range of storage solutions: objects, block storage, databases, archiving, etc. Details on the AWS Blog. The Amazon Elastic File System.

Lambda 122

Lambda Cloud IoT AWS 122

All Things Distributed

OCTOBER 2, 2017

We were pushing the limits of what was a leading commercial database at the time and were unable to sustain the availability, scalability and performance needs that our growing Amazon business demanded. So, we set out to build a fully hosted AWS database service based upon the original Dynamo design.

Internet 128

Internet Internet AWS Performance 128

All Things Distributed

JUNE 21, 2017

You can add DAX to your existing DynamoDB applications with just a few clicks in the AWS Management Console – no application rewrites required. Based on this experience and learning, we built DynamoDB to be a fast, highly scalable NoSQL database to meet the needs of Internet-scale applications. Saving crucial milliseconds.

Speed 121

Speed Cache Latency AWS 121

Percona

MAY 2, 2023

Also, we will take a look at our open-source backup utility custom-built to help avoid costs and proprietary software – Percona Backup for MongoDB or PBM. It also supports storing the backups on filesystem and AWS S3 buckets. root@ip-172-31-37-92 ~]# aws ec2 describe-snapshots --query "sort_by(Snapshots, &StartTime)[-1].{SnapshotId:SnapshotId,StartTime:StartTime}"

Best Practices 52

Best Practices Database Storage Servers 52

The Netflix TechBlog

AUGUST 16, 2023

Each of the 3 colors represents our deployment regions in AWS, and the blue vertical line shows when we turned on the feature. System Metrics Given the significant reduction in connections, we saw reduced CPU utilization (~4%), heap usage (~15%), and latency (~3%) on Zuul, as well.

Traffic 170

Traffic Servers Google Metrics 170

All Things Distributed

NOVEMBER 29, 2017

Though the AWS Cloud gives you access to the storage and processing power required for ML, the process for building, training, and deploying ML models has unique challenges that often block successful use of this powerful new technology. An elastic, secure, and scalable environment to host your models, with one-click deployment.

Tuning 94

Tuning AWS Scalability Infrastructure 94

Smashing Magazine

AUGUST 20, 2021

Throughout the web’s history, static websites have always been a popular option due to their simplicity, scalability, and security. Twitch developer documentation hosted on AWS, edited on CloudCannon. Netlify , Vercel , CloudFlare , and AWS all have the concept of serverless functions run at edge nodes of a CDN. Large preview ).

Ecommerce 139

Ecommerce Website Government Internet 139