The Netflix TechBlog

OCTOBER 27, 2020

Usually Data scientists and engineers write Extract-Transform-Load (ETL) jobs and pipelines using big data compute technologies, like Spark or Presto , to process this data and periodically compute key information for a member or a video. The processed data is typically stored as data warehouse tables in AWS S3.

Latency 243

Latency Storage Big Data Tuning 243

All Things Distributed

AUGUST 20, 2012

Werner Vogels weblog on building scalable and robust distributed systems. Managing Cold Storage with Amazon Glacier. With the introduction of Amazon Glacier , IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage. All Things Distributed. Comments ().

Storage 122

Storage Cloud AWS Media 122

All Things Distributed

JUNE 26, 2016

In June 2015, Amazon Web Services announced that it would launch a new AWS infrastructure region in India. Examples of continuous sensing are found in the managed cloud platform built by Rachio on AWS IoT to enable the secure interaction of its connected devices with cloud applications/other devices. The opportunity to revolutionize.

AWS 90

AWS Cloud Healthcare Blockchain 90

All Things Distributed

JULY 7, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - AWS Import/Export Support for Amazon EBS. The AWS Import/Export team has announced today that they have expanded their functionality significantly by adding Import into Amazon EBS. Driving Storage Costs Down for AWS Customers.

AWS 96

AWS Cloud Storage Internet 96

All Things Distributed

DECEMBER 14, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Introducing the AWS South America (Sao Paulo) Region. Today, Amazon Web Services is expanding its worldwide coverage with the launch of a new AWS Region in Sao Paulo, Brazil. Additionally, it allows them to keep their data inside of Brazil.

AWS 114

AWS Latency Storage Big Data 114

All Things Distributed

AUGUST 16, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - The AWS GovCloud (US) Region. Today AWS announced the launch of the AWS GovCloud (US) Region. The concept of regions gives AWS customers control over the placement of their resources and services. All Things Distributed.

AWS 111

AWS Government Big Data Cloud 111

All Things Distributed

FEBRUARY 17, 2011

Werner Vogels weblog on building scalable and robust distributed systems. New AWS feature: Run your website from Amazon S3. Since a few days ago this weblog serves 100% of its content directly out of the Amazon Simple Storage Service (S3) without the need for a web server to be involved. Countdown to What is Next in AWS.

AWS 134

AWS Website Storage Servers 134

All Things Distributed

JULY 16, 2012

Werner Vogels weblog on building scalable and robust distributed systems. Register for AWS re: Invent. The first annual AWS user and partner conference will be held November 27-29 at The Venetian in Las Vega. All Things Distributed. By Werner Vogels on 16 July 2012 09:00 AM. Comments (). See the site for more details.

AWS 83

AWS Big Data Media Storage 83

The Netflix TechBlog

OCTOBER 15, 2019

Another issue exists for the capture of schema changes, where some systems, like MySQL, don’t support transactional schema changes [1][2]. By their nature, they can only rely on the lowest common denominator of participating systems. That system quickly grew very complex and became difficult to maintain.

Transportation 192

Transportation Architecture Processing Storage 192

All Things Distributed

MARCH 2, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - Introducing the AWS Asia Pacific (Tokyo) Region. Today Amazon Web Services is expanding its world-wide coverage with the launch of a new AWS Region located in Tokyo, Japan. For more details on AWS in Japan see [link].

AWS 112

AWS Cloud Games Latency 112

All Things Distributed

FEBRUARY 25, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Simplifying IT - Create Your Application with AWS CloudFormation. With the launch of AWS CloudFormation today another important step has been taken in making it easier for customers to deploy applications to the cloud. All Things Distributed. Comments ().

AWS 121

AWS Cloud Scalability Storage 121

All Things Distributed

MAY 18, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - Amazon S3 Reduced Redundancy Storage. Today a new storage option for Amazon S3 has been launched: Amazon S3 Reduced Redundancy Storage (RRS). Under the covers Amazon S3 is a marvel of distributed systems technologies.

Storage 71

Storage Cloud AWS Scalability 71

All Things Distributed

JUNE 29, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving Bandwidth Cost Down for AWS Customers. Any work we can do to improve over these dimensions generates long term benefits for AWS customers. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway.

AWS 104

AWS Retail Innovation Strategy 104

All Things Distributed

AUGUST 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Job Openings in AWS - Senior Leader in Database Services. This week it is an opening for senior leaders with AWS Database Services. The ideal candidate will be someone who has built and ran large scale distributed systems and/or databases.

AWS 75

AWS Database Storage Scalability 75

All Things Distributed

JANUARY 19, 2011

Werner Vogels weblog on building scalable and robust distributed systems. AWS Elastic Beanstalk: A Quick and Simple Way into the Cloud. Additionally customers are not restricted to AWS services; they can mix-and-match services from other providers to best meet their needs. By Werner Vogels on 18 January 2011 04:00 PM.

AWS 115

AWS Cloud Java Scalability 115

All Things Distributed

MARCH 24, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Hacking with AWS at The Next Web Hackaton. It is likely that the Amazon Web Services will be used by many of the participants for their compute, storage, database and other cloud resource needs. Driving Storage Costs Down for AWS Customers.

AWS 93

AWS Internet Internet Storage 93

All Things Distributed

APRIL 27, 2011

Werner Vogels weblog on building scalable and robust distributed systems. And while many of our systems are based on the latest in computer science research, this often hasnt been sufficient: our architects and engineers have had to advance research in directions that no academic had yet taken. All Things Distributed. Comments ().

Technology 151

Technology Technology AWS Storage 151

All Things Distributed

APRIL 28, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - Opening the AWS Asia Pacific (Singapore) Region. Today Amazon Web Services has taken another important step in serving customers worldwide: the AWS Asia Pacific (Singapore) Region is now launched. Comments (). and European Regions.

AWS 61

AWS Cloud Latency Storage 61

All Things Distributed

AUGUST 22, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Today AWS has launched Amazon ElastiCache , a new service that makes it easy to add distributed in-memory caching to any application. Systems that make extensive use of caching almost all report a significant reduction in the cost of their database tier.

Cloud 117

Cloud Cache AWS Storage 117

All Things Distributed

AUGUST 17, 2011

Werner Vogels weblog on building scalable and robust distributed systems. As some of you may remember I was pretty excited when Amazon Simple Storage Service (S3) released its website feature such that I could serve this weblog completely from S3. Driving Storage Costs Down for AWS Customers. All Things Distributed.

Servers 126

Servers Social Media AWS Website 126

All Things Distributed

DECEMBER 3, 2009

Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - New AWS Region: US-West (Northern California). We have expanded the AWS footprint in the US and starting today a new AWS Region is available for use: US-West (Northern California). Driving Storage Costs Down for AWS Customers.

AWS 60

AWS Cloud Latency Storage 60

All Things Distributed

MARCH 28, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Our smart phones and tablets are obvious examples, but many other devices are quickly gaining these capabilities; TV Sets and Hifi systems are internet enabled, and soon our treadmills and automobiles will be equally plugged into the digital world. Comments ().

AWS 138

AWS Cloud Storage Internet 138

All Things Distributed

DECEMBER 5, 2010

Werner Vogels weblog on building scalable and robust distributed systems. I am very excited that today we have launched Amazon Route 53, a high-performance and highly-available Domain Name System (DNS) service. Naming is one of the fundamental concepts in Distributed Systems. By Werner Vogels on 05 December 2010 02:00 PM.

Cloud 117

Cloud Internet Internet AWS 117

All Things Distributed

JANUARY 11, 2011

Werner Vogels weblog on building scalable and robust distributed systems. DROAM - Dreaming about Cheap Data Roaming. The one thing that I have always struggled with during my travels are the data plans of the cell phone companies. Theyre not even an AWS customer :-). Driving Storage Costs Down for AWS Customers.

Wireless 107

Wireless AWS Internet Internet 107

All Things Distributed

FEBRUARY 23, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Countdown to What is Next in AWS. Introducing the AWS South America (Sao Paulo) Region. Driving down the cost of Big-Data analytics.

AWS 78

AWS Storage Big Data Servers 78

All Things Distributed

MAY 24, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Many AWS customers use Elastic Load Balancers (ELB) to front their Internet facing applications. More information can found on the Elastic Load Balancing and the Route 53 detail pages and on the two blog posts on the AWS developer blog. Comments ().

Internet 114

Internet Internet AWS Scalability 114

All Things Distributed

JULY 3, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Next to customer visits I will take part in a number of events organized by AWS and by our partners. This week in Japan there are three public events planned: July 4 - AWS HPC Night at Fuji Soft Hall in Akihabara. Countdown to What is Next in AWS.

AWS 67

AWS Storage Big Data Cloud 67

All Things Distributed

DECEMBER 9, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Big Just Got Bigger - 5 Terabyte Object Support in Amazon S3. By supporting such large object sizes, Amazon S3 better enables a variety of interesting big data use cases. Driving Storage Costs Down for AWS Customers. Comments ().

AWS 80

AWS Big Data Scalability Storage 80

All Things Distributed

JULY 11, 2011

Werner Vogels weblog on building scalable and robust distributed systems. To get started using Spot or for more details visit the Amazon EC2 Spot Instance web page, the AWS developer blog , and the EC2 Release Notes. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway.

AWS 85

AWS Storage Cloud Big Data 85

All Things Distributed

JULY 13, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Not just for HPC but for mission critical enterprise systems such as OLTP. Dedicated High Performance Compute clusters require significant capital investments and their procurement often has longer lead times than many enterprise class server systems.

Cloud 118

Cloud AWS Automotive Latency 118

All Things Distributed

FEBRUARY 27, 2011

Werner Vogels weblog on building scalable and robust distributed systems. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Countdown to What is Next in AWS. Introducing the AWS South America (Sao Paulo) Region. Driving down the cost of Big-Data analytics.

AWS 60

AWS Storage Big Data Scalability 60

All Things Distributed

SEPTEMBER 29, 2010

Werner Vogels weblog on building scalable and robust distributed systems. I did write for those occasions where I thought additional background on AWS service and feature launches was relevant, but beyond that not much materialized. Driving Storage Costs Down for AWS Customers. Countdown to What is Next in AWS.

AWS 69

AWS Storage Big Data Scalability 69

All Things Distributed

SEPTEMBER 30, 2010

Werner Vogels weblog on building scalable and robust distributed systems. With existing customers I get a change to dive deep on their AWS usage and understand what works well and where we can do better. AWS Solutions Architect Matt Tavis will present on " Architecting for the Cloud ". Countdown to What is Next in AWS.

AWS 75

AWS Storage Cloud Best Practices 75

All Things Distributed

NOVEMBER 15, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Now that our ability to generate higher and higher clock rates has stalled and CPU architectural improvements have shifted focus towards multiple cores, we see that it is becoming harder to efficiently use these computer systems. All Things Distributed.

AWS 136

AWS Latency Programming Architecture 136

The Morning Paper

JUNE 23, 2019

Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. Why are developers using RInK systems as part of their design? Fetching too much data in a single query (i.e.,

Cache 79

Cache Latency Google Lambda 79

All Things Distributed

APRIL 23, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Countdown to What is Next in AWS. Introducing the AWS South America (Sao Paulo) Region. Driving down the cost of Big-Data analytics.

AWS 60

AWS Storage Cloud Big Data 60

All Things Distributed

NOVEMBER 19, 2010

Werner Vogels weblog on building scalable and robust distributed systems. Big news this week was of course the launch of Cluster GPU instances for Amazon EC2. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Countdown to What is Next in AWS. Comments ().

AWS 67

AWS Cloud Benchmarking Storage 67