The Netflix TechBlog

OCTOBER 28, 2021

Netflix’s unique work culture and petabyte-scale data problems are what drew me to Netflix. During earlier years of my career, I primarily worked as a backend software engineer, designing and building the backend systems that enable big data analytics. You can learn more about it from my talk at the Flink forward conference.

Data Engineering 173

Data Engineering Engineering Big Data Software Engineering 173

Dynatrace

JANUARY 11, 2023

With more organizations taking the multicloud plunge, monitoring cloud infrastructure is critical to ensure all components of the cloud computing stack are available, high-performing, and secure. For example, uptime detection can identify database instability and help to improve mean time to restoration. Cloud storage monitoring.

Cloud 219

Cloud Monitoring Best Practices Infrastructure 219

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3.

Latency 243

Latency Storage Big Data Tuning 243

The Netflix TechBlog

MAY 4, 2023

It provides a good read on the availability and latency ranges under different production conditions. Given the scale of the data being generated using replay traffic, we record the responses from the two sides to a cost-effective cold storage facility using technology like Apache Iceberg.

Traffic 339

Traffic Latency Tuning Systems 339

All Things Distributed

AUGUST 20, 2012

Managing Cold Storage with Amazon Glacier. With the introduction of Amazon Glacier , IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage. With Amazon Glacier any organization now has access to the same data archiving capabilities as the worldâ??s

Storage 122

Storage Cloud AWS Media 122

Dynatrace

FEBRUARY 16, 2023

As teams try to gain insight into this data deluge, they have to balance the need for speed, data fidelity, and scale with capacity constraints and cost. To solve this problem, Dynatrace launched Grail, its causational data lakehouse , in 2022. Logs on Grail Log data is foundational for any IT analytics.

Analytics 182

Analytics Innovation Metrics Database 182

Percona

JANUARY 19, 2023

However, there are cases where the same column is defined on multiple indexes in order to serve different query patterns, and sometimes some of the indexes created for the same column are redundant, leading to more overhead when inserting or deleting data (as indexes are updated) and increased disk space for storing the indexes for the table.

Open Source 87

Open Source Storage Database Big Data 87

The Netflix TechBlog

OCTOBER 15, 2019

For example, XA transactions block execution if the application process fails during the prepare phase; moreover, XA provides no deadlock detection and no support for optimistic concurrency-control schemes. Thus, ensuring the atomicity of writes across different storage technologies remains a challenging problem for applications [3].

Transportation 192

Transportation Architecture Processing Storage 192

All Things Distributed

MAY 18, 2010

Expanding the Cloud - Amazon S3 Reduced Redundancy Storage. Today a new storage option for Amazon S3 has been launched: Amazon S3 Reduced Redundancy Storage (RRS). This new storage option enables customers to reduce their costs by storing non-critical, reproducible data at lower levels of redundancy. Comments ().

Storage 71

Storage Cloud AWS Scalability 71

All Things Distributed

DECEMBER 13, 2016

Today, I'm happy to announce that the AWS Europe (London) Region, our 16th technology infrastructure region globally, is now generally available for use by customers worldwide. Take Peterborough City Council as an example. Take GoSquared , a UK startup that runs all its development and production processes on AWS, as an example.

AWS 166

AWS Cloud Artificial Intelligence IoT 166

All Things Distributed

JUNE 26, 2016

Today, I’m happy to announce that the Asia Pacific (Mumbai) Region is generally available for use by customers worldwide. Here are the benefits of a comprehensive platform, with customer examples: A connected platform to sense the business environment. Advanced problem solving that connects big data with machine learning.

AWS 90

AWS Cloud Healthcare Blockchain 90

ScaleOut Software

APRIL 3, 2020

For example, the parameters for a ventilator could include its identifier, make and model, current location, status (in use, in storage, broken), time in use, technical issues and repairs, and contact information. Show me a list of currently available or soon to be available ventilators in my county right now.”.

Logistics 52

Logistics Analytics Scalability Cloud 52

ScaleOut Software

APRIL 3, 2020

For example, the parameters for a ventilator could include its identifier, make and model, current location, status (in use, in storage, broken), time in use, technical issues and repairs, and contact information. Show me a list of currently available or soon to be available ventilators in my county right now.”.

Logistics 52

Logistics Analytics Scalability Cloud 52

All Things Distributed

DECEMBER 8, 2016

Today, I'm happy to share that the Canada (Central) Region is available for use by customers worldwide. The AWS Cloud now operates in 40 Availability Zones within 15 geographic regions around the world, with seven more Availability Zones and three more regions coming online in China, France, and the U.K. in the coming year.

AWS 155

AWS Cloud Lambda Innovation 155

All Things Distributed

OCTOBER 7, 2015

However, the data infrastructure to collect, store and process data is geared toward developers (e.g., In AWS’ quest to enable the best data storage options for engineers, we have built several innovative database solutions like Amazon RDS, Amazon RDS for Aurora, Amazon DynamoDB, and Amazon Redshift. Big data challenges.

Cloud 137

Cloud Big Data AWS Analytics 137

All Things Distributed

FEBRUARY 17, 2011

Since a few days ago this weblog serves 100% of its content directly out of the Amazon Simple Storage Service (S3) without the need for a web server to be involved. been running at a traditional hosting site for many years until this preferred simple solution became available: today marks that day and I couldnt be happier about it.

AWS 134

AWS Website Storage Servers 134

All Things Distributed

APRIL 27, 2011

For example, to construct a product detail page for a customer visiting Amazon.com, our software calls on between 200 and 300 services to present a highly personalized experience for that customer. The storage systems weve pioneered demonstrate extreme scalability while maintaining tight control over performance, availability, and cost.

Technology 151

Technology Technology AWS Storage 151

Testsigma

MAY 7, 2020

IBM Big Data and Analytics Hub website cited a case study, where a US insurance company was estimating 15% of their testing efforts to be just test data collection for the backend system and the frontend system. The test data management for the company had become a big problem and had to be solved.

Testing 60

Testing Storage Database Processing 60

All Things Distributed

AUGUST 16, 2011

For example a number of our European customers are subject to data residency requirements when it comes to PII data and they use the EU Region to meet to those requirements. Government and Big Data. One particular early use case for AWS GovCloud (US) will be massive data processing and analytics.

AWS 111

AWS Government Big Data Cloud 111

All Things Distributed

FEBRUARY 25, 2011

When a new customer is onboarded, the ISV has to spin up a collection of AWS resources to run their web-servers, app-servers and databases in a multi-AZ (availability zone) setting to achieve high-availability. A simple scenario is for example the ability to clearly identify production from staging and development environments.

AWS 121

AWS Cloud Scalability Storage 121

All Things Distributed

MARCH 28, 2011

As a big music fan with well over 100Gb in digital music I am particularly excited that I now have access to all my digital music anywhere I go. What used to be only available in physical formats now often has digital equivalents and this digitalization is driving great new innovations. Driving Storage Costs Down for AWS Customers.

AWS 138

AWS Cloud Storage Internet 138

All Things Distributed

DECEMBER 9, 2010

Amazon S3 has always been a scalable, durable and available data repository for almost any customer workload. This is especially true for customers managing HD video or data-intensive instruments such as genomic sequencers. For example, a 2-hour movie on Blu-ray can be 50 gigabytes. At werner.ly Syndication. or rss feed.

AWS 80

AWS Big Data Scalability Storage 80

All Things Distributed

MAY 24, 2011

I am excited that today both the Route 53 , the highly available and scalable DNS service, and the Elastic Load Balancing teams are releasing new functionality that has been frequently requested by their customers: Route 53 now GA : Route 53 is now Generally Available and will provide an availability SLA of 100%. At werner.ly

Internet 114

Internet Internet AWS Scalability 114

All Things Distributed

DECEMBER 3, 2009

We have expanded the AWS footprint in the US and starting today a new AWS Region is available for use: US-West (Northern California). This new Region consists of multiple Availability Zones and provides low-latency access to the AWS services from for example the Bay Area. Driving Storage Costs Down for AWS Customers.

AWS 60

AWS Cloud Latency Storage 60

Highly Scalable

MARCH 1, 2012

And this was where a new evolution of data models began: Key-Value storage is a very simplistic, but very powerful model. NoSQL data modeling is typically driven by application-specific access patterns, i.e. the types of queries to be supported. Many techniques that are described below are perfectly applicable to this model.

Database 279

Database Ecommerce Efficiency Engineering 279

All Things Distributed

DECEMBER 5, 2010

I am very excited that today we have launched Amazon Route 53, a high-performance and highly-available Domain Name System (DNS) service. A simple example is the situation with Persons and Telephones; a person has a name, a person can have one or more telephones and each phone can have one or more telephone numbers. Comments ().

Cloud 117

Cloud Internet Internet AWS 117

All Things Distributed

NOVEMBER 15, 2010

This incredible power is available for anyone to use in the usual pay-as-you-go model, removing the investment barrier that has kept many organizations from adopting GPUs for their workloads even though they knew there would be significant performance benefit. The different stages were then load balanced across the available units.

AWS 136

AWS Latency Programming Architecture 136

All Things Distributed

DECEMBER 3, 2009

Today a powerful new feature is available for our Amazon EC2 customers: the ability to boot their instances from Amazon EBS (Elastic Block Store). A wide variety of operating systems and software configurations is available for use. with new security patches installed), or add new user data. Comments (). At werner.ly

AWS 60

AWS Storage Operating System Cloud 60

All Things Distributed

APRIL 28, 2010

Availability - The cloud makes it possible to build resilient applications to make sure they can survive different failure scenarios. Currently, each AWS Region contains multiple Availability Zones, which are distinct locations that are engineered to be insulated from failures in other Availability Zones.

AWS 61

AWS Cloud Latency Storage 61

All Things Distributed

JULY 13, 2010

Today, I am very proud to be a part of the Amazon Web Services team as we truly make HPC available as an on-demand commodity for every developer to use. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Driving down the cost of Big-Data analytics. HPC and Amazon EC2.

Cloud 118

Cloud AWS Automotive Latency 118

All Things Distributed

FEBRUARY 24, 2010

There are many factors that come into play when you need to meet stringent availability and performance requirements under ultra-scalable conditions. If you need to achieve high-availability and scalable performance, you will need to resort to data replication techniques. Data Consistency Models in the Amazon Services.

AWS 69

AWS Latency Database Scalability 69

All Things Distributed

DECEMBER 13, 2009

Consistently we have lowered compute, storage and bandwidth prices based on such cost savings. You are assured that your Reserved Instance will always be available in the Availability Zone in which you purchased it. This snapshot-restart technique is a well known methodology already available to many batch oriented applications.

Cloud 63

Cloud AWS Storage Innovation 63

Percona

JANUARY 14, 2019

Public API as -a-service has become a good business model: examples include social networks like Facebook/Twitter, messaging as a service like Twilio, and even credit card authorization platforms like Marqeta. With the latest ClickHouse version, all of these features are available, but some of them may not perform fast enough.

Database 40

Database Analytics Blockchain Healthcare 40