All Things Distributed

JANUARY 23, 2012

Expanding the Cloud - The AWS Storage Gateway. Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customersâ?? AWS Identity and Access Management brings together on-premises and cloud identity management. Comments ().

Storage 115

Storage AWS Virtualization Cloud 115

Percona

JULY 1, 2023

Now that Database-as-a-service (DBaaS) is in high demand, there are multiple questions regarding AWS services that cannot always be answered easily: When should I use Aurora and when should I use RDS MySQL ? Amazon Aurora is a proprietary, cloud-native, fully managed relational database service developed by Amazon Web Services (AWS).

AWS 52

AWS Database Serverless Storage 52

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which Our distributed tracing infrastructure is grouped into three sections: tracer library instrumentation, stream processing, and storage. Storage: don’t break the bank!

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Scalegrid

JANUARY 12, 2024

Various forms can take shape when discussing workloads within the realm of cloud computing environments – examples include order management databases, collaboration tools, videoconferencing systems, virtual desktops, and disaster recovery mechanisms. Storage is a critical aspect to consider when working with cloud workloads.

Cloud 130

Cloud Virtualization Storage Efficiency 130

All Things Distributed

NOVEMBER 13, 2014

When AWS launched, it changed how developers thought about IT services: What used to take weeks or months of purchasing and provisioning turned into minutes with Amazon EC2. Capital-intensive storage solutions became as simple as PUTting and GETting objects in Amazon S3. Our answer is a new compute service called AWS Lambda.

Lambda 136

Lambda AWS Cloud Scalability 136

All Things Distributed

OCTOBER 16, 2019

There are places so remote, so harsh that humans can't safely explore them (for example, hundreds of miles below the earth, areas that experience extreme temperatures, or on other planets). Here's a few examples: IoT sensors in a factory can monitor and predict equipment failure before an accident.

IoT 126

IoT Healthcare Internet Internet 126

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows.

AWS 37

AWS Entertainment Open Source Benchmarking 37

All Things Distributed

AUGUST 16, 2011

Expanding the Cloud - The AWS GovCloud (US) Region. Today AWS announced the launch of the AWS GovCloud (US) Region. The concept of regions gives AWS customers control over the placement of their resources and services. All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems.

AWS 111

AWS Government Big Data Cloud 111

All Things Distributed

SEPTEMBER 25, 2012

Following the huge success of being able to provision a consistent, user-requested I/O rate for DynamoDB and Elastic Block Store (EBS), the AWS Database Services team has now released Provisioned IOPS, a new high performance storage option for the Amazon Relational Database Service (Amazon RDS). Provisioned IOPS storage in RDS.

Cloud 117

Cloud AWS Storage Database 117

All Things Distributed

DECEMBER 3, 2009

Expanding the Cloud - New AWS Region: US-West (Northern California). We have expanded the AWS footprint in the US and starting today a new AWS Region is available for use: US-West (Northern California). AWS is committed to making its services available at low cost. Driving Storage Costs Down for AWS Customers.

AWS 60

AWS Cloud Latency Storage 60

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203

Storage Latency Efficiency Data Engineering 203

All Things Distributed

APRIL 28, 2010

Expanding the Cloud - Opening the AWS Asia Pacific (Singapore) Region. Today Amazon Web Services has taken another important step in serving customers worldwide: the AWS Asia Pacific (Singapore) Region is now launched. The new Singapore Region offers customers in APAC lower-latency access to AWS services. Comments ().

AWS 61

AWS Cloud Latency Storage 61

The Morning Paper

MARCH 3, 2020

It takes you through the thinking processes and engineering practices behind the design of a key part of the control plane for AWS Elastic Block Storage (EBS): the Physalia database that stores configuration information. For Physalia, and for AWS more generally, the guiding principle is minimise the blast radius. NSDI’20.

Database 106

Database AWS Network Design 106

Percona

MAY 2, 2023

The speed of backup also depends on allocated IOPS and type of storage since lots of read/writes would be happening during this process. Back up anywhere – to the cloud (use any S3-compatible storage) or on-premise with a locally-mounted remote file system It allows you to choose which compression algorithms to use.

Best Practices 52

Best Practices Database Storage Servers 52

Percona

APRIL 17, 2023

The CFQ works well for many general use cases but lacks latency guarantees. The deadline excels at latency-sensitive use cases ( like databases ), and noop is closer to no schedule at all. For example, on a 128GB memory host, this can allow up to 38.4GB of dirty pages. Two other schedulers are deadline and noop.

Best Practices 52

Best Practices Design Tuning Database 52

The Morning Paper

JUNE 23, 2019

Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. The network latency of fetching data over the network, even considering fast data center networks. Who knew! ;).

Cache 79

Cache Latency Google Lambda 79

Dotcom-Montior

OCTOBER 6, 2021

For example, a site reliability engineer may be involved with help desk tickets, on-call incidents, manual tasks, etc. One minute an SRE might be provisioning storage in AWS, the next minute an SRE might have to talk to customers or go write some Python code for a new project. What Does a Site Reliability Engineer Do?

Engineering 67

Engineering DevOps Monitoring Google 67

All Things Distributed

JULY 13, 2010

Further computationally intensive, highly parallel workloads have found their way to Amazon EC2 as businesses have explored using HPC types of algorithms for other application categories, for example to to process very large unstructured data sets for Business Intelligence applications. Driving Storage Costs Down for AWS Customers.

Cloud 118

Cloud AWS Automotive Latency 118

All Things Distributed

DECEMBER 5, 2010

Route 53 has the business properties that you have come to expect from an AWS service: fully self-service and programmable, with transparent pay-as-you-go pricing and no minimum usage commitments. Modern systems require much faster update propagation to for example deal with outages. Some fundamentals on Naming. No lock-in.

Cloud 117

Cloud Internet Internet AWS 117

Adrian Cockcroft

APRIL 18, 2018

This is true for CPU, network, memory and storage, and also for bare metal, virtual machines, containers, processes, functions and threads. I’ve seen problematic examples in the past. For each resource there’s usually an underlying or bare metal version, a virtualized resource and sometimes also a containerized resource.

Monitoring 40

Monitoring Systems Virtualization Metrics 40

The Morning Paper

MAY 28, 2019

We don’t talk about energy as often as we probably should on this blog, but it’s certainly true that our data centres and various IT systems consume an awful lot of it. For example, a value of ‘2’ can be represented by a signal that is low until time 2 and then high from then on. . ASPLOS’19. Introducing race logic.

Energy 52

Energy Hardware Efficiency Architecture 52

Adrian Cockcroft

NOVEMBER 11, 2019

Collecting some critical metrics at one second intervals, with a total observability latency of ten seconds or less matches the human attention span much better. Systems that have an identifiable capacity trend, for example are filling up disk space at a predictable rate, have a “time to live” (TTL) that can be calculated.

Latency 52

Latency Engineering Systems Hardware 52

The Netflix TechBlog

SEPTEMBER 24, 2021

As an example, cloud-based post-production editing and collaboration pipelines demand a complex set of functionalities, including the generation and hosting of high quality proxy content. The following table gives us an example of file sizes for 4K ProRes 422 HQ proxies.

Cloud 237

Cloud Media Storage Cache 237

Adrian Cockcroft

NOVEMBER 11, 2019

Collecting some critical metrics at one second intervals, with a total observability latency of ten seconds or less matches the human attention span much better. Systems that have an identifiable capacity trend, for example are filling up disk space at a predictable rate, have a “time to live” (TTL) that can be calculated.

Latency 53

Latency Engineering Systems Hardware 53

The Netflix TechBlog

OCTOBER 27, 2020

The processed data is typically stored as data warehouse tables in AWS S3. For example, in order to enhance our user experience, one online application fetches subscribers’ preferences data to recommend movies and TV shows. The data warehouse is not designed to serve point requests from microservices with low latency.

Latency 243

Latency Storage Big Data Tuning 243

Brendan Gregg

SEPTEMBER 5, 2017

Here's some output from my zfsdist tool, in bcc/BPF, which measures ZFS latency as a histogram on Linux: # zfsdist. Tracing ZFS operation latency. For example, although it's very minor, the /dev/*random devices were faster on Solaris last time I checked, as Linux was spending more effort on entropy for improving security. (Or,

Virtualization 76

Virtualization AWS Engineering Hardware 76

The Netflix TechBlog

NOVEMBER 3, 2022

When a new leader is elected it loads all data from external storage. For example, it is OK to send writes through one instance, and do reads from another one with full data read consistency guarantees. For example, a batch workflow orchestration system may create multiple jobs which are part of a single workflow execution.

Cache 224

Cache Latency Traffic Systems 224

Dynatrace

AUGUST 5, 2019

In this role, I am leading a global team that works closely with our strategic partners such as AWS, Microsoft, Google, Pivotal, Red Hat and others. Remember: This is a critical aspect as you do not want to migrate a service and suddenly introduce high latency or costs to a system that you forgot about having a dependency with!

Cloud 31

Cloud Traffic Database Network 31

The Netflix TechBlog

APRIL 26, 2022

An example of data about members is the video they had watched or added to their My List. An example of video data is video metadata, like the length of a video. The first version of our logger library optimized for storage by deduplicating facts and optimized for network i/o using different compression methods for each fact.

Storage 187

Storage Design Scalability Latency 187

The Netflix TechBlog

FEBRUARY 9, 2021

Server-generated assets, since client-side generation would require the retrieval of many individual images, which would increase latency and time-to-render. To reduce latency, assets should be generated in an offline fashion and not in real time. This requires an asset storage solution. We call these adaptive fields.

Engineering 165

Engineering Storage Latency Entertainment 165

All Things Distributed

MARCH 11, 2016

The epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Given that AWS is a pioneer in building and operating these services world-wide, these lessons have been of crucial importance to our business. AWS helps its customers do this too.

AWS 144

AWS Hardware Retail Virtualization 144

All Things Distributed

APRIL 9, 2015

Details on the AWS Blog. AWS has been offering a range of storage solutions: objects, block storage, databases, archiving, etc. When we designed Amazon EFS we decided to build along the AWS principles: Elastic, scalable, highly available, consistent performance, secure, and cost-effective. Details on the AWS Blog.

Lambda 122

Lambda Cloud IoT AWS 122

Smashing Magazine

JANUARY 7, 2019

One way or another, rather than focusing on full page loading time (via onLoad and DOMContentLoaded timings, for example), prioritize page loading as perceived by your customers. Estimated Input Latency tells us if we are hitting that threshold, and ideally, it should be below 50ms.

Performance 103

Performance Cache Metrics Network 103

Smashing Magazine

JANUARY 11, 2021

For example, to make performance more tangible, you could expose the revenue performance impact by showing the correlation between the conversion rate and time to application load, as well as rendering performance. Run performance experiments and measure outcomes — both on mobile and on desktop (for example, with Google Analytics ).

Performance 145

Performance Cache Media Metrics 145

Smashing Magazine

JANUARY 6, 2020

Run performance experiments and measure outcomes — both on mobile and on desktop (for example, with Google Analytics ). Rather than focusing on full page loading time (via onLoad and DOMContentLoaded timings, for example), prioritize page loading as perceived by your customers. What does it mean?

Performance 145

Performance Cache Network Metrics 145

High Scalability

JULY 20, 2018

And if you know anyone looking for a simple book that uses lots of pictures and lots of examples to explain the cloud, then please recommend my new book: Explain the Cloud Like I'm 10. Do you like this sort of Stuff? Please lend me your support on Patreon. It would mean a great deal to me. They'll love you even more.

Internet 121

Internet Internet Scalability Automotive 121

All Things Distributed

JUNE 21, 2018

In fact, this is been proven by our customers as Amazon Aurora remains the fastest growing service in AWS history. Airbnb is a great example of a customer building high-performance and scalable applications with Amazon Aurora. The purpose of DynamoDB is to provide consistent single-digit millisecond latency for any scale of workloads.

Database 167

Database AWS Games Latency 167