All Things Distributed

SEPTEMBER 3, 2013

Today, we added two important choices for customers running high performance apps in the cloud: support for Redis in Amazon ElastiCache and a new high memory database instance (db.cr1.8xlarge) for Amazon RDS. No single database architecture or solution can meet all of Amazon.com’s or our customers’ needs.

Cache 131

Cache Cloud Performance Retail 131

Scalegrid

FEBRUARY 8, 2024

At its core, a distributed storage system comprises three main components: a controller for managing the system’s operations, an internal datastore where information is held, and databases geared towards ensuring scalability, partitioning capabilities, and high availability for all types of data.

Storage 130

Storage Systems Big Data Azure 130

Percona

MAY 2, 2023

That’s why it’s essential to implement the best practices and strategies for MongoDB database backups. Why are MongoDB database backups important? Regular database backups are essential to protect against data loss caused by system failures, human errors, natural disasters, or cyber-attacks.

Best Practices 52

Best Practices Database Storage Servers 52

All Things Distributed

JUNE 20, 2017

This enables customers to serve content to their end users with low latency, giving them the best application experience. In 2008, AWS opened a point of presence (PoP) in Hong Kong to enable customers to serve content to their end users with low latency. Since then, AWS has added two more PoPs in Hong Kong, the latest in 2016.

AWS 149

AWS Logistics Cloud Social Media 149

Dynatrace

NOVEMBER 3, 2021

For example, when monitoring a database, you’ll want to know about any latency when writing data to a disk or average query response time. Experienced database administrators learn to spot patterns that can lead to common problems. An application may depend on multiple services, each with its own set of metrics and logs.

Monitoring 226

Monitoring Metrics DevOps Scalability 226

Percona

MARCH 23, 2023

Hardware Memory The amount of RAM to be provisioned for database servers can vary greatly depending on the size of the database and the specific requirements of the company. By caching hot datasets, indexes, and ongoing changes, InnoDB can provide faster response times and utilize disk IO in a much more optimal way.

Performance 123

Performance Hardware Tuning Storage 123

The Morning Paper

AUGUST 29, 2019

Choosing a cloud DBMS: architectures and tradeoffs Tan et al., If you’re moving an OLAP workload to the cloud (AWS in the context of this paper), what DBMS setup should you go with? It is advantageous in the cloud to shut down compute resources when they are not being used, but there is then a query latency cost.

Architecture 76

Architecture Cloud Storage Serverless 76

All Things Distributed

JANUARY 18, 2012

a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Today is a very exciting day as we release Amazon DynamoDB , a fast, highly reliable and cost-effective NoSQL database service designed for internet scale applications. Amazon DynamoDB offers low, predictable latencies at any scale. Comments ().

Scalability 167

Scalability Database Ecommerce Latency 167

The Morning Paper

FEBRUARY 6, 2020

Last week we looked at a function shipping solution to the problem; Cloudburst uses the more common data shipping to bring data to caches next to function runtimes (though you could also make a case that the scheduling algorithm placing function execution in locations where the data is cached a flavour of function-shipping too).

Lambda 98

Lambda Serverless Cache Latency 98

All Things Distributed

JULY 14, 2015

I am excited to share with you that today we are expanding DynamoDB with streams, cross-region replication, and database triggers. Streams provide you with the underlying infrastructure to create new applications, such as continuously updated free-text search indexes, caches, or other creative extensions requiring up-to-date table changes.

Database 167

Database Lambda AWS IoT 167

O'Reilly Software

JUNE 12, 2018

The advent of the cloud was a huge boon to the work we were doing, as the improved automation made us even more productive. Back then, the most common pattern I saw for service-based systems was sharing a database among multiple services. But I kept hitting up against some fundamental issues. And data was at the heart of the problem.

Database 72

Database Cache Architecture Latency 72

All Things Distributed

JUNE 21, 2017

Today, I'm excited to announce the general availability of Amazon DynamoDB Accelerator (DAX) , a fully managed, highly available, in-memory cache that can speed up DynamoDB response times from milliseconds to microseconds, even at millions of requests per second. Adding caching when your app is already experiencing load is not easy.

Speed 123

Speed Cache Latency AWS 123

All Things Distributed

NOVEMBER 26, 2013

and enable you to maintain a nearly up-to-date copy of your master database in a different AWS Region. Availability Zones have since become the foundational elements for AWS customers to create a new generation of highly available distributed applications in the cloud that are designed to be fault tolerant from the get go.

Cloud 147

Cloud AWS Traffic Latency 147

Percona

MARCH 20, 2023

When deciding what to pick, there are many things to consider, like where the proxy needs to be, if it “just” needs to redirect the connections, or if more features need to be in, like caching and filtering, or if it needs to be integrated with some MySQL embedded automation. Given that, there never was a single straight answer.

Games 118

Games Latency Traffic Cache 118

The Morning Paper

JUNE 23, 2019

Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. The network latency of fetching data over the network, even considering fast data center networks. Oh, you mean a cache?

Cache 79

Cache Latency Google Lambda 79

All Things Distributed

OCTOBER 12, 2016

Fast Data is an emerging industry term for information that is arriving at high volume and incredible rates, faster than traditional databases can manage. Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis , a fully managed in-memory data store that operates at sub-millisecond latency.

Scalability 131

Scalability Analytics Cache AWS 131

Percona

JULY 17, 2023

MongoDB Community Edition software might set the stage for achieving your high-volume database goals, but you quickly learn that its features fall short of your enterprise needs. The popularity of MongoDB MongoDB has emerged as a popular database platform. 5 among all database management systems and No. It ranks No.

Open Source 74

Open Source Database Scalability Software 74

All Things Distributed

DECEMBER 5, 2010

Expanding the Cloud with DNS - Introducing Amazon Route 53. There are two main types of DNS servers: authoritative servers and caching resolvers. But the real robustness of the DNS system comes through the way lookups are handled, which is what caching resolvers do. All Things Distributed. Comments ().

Cloud 117

Cloud Internet Internet AWS 117

The Morning Paper

MAY 14, 2019

Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., Seer is an online system that observes the behaviour of cloud applications (using the DeathStarBench microservices for the evaluation) and predicts when QoS violations may be about to occur. ASPLOS’19.

Big Data 45

Big Data Cloud Performance Hardware 45

All Things Distributed

OCTOBER 2, 2017

Today marks the 10 year anniversary of Amazon's Dynamo whitepaper , a milestone that made me reflect on how much innovation has occurred in the area of databases over the last decade and a good reminder on why taking a customer obsessed approach to solving hard problems can have lasting impact beyond your original expectations.

Internet 130

Internet Internet AWS Performance 130

The Morning Paper

SEPTEMBER 10, 2019

That’s hard for many reasons, including the differing trade-offs between throughput and latency that need to be made across the use cases. to understand YouTube video performance) drive tens of thousands of canned (known in advance) queries per second, that need to be served with latency in the tens of milliseconds.

Analytics 75

Analytics Latency Cache Google 75

Percona

MAY 15, 2023

The main reason behind this is that MySQL is a relational database system (RDBMS), and any data that is going to be written in it must respect the RDBMS rules. In shard-nothing, each shard can live in a totally separate logical schema instance / physical database server/data center/continent. The POC Why this POC?

Traffic 121

Traffic Scalability Database Servers 121

n0derunner

SEPTEMBER 17, 2018

It is very common to have large resource-hungry databases separated across nodes using anti-affinity rules. In such a case we have a Bandwidth heavy workload profile (reporting) sharing with a Latency Sensitive workload (transactional). The key thing to observe is the impact on the latency sensitive (OLTP) workload.

Performance Testing 40

Performance Testing Benchmarking Testing Performance 40

Brendan Gregg

JANUARY 17, 2018

Here is a MySQL database server (mysqld), PID 423, and measuring its working set size for 0.1 Short durations can be useful for understanding how well a WSS will fit into the CPU caches (L1/L2/L3, TLB L1/L2, etc). I just wrote [wss.pl] as a front-end to this feature. seconds (100 milliseconds): #./wss.pl Est(s) RSS(MB) PSS(MB) Ref(MB).

Cache 71

Cache Latency C++ Programming 71

Abhishek Tiwari

DECEMBER 27, 2017

ETL is a product of the relational database era and it has not evolved much in last decade. With the arrival of new cloud-native tools and platform, ETL is becoming obsolete. In contrast, Alluxio a middleware for data access - think Alluxio storage layer as fast cache.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

Alex Russell

MARCH 6, 2021

A then-representative $200USD device had 4-8 slow (in-order, low-cache) cores, ~2GiB of RAM, and relatively slow MLC NAND flash storage. A silver lining on this dark cloud is that mobile JavaScript payload growth paused in 2020. The Moto G4 , for example. Don't pay a lot for an Android-shaped muffler.

Performance 124

Performance Network Cache Metrics 124

Percona

AUGUST 10, 2018

The performance of a PostgreSQL database can be compromised by dead tuples, since they continue to occupy space and can lead to bloat. Now, though, it’s time to look at autovacuum for postgres, and the internals you to know to maintain a high performance PostgreSQL database needed by demanding applications. Is this a problem ?

Tuning 44

Tuning Cache Database Storage 44

Smashing Magazine

JANUARY 6, 2020

Estimated Input Latency tells us if we are hitting that threshold, and ideally, it should be below 50ms. Designed for the modern web, it responds to actual congestion, rather than packet loss like TCP does, it is significantly faster , with higher throughput and lower latency — and the algorithm works differently.

Performance 145

Performance Cache Network Metrics 145

Brendan Gregg

JANUARY 16, 2018

Here is a MySQL database server (mysqld), PID 423, and measuring its working set size for 0.1 Short durations can be useful for understanding how well a WSS will fit into the CPU caches (L1/L2/L3, TLB L1/L2, etc). I just wrote [wss.pl] as a front-end to this feature. seconds (100 milliseconds): #./wss.pl Est(s) RSS(MB) PSS(MB) Ref(MB).

Cache 40

Cache Latency C++ Programming 40

Smashing Magazine

JANUARY 7, 2019

Estimated Input Latency tells us if we are hitting that threshold, and ideally, it should be below 50ms. PRPL stands for Pushing critical resource, Rendering initial route, Pre-caching remaining routes and Lazy-loading remaining routes on demand. An application shell is the minimal HTML, CSS, and JavaScript powering a user interface.

Performance 103

Performance Cache Metrics Network 103

The Morning Paper

SEPTEMBER 12, 2019

Declarative recursive computation on an RDBMS… or, why you should use a database for distributed machine learing Jankov et al., Machine learning inside a relational database has been done before, most notably in the form of MADlib , which was integrated into Greenplum during my time at Pivotal. VLDB’19.

Network 44

Network Database Programming Hardware 44

The Netflix TechBlog

DECEMBER 17, 2019

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events. Designed with High Availability in mind.

Database 212

Database Traffic Transportation Open Source 212

The Netflix TechBlog

DECEMBER 17, 2019

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events. Designed with High Availability in mind.

Database 197

Database Traffic Transportation Open Source 197

The Netflix TechBlog

MARCH 6, 2019

Mounting object storage in Netflix’s media processing platform By Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. Disk Caching? — ? Regional caching? —?Netflix Replays? —?More

Media 214

Media Storage Processing Cache 214