The Netflix TechBlog

APRIL 27, 2023

Atlas is an in-memory time-series database that ingests multiple billions of time-series per day and retains the last two weeks of data. To serve the increasing number of push down queries, the in-memory storage layer would need to scale up as well, and it became clear that this would push the already expensive storage costs far higher.

Storage 288

Storage Cache Metrics Database 288

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. This paper describes the design decisions behind the Snowflake cloud-based data warehouse. An increasingly large fraction of data in modern workloads comes from less predictable and highly variable sources. joins) during query processing.

Storage 112

Storage Engineering Cache Serverless 112

Percona

JUNE 22, 2023

To make data count and to ensure cloud computing is unabated, companies and organizations must have highly available databases. This guide provides an overview of what high availability means, the components involved, how to measure high availability, and how to achieve it. More in the following sub-section.)

Availability 105

Availability Database Open Source Hardware 105

Percona

FEBRUARY 1, 2023

PostgreSQL is a fantastic database, but if you’re storing images, video, audio files, or other large data objects, you need to “toast” them to get optimal performance. This post will look at using The Oversized-Attribute Storage Technique (TOAST) to improve performance and scalability.

Storage 87

Storage Scalability Strategy Performance 87

ScaleOut Software

APRIL 22, 2020

In this time of extremely high online usage, web sites and services have quickly become overloaded, clogged trying to manage high volumes of fast-changing data. Maintaining rapidly changing data in back-end databases creates bottlenecks that impact responsiveness. The Solution: Distributed Caching.

Cache 52

Cache Storage Servers Database 52

ScaleOut Software

APRIL 22, 2020

In this time of extremely high online usage, web sites and services have quickly become overloaded, clogged trying to manage high volumes of fast-changing data. Maintaining rapidly changing data in back-end databases creates bottlenecks that impact responsiveness. The Solution: Distributed Caching.

Cache 52

Cache Storage Servers Database 52

The Netflix TechBlog

OCTOBER 15, 2019

Part I: Overview Andreas Andreakis , Falguni Jhaveri , Ioannis Papapanagiotou , Mark Cho , Poorna Reddy , Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), caching (Memcached etc.),

Transportation 192

Transportation Architecture Processing Storage 192

Scalegrid

JANUARY 25, 2024

Effective management of memory stores with policies like LRU/LFU proactive monitoring of the replication process and advanced metrics such as cache hit ratio and persistence indicators are crucial for ensuring data integrity and optimizing Redis’s performance. All these contribute significantly towards ensuring smooth functioning.

Metrics 130

Metrics Monitoring Latency Cache 130

Scalegrid

DECEMBER 21, 2023

In today’s data-driven world, the ability to effectively monitor and manage data is of paramount importance. Redis®, a powerful in-memory data store, is no exception. To monitor Redis® instances effectively, collect Redis metrics focusing on cache hit ratio, memory allocated, and latency threshold.

Strategy 130

Strategy Monitoring Latency DevOps 130

Dynatrace

JANUARY 16, 2023

The study analyzes factual Kubernetes production data from thousands of organizations worldwide that are using the Dynatrace Software Intelligence Platform to keep their Kubernetes clusters secure, healthy, and high performing. Big data : To store, search, and analyze large datasets, 32% of organizations use Elasticsearch.

Open Source 347

Open Source Java Operating System Programming 347

Dynatrace

OCTOBER 7, 2021

This means you no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems. Speed is next; serverless solutions are quick to spin up or down as needed, and there are no delays due to limited storage or resource access. Data Store. Improving data processing.

Serverless 286

Serverless AWS Lambda Storage 286

Percona

APRIL 24, 2023

Effectively, the memory available for pages of the table gets less. The more indexes, the more the requirement of memory for effective caching. If we don’t increase the available memory, this starts hurting the entire performance of the system. Less-effective cache results in more datafile read, so read I/O is increased.

Tuning 124

Tuning Cache Storage Database 124

Percona

FEBRUARY 15, 2023

With MyRocks/RocksDB, data is stored in a set of large SST files. For good performance, the filter blocks are cached in the RocksDB block cache and normally stay there since they are accessed frequently. LSM storage engines like MyRocks are very different from the more common B-Tree-based storage engines like InnoDB.

Storage 125

Storage Tuning Cache Engineering 125

Percona

MARCH 28, 2023

It’s to ensure that our data is durable after each write operation and to make it persistent and consistent without compromising the performance. In terms of MongoDB, it achieves WAL and data durability using a combination of both Journaling and Checkpoints. Starting with the basics, why is WAL needed in the first place?

Hardware 85

Hardware C++ Storage Cache 85

Percona

APRIL 17, 2023

We’ll also go over some best practices for MongoDB security as well as MongoDB data modeling. Note that the intent of tuning the settings is not exclusively about improving performance but also enhancing the high availability and resilience of the MongoDB database. Spoiler alert: This post focuses on MongoDB 3.6.X

Best Practices 52

Best Practices Design Tuning Database 52

Percona

APRIL 7, 2023

I’ve used a fourth instance to host a PMM server to monitor servers A and B and used the data collected by the PMM agents installed on the database servers to compare performance. MySQL comes pre-configured to be conservative instead of making the most of the resources available in the server. Why is that?

Tuning 134

Tuning Cache Servers Benchmarking 134

Percona

MARCH 23, 2023

By caching hot datasets, indexes, and ongoing changes, InnoDB can provide faster response times and utilize disk IO in a much more optimal way. Storage The type of storage and disk used for database servers can have a significant impact on performance and reliability. Typically a good value is 70%-80% of available memory.

Performance 123

Performance Hardware Tuning Storage 123

Dynatrace

SEPTEMBER 13, 2023

These workflows also utilize Davis® , the Dynatrace causal AI engine, and all your observability and security data across all platforms, in context, at scale, and in real-time. Storing frequently accessed data in faster storage, usually in-memory caching, improves data retrieval speed and overall system performance. Beyond

AWS 252

AWS Efficiency Azure Cloud 252

All Things Distributed

OCTOBER 12, 2016

Fast Data is an emerging industry term for information that is arriving at high volume and incredible rates, faster than traditional databases can manage. Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis , a fully managed in-memory data store that operates at sub-millisecond latency.

Scalability 130

Scalability Analytics Cache AWS 130

Percona

JUNE 22, 2023

Database uptime and availability Monitoring database uptime and availability is crucial as it directly impacts the availability of critical data and the performance of applications or websites that rely on the MySQL database. The point we’re making here is they are critical for MySQL Performance.

Performance 102

Performance Monitoring Traffic Database 102

The Netflix TechBlog

MAY 4, 2023

It can happen on an edge API system servicing customer devices, between the edge and mid-tier services, or from mid-tiers to data stores. It provides a good read on the availability and latency ranges under different production conditions. For instance, envision a response payload that delivers media streams for a playback session.

Traffic 339

Traffic Latency Tuning Systems 339

The Morning Paper

SEPTEMBER 10, 2019

Procella: unifying serving and analytical data at YouTube Chattopadhyay et al., Anchored in the primary use case of supporting Google’s YouTube business, what we’re looking at here could well be the future of data processing at Google. Because they had too many data processing systems! ;). VLDB’19. are divided.

Analytics 75

Analytics Latency Cache Google 75

Percona

DECEMBER 23, 2022

I wrote this post on MyRocks because I believe it is the most interesting new MySQL storage engine to have appeared over the last few years. I decided to use a virtual machine with two CPU cores, four GB of memory, and storage limited to a maximum of 1000 IOPs of 16KB. I emulated these limits using the KVM iotune settings in my lab. <iotune>

Storage 55

Storage Cache C++ Virtualization 55

Percona

JULY 25, 2023

When a new node joins the cluster, SST is performed to receive the data from DONOR to the JOINER. JOINER uses PXB to stream the data directory from DONOR. Percona XtraBackup copies the InnoDB data files. Percona XtraBackup copies the InnoDB data files. ibd2sdi data/test/t1.ibd CREATE TABLE test.t1(a ibd > t1.sdi

Cache 84

Cache Benchmarking Servers Design 84

Dynatrace

DECEMBER 20, 2021

Replay tools log data from users’ interactions with your website or application, including the pages they looked at, how long they stayed, and what they clicked on while they were there. These tools pull this data to recreate anonymized versions of user sessions that your team can watch back on demand. Improved analytic context.

Mobile 251

Mobile Website Analytics Cache 251

Percona

SEPTEMBER 1, 2023

This results in expedited query execution, reduced resource utilization, and more efficient exploitation of the available hardware resources. Scalability As your data volume and user base expand, a finely tuned database can seamlessly accommodate increased workloads without compromising performance.

Tuning 52

Tuning Database Performance Hardware 52

Smashing Magazine

MAY 25, 2022

The code below illustrates how to use WeakSet() and some of the methods available: const human = new WeakSet(); let paul = {name: "Paul"}; let mary = {gender: "Mary"}; // Add the human with the name paul to the classroom. WeakMap can be used in two areas of web development: caching and additional data storage.

Cache 117

Cache Programming Storage Code 117

Percona

APRIL 3, 2023

Below are some of the upgrade options available to upgrade to PG11 or above. We are all aware that PostgreSQL 10 has reached end of support , and there is a need to upgrade to the latest version or any application-compatible PG version.

C++ 99

C++ Cache Database Storage 99

ScaleOut Software

MAY 15, 2014

We have spent a great deal of time at ScaleOut Software re-architecting our in-memory data grid (IMDG)’s code base to make best use of many cores and large memory. For example, the IMDG must be able to efficiently create millions of objects in each server to make use of its huge storage capacity. Please retry later.

Cache 40

Cache Servers Network Design 40

ScaleOut Software

JUNE 22, 2020

From Distributed Caches to Real-Time Digital Twins. The pace of these changes has made it challenging for server-based infrastructures to manage fast-growing populations of users and data sources while maintaining fast response times.

Ecommerce 52

Ecommerce Cache Scalability IoT 52

ScaleOut Software

JUNE 22, 2020

From Distributed Caches to Real-Time Digital Twins. The pace of these changes has made it challenging for server-based infrastructures to manage fast-growing populations of users and data sources while maintaining fast response times.

Ecommerce 52

Ecommerce Cache Scalability IoT 52

Percona

MARCH 17, 2023

PMM2 uses VictoriaMetrics (VM) as its metrics storage engine. VictoriaMetrics maintains an in-memory cache for mapping active time series into internal series IDs. 25 Bytes, or you should plan storage roughly between 500 GB – one TB for a default 30 day retention period.

Monitoring 96

Monitoring Scalability Cache Database 96

The Morning Paper

NOVEMBER 19, 2019

Local-first software: you own your data, in spite of the cloud Kleppmann et al., The primary issue with cloud apps (the SaaS model) is ownership of the data. Although they let you access your data anywhere, all data access must go via the server, and you can only do the things that the server will let you do.

Cloud 83

Cloud Software Software Internet 83

The Morning Paper

AUGUST 29, 2019

There’s a broad set of choices including where you store the data, whether you run your own DBMS nodes or use a service, the kinds of instance types to go for if you do run your own, and so forth. We focused on OLAP-oriented parallel data warehouse products available for AWS and restricted our attention to commercially available systems.

Architecture 76

Architecture Cloud Storage Serverless 76

ScaleOut Software

JULY 27, 2018

For more than a decade, in-memory data grids (IMDGs) have proven their usefulness for storing fast-changing data in enterprise applications. Whether it’s ecommerce shopping carts, financial trading data, IoT telemetry, or airline reservations, these data sets need fast, reliable access for large, mission-critical workloads.

Scalability 40

Scalability Cache Airlines Healthcare 40

ScaleOut Software

JULY 27, 2018

For more than a decade, in-memory data grids (IMDGs) have proven their usefulness for storing fast-changing data in enterprise applications. Whether it’s ecommerce shopping carts, financial trading data, IoT telemetry, or airline reservations, these data sets need fast, reliable access for large, mission-critical workloads.

Scalability 40

Scalability Cache Airlines Healthcare 40

All Things Distributed

JUNE 21, 2017

Today, I'm excited to announce the general availability of Amazon DynamoDB Accelerator (DAX) , a fully managed, highly available, in-memory cache that can speed up DynamoDB response times from milliseconds to microseconds, even at millions of requests per second. DynamoDB was the first service at AWS to use SSD storage.

Speed 121

Speed Cache Latency AWS 121