The Netflix TechBlog

APRIL 27, 2023

Atlas is an in-memory time-series database that ingests multiple billions of time-series per day and retains the last two weeks of data. To serve the increasing number of push down queries, the in-memory storage layer would need to scale up as well, and it became clear that this would push the already expensive storage costs far higher.

Storage 288

Storage Cache Metrics Database 288

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. This paper describes the design decisions behind the Snowflake cloud-based data warehouse. An increasingly large fraction of data in modern workloads comes from less predictable and highly variable sources. joins) during query processing.

Storage 112

Storage Engineering Cache Serverless 112

Smashing Magazine

SEPTEMBER 28, 2022

Five Data-Loading Patterns To Improve Frontend Performance. Five Data-Loading Patterns To Improve Frontend Performance. Data loading patterns are an essential part of your application as they will determine which parts of your application are directly usable by visitors. But isn’t waiting for the data the point?

Cache 125

Cache Performance Servers Social Media 125

High Scalability

JANUARY 11, 2022

from a client it performs two parallel operations: i) persisting the action in the data store ii) publish the action in a streaming data store for a pub-sub model. User Feed Service, Media Counter Service) read the actions from the streaming data store and performs their specific tasks. Data Models. Graph Data Models.

Design 334

Design Media Storage Logistics 334

Percona

FEBRUARY 1, 2023

PostgreSQL is a fantastic database, but if you’re storing images, video, audio files, or other large data objects, you need to “toast” them to get optimal performance. This post will look at using The Oversized-Attribute Storage Technique (TOAST) to improve performance and scalability.

Storage 89

Storage Scalability Strategy Performance 89

ScaleOut Software

APRIL 22, 2020

In this time of extremely high online usage, web sites and services have quickly become overloaded, clogged trying to manage high volumes of fast-changing data. Maintaining rapidly changing data in back-end databases creates bottlenecks that impact responsiveness. The Solution: Distributed Caching.

Cache 52

Cache Storage Servers Database 52

ScaleOut Software

APRIL 22, 2020

In this time of extremely high online usage, web sites and services have quickly become overloaded, clogged trying to manage high volumes of fast-changing data. Maintaining rapidly changing data in back-end databases creates bottlenecks that impact responsiveness. The Solution: Distributed Caching.

Cache 52

Cache Storage Servers Database 52

Scalegrid

JANUARY 25, 2024

Effective management of memory stores with policies like LRU/LFU proactive monitoring of the replication process and advanced metrics such as cache hit ratio and persistence indicators are crucial for ensuring data integrity and optimizing Redis’s performance. All these contribute significantly towards ensuring smooth functioning.

Metrics 130

Metrics Monitoring Latency Cache 130

The Netflix TechBlog

OCTOBER 15, 2019

Part I: Overview Andreas Andreakis , Falguni Jhaveri , Ioannis Papapanagiotou , Mark Cho , Poorna Reddy , Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), caching (Memcached etc.),

Transportation 192

Transportation Architecture Processing Storage 192

Dynatrace

JANUARY 16, 2023

The study analyzes factual Kubernetes production data from thousands of organizations worldwide that are using the Dynatrace Software Intelligence Platform to keep their Kubernetes clusters secure, healthy, and high performing. Big data : To store, search, and analyze large datasets, 32% of organizations use Elasticsearch.

Open Source 344

Open Source Java Operating System Programming 344

Dynatrace

AUGUST 1, 2022

Kubernetes was initially designed with a strong focus on stateless workloads, meaning these workloads do not need to store any persistent data. Interestingly, our partner RedHat reported in 2021 that around 80% of deployed workloads are databases or data caches, storing data in persistent volume claims (PVCs).

Storage 189

Storage Database Network Tuning 189

Dynatrace

OCTOBER 7, 2021

This means you no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems. Speed is next; serverless solutions are quick to spin up or down as needed, and there are no delays due to limited storage or resource access. Data Store. Improving data processing.

Serverless 282

Serverless AWS Lambda Storage 282

Scalegrid

DECEMBER 21, 2023

In today’s data-driven world, the ability to effectively monitor and manage data is of paramount importance. Redis®, a powerful in-memory data store, is no exception. To monitor Redis® instances effectively, collect Redis metrics focusing on cache hit ratio, memory allocated, and latency threshold.

Strategy 130

Strategy Monitoring Latency DevOps 130

Dynatrace

DECEMBER 13, 2023

This includes how quickly the application loads, how much load it is putting on the device, how much storage is being used, and how frequently it crashes. Here are some ways observability data is important to mobile app performance monitoring. Issue remediation. Proactive monitoring. Performance optimization.

Best Practices 199

Best Practices Mobile Metrics Performance 199

Percona

APRIL 24, 2023

The more indexes, the more the requirement of memory for effective caching. Indexes need more cache than tables Due to random writes and reads, indexes need more pages to be in the cache. Cache requirements for indexes are generally much higher than associated tables.

Tuning 125

Tuning Cache Storage Database 125

Percona

FEBRUARY 15, 2023

With MyRocks/RocksDB, data is stored in a set of large SST files. For good performance, the filter blocks are cached in the RocksDB block cache and normally stay there since they are accessed frequently. LSM storage engines like MyRocks are very different from the more common B-Tree-based storage engines like InnoDB.

Storage 126

Storage Tuning Cache Engineering 126

Percona

APRIL 17, 2023

We’ll also go over some best practices for MongoDB security as well as MongoDB data modeling. You can set this by adding the following lines to the /etc/sysctl.conf : vm.dirty_ratio = 15 vm.dirty_background_ratio = 5 Filesystems mount options MongoDB recommends the XFS filesystem for on-disk database data.

Best Practices 52

Best Practices Design Tuning Database 52

n0derunner

JULY 23, 2019

Effect of removing CPU constraints and maintaining data locality on a running DB instance. As the DB continues to run on the new host – the Nutanix storage detects the access patterns and “localizes” the data that the DB is accessing. The post Impact of Data locality on DB workloads.

Benchmarking 76

Benchmarking Storage Tuning Cache 76

The Morning Paper

NOVEMBER 5, 2019

File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution Aghayev et al., In this case, the assumption that a distributed storage backend should clearly be layered on top of a local file system. What is a distributed storage backend? SOSP’19. This is not surprising in hindsight.

Storage 64

Storage Systems Hardware Efficiency 64

Percona

MARCH 28, 2023

It’s to ensure that our data is durable after each write operation and to make it persistent and consistent without compromising the performance. In terms of MongoDB, it achieves WAL and data durability using a combination of both Journaling and Checkpoints. Starting with the basics, why is WAL needed in the first place?

Hardware 87

Hardware C++ Storage Cache 87

The Morning Paper

SEPTEMBER 10, 2019

Procella: unifying serving and analytical data at YouTube Chattopadhyay et al., Anchored in the primary use case of supporting Google’s YouTube business, what we’re looking at here could well be the future of data processing at Google. Because they had too many data processing systems! ;). VLDB’19. are divided.

Analytics 75

Analytics Latency Cache Google 75

Percona

APRIL 7, 2023

I’ve used a fourth instance to host a PMM server to monitor servers A and B and used the data collected by the PMM agents installed on the database servers to compare performance. In MySQL, considering the standard storage engine, InnoDB , the data cache is called Buffer Pool.

Tuning 134

Tuning Cache Servers Benchmarking 134

Percona

MARCH 23, 2023

By caching hot datasets, indexes, and ongoing changes, InnoDB can provide faster response times and utilize disk IO in a much more optimal way. Storage The type of storage and disk used for database servers can have a significant impact on performance and reliability. Setting oom_score_adj to -800.

Performance 124

Performance Hardware Tuning Storage 124

Percona

DECEMBER 23, 2022

I wrote this post on MyRocks because I believe it is the most interesting new MySQL storage engine to have appeared over the last few years. I decided to use a virtual machine with two CPU cores, four GB of memory, and storage limited to a maximum of 1000 IOPs of 16KB. I emulated these limits using the KVM iotune settings in my lab. <iotune>

Storage 56

Storage Cache C++ Virtualization 56

Percona

JULY 25, 2023

When a new node joins the cluster, SST is performed to receive the data from DONOR to the JOINER. JOINER uses PXB to stream the data directory from DONOR. Percona XtraBackup copies the InnoDB data files. Percona XtraBackup copies the InnoDB data files. ibd2sdi data/test/t1.ibd ibd2sdi data/test/t1.ibd

Cache 84

Cache Benchmarking Servers Design 84

All Things Distributed

SEPTEMBER 20, 2013

Traditionally records in a database were stored as such: the data in a row was stored together for easy and fast retrieval. Combined with the rise of data warehouse workloads, where there is often significant redundancy in the values stored in columns, and database models based on column oriented storage took off.

Storage 90

Storage Hardware Cache C++ 90

Dynatrace

DECEMBER 20, 2021

Replay tools log data from users’ interactions with your website or application, including the pages they looked at, how long they stayed, and what they clicked on while they were there. These tools pull this data to recreate anonymized versions of user sessions that your team can watch back on demand. Improved analytic context.

Mobile 248

Mobile Website Analytics Cache 248

Percona

JUNE 22, 2023

Database uptime and availability Monitoring database uptime and availability is crucial as it directly impacts the availability of critical data and the performance of applications or websites that rely on the MySQL database. Disk space usage Monitor the disk space usage of MySQL data files, log files, and temporary files.

Performance 104

Performance Monitoring Traffic Database 104

Percona

APRIL 3, 2023

We have PostgreSQL 10 running with the below list of extensions and some tables with OID columns: List of installed extensions: Name | Version | Schema | Description --+ + + pg_buffercache | 1.3 | public | examine the shared buffer cache pg_stat_statements | 1.6 | public | track execution statistics of all SQL statements executed plpgsql | 1.0 | pg_catalog (..)

C++ 101

C++ Database Cache Storage 101

Smashing Magazine

MAY 25, 2022

Properties of a data structure are considered reachable while the data structure is in memory, and they are usually kept in memory. WeakMap can be used in two areas of web development: caching and additional data storage. With caching, a copy of the result from a request is saved locally. Additional Data.

Cache 116

Cache Programming Storage Code 116

Percona

SEPTEMBER 1, 2023

Scalability As your data volume and user base expand, a finely tuned database can seamlessly accommodate increased workloads without compromising performance. An unoptimized indexing strategy can impede data insertion and retrieval operations. This can significantly elevate user satisfaction and engagement.

Tuning 52

Tuning Database Performance Hardware 52

The Morning Paper

FEBRUARY 6, 2020

Last week we looked at a function shipping solution to the problem; Cloudburst uses the more common data shipping to bring data to caches next to function runtimes (though you could also make a case that the scheduling algorithm placing function execution in locations where the data is cached a flavour of function-shipping too).

Lambda 98

Lambda Serverless Cache Latency 98

Dynatrace

SEPTEMBER 13, 2023

These workflows also utilize Davis® , the Dynatrace causal AI engine, and all your observability and security data across all platforms, in context, at scale, and in real-time. Storing frequently accessed data in faster storage, usually in-memory caching, improves data retrieval speed and overall system performance. Beyond

AWS 248

AWS Efficiency Azure Cloud 248

ScaleOut Software

MAY 15, 2014

We have spent a great deal of time at ScaleOut Software re-architecting our in-memory data grid (IMDG)’s code base to make best use of many cores and large memory. For example, the IMDG must be able to efficiently create millions of objects in each server to make use of its huge storage capacity. Please retry later.

Cache 40

Cache Servers Network Design 40

Scalegrid

NOVEMBER 23, 2023

Configure the PostgreSQL hostname by editing configuration files and restarting the server, with secure storage of connection details to enhance security. or localhost would suffice when making connection attempts to establish access for data manipulation of the server’s database information via programs such as SQL Shell.

Database 130

Database Servers Storage Network 130

The Netflix TechBlog

MAY 4, 2023

It can happen on an edge API system servicing customer devices, between the edge and mid-tier services, or from mid-tiers to data stores. Given the scale of the data being generated using replay traffic, we record the responses from the two sides to a cost-effective cold storage facility using technology like Apache Iceberg.

Traffic 339

Traffic Latency Tuning Systems 339

HammerDB

APRIL 19, 2023

As database performance is heavily influenced by the performance of storage, network, memory, and processors, we must understand the upper limit of these key components. For storage, FIO is generally used. Storage: The system has a SATA drive for the operating system and one NVMe (Intel SSD D7-P5510 (3.84 Database: MySQL 8.0.31

Performance 54

Performance Benchmarking Cache Storage 54