article thumbnail

Building an elastic query engine on disaggregated storage

The Morning Paper

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. Snowflake is a data warehouse designed to overcome these limitations, and the fundamental mechanism by which it achieves this is the decoupling (disaggregation) of compute and storage. joins) during query processing. Disaggregation (or not).

Storage 112
article thumbnail

Log Analysis: How to Digest 15 Billion Logs Per Day and Keep Big Queries Within 1 Second

DZone

From an architectural perspective, the system should be able to undertake real-time analysis of various formats of logs, and of course, be scalable to support the huge and ever-enlarging data size.

Analytics 133
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

What is a Distributed Storage System

Scalegrid

A distributed storage system is foundational in today’s data-driven landscape, ensuring data spread over multiple servers is reliable, accessible, and manageable. Understanding distributed storage is imperative as data volumes and the need for robust storage solutions rise.

Storage 130
article thumbnail

Announcing the Availability of Percona Backup for MongoDB 2.2.0

Percona

PBM) introduced a GA version of incremental physical backups , which can greatly impact both the recovery time and the cost of backup (considering storage and data transfer cost). Of course, each release also includes bug fixes and refining some of the existing features. Fixes, bugs, improvements – all is here!

article thumbnail

Follower Clusters – 3 Major Use Cases for Syncing SQL & NoSQL Deployments

Scalegrid

Note: ScaleGrid implements follower clusters using storage snapshots. Not actual production of course – that will be flirting with disaster. And since the entire import is performed using storage snapshots, rather than a logical dump, the process is nearly instantaneous. Database Dev/Test Setup.

Analytics 185
article thumbnail

A MyRocks Use Case

Percona

I wrote this post on MyRocks because I believe it is the most interesting new MySQL storage engine to have appeared over the last few years. I decided to use a virtual machine with two CPU cores, four GB of memory, and storage limited to a maximum of 1000 IOPs of 16KB. I emulated these limits using the KVM iotune settings in my lab. <iotune>

Storage 55
article thumbnail

Help! I Am Out of Disk Space!

Percona

When using cloud storage like EBS or similar, it is normally easy(er) to extend volumes, which gives us the luxury to plan the space to allocate for data with a good grade of relaxation. To enable volume expansion, you need to delete the storage class and enable it again. The story The case was on AWS using EKS.

AWS 90