All Things Distributed

AUGUST 22, 2024

Marc Olson, a long-time Amazonian, discusses the evolution of EBS, highlighting hard-won lessons in queueing theory, the importance of comprehensive instrumentation, and the value of incrementalism versus radical changes. It's an insightful look at how one of AWS’s foundational services has evolved to meet the needs of our customers.

Storage 203

Storage AWS 203

Percona

FEBRUARY 10, 2023

Today along with their team, we will see how pvc-autoresizer can automate storage scaling for MongoDB clusters on Kubernetes. Our goal is to automate storage scaling when our disk reaches a certain threshold of use and simultaneously reduce the amount of alert noise related to that. kubectl annotate pvc --all resize.topolvm.io/storage_limit="100Gi"

Storage 116

Storage Blockchain AWS Cloud 116

DZone

AUGUST 6, 2021

On a small scale, this isn’t problematic but when dealing with large-scale deployments, organizations end up using lots of computing, storage, and human resources just to manage their indexes. As the database grows in size, so does the index management cost.

Storage 189

Storage Database Systems Efficiency 189

InfoQ

MAY 15, 2023

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud. By Facundo Agriel

Storage 113

Storage Systems Scalability Cloud 113

All Things Distributed

JULY 27, 2023

Three distinct perspectives on scale that come along with building and operating a storage system the size of S3.

Storage 147

Storage Systems 147

Percona Community

JANUARY 9, 2024

Automated storage scaling stands out as a critical challenge. In the world of Kubernetes, where everything evolves quickly. Members of the Data on Kubernetes Community have proposed a solution to address this issue for Kubernetes operators.

Storage 68

Storage 68

InfoQ

OCTOBER 14, 2023

times the storage per node than before” without a price change. Google recently announced various improvements to Cloud Spanner, its distributed, decoupled relational database service with a “50% increase in throughput and 2.5 By Steef-Jan Wiggers

Google 101

Google Storage Cloud Database 101

DZone

JANUARY 29, 2020

To resolve the problem it was suggested to find more suitable data storage. There are several limitations to store and fetch such data (all restrictions could be found in official documentation ). For some internal reasons well known Amazon S3 bucket was chosen for this purpose. The choice affected the project's unit test base.

Storage 124

Storage Efficiency Testing Database 124

n0derunner

SEPTEMBER 15, 2023

Background For the past few months engineers from Nutanix have been participating in the MLPerftm Storage benchmark which is designed to measure the storage performance required for ML training workloads. appeared first on n0derunner.

Benchmarking 59

Benchmarking Storage Servers Engineering 59

DZone

JUNE 4, 2019

NVMe Storage Use Cases. NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. There are several AI/ML focused use cases to highlight.

Storage 100

Storage Artificial Intelligence FinTech Performance 100

DZone

JUNE 3, 2019

For example, one well-respected vendor's standard solution is limited to 7.5TB of internal storage, and it can only scale to 30TB. Normally, GPU nodes don't have much room for SSDs, which limits the opportunity to train very deep neural networks that need more data.

Storage 100

Storage Performance Network Scalability 100

Percona

SEPTEMBER 6, 2024

Database novices often wonder how ‘things work behind the scenes’ when starting with PostgreSQL. Many things happen when creating a table and adding data that are not apparent. You might ask, ‘Where does the data go?’ ’ Luckily, it is easy to discover the details.

Storage 99

Storage Database 99

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. Snowflake is a data warehouse designed to overcome these limitations, and the fundamental mechanism by which it achieves this is the decoupling (disaggregation) of compute and storage. joins) during query processing. Disaggregation (or not).

Storage 112

Storage Engineering Cache Serverless 112

DZone

JUNE 28, 2024

At first, data tiering was a tactic used by storage systems to reduce data storage costs. This involved grouping data that was not accessed as often into more affordable, if less effective, storage array choices. Even though they are quite costly, SSDs and flash can be categorized as high-performance storage classes.

Azure 223

Azure Storage Performance Cloud 223

DZone

SEPTEMBER 3, 2024

However, the increasing sizes of both data volumes and distributed system clusters raise significant cost challenges for all-flash storage and vast operational challenges for kernel clients. It improves I/O throughput substantially through the distributed cache and uses cost-effective object storage for data storage.

Cache 144

Cache Performance Storage Architecture 144

SQL Shack

JULY 28, 2020

In this article, we will learn how to test our storage subsystems performance using Diskspd. The storage subsystem is one of the key performance factors for SQL Server because SQL Server storage engine stores database objects, tables, and indexes on the physical files.

Storage 82

Storage Servers Testing Engineering 82

n0derunner

JULY 1, 2022

I was recently asked to investigate why Nutanix storage was not as fast as a competing solution in a PoC environment. One thing that seemed really odd was that the working set size for the tests were in the order of … The post Beware of tiny working-set-sizes when testing storage performance. appeared first on n0derunner.

Storage 52

Storage Testing Performance Benchmarking 52

DZone

SEPTEMBER 25, 2023

It is the second of a series of articles that is built on top of that project, representing experiments with various statistical and machine learning models, data pipelines implemented using existing DAG tools, and storage services, both cloud-based and alternative on-premises solutions.

Storage 261

Storage AWS Architecture Cloud 261

Percona Community

OCTOBER 23, 2020

In this blog post we’re going to see those technologies at work to give us awesome block storage performance with flexibility and simple operations. It’s a new generation in storage software, designed for super high speed low latency NVMe devices. Why is SPDK exciting?

Storage 52

Storage Latency Speed Database 52

DZone

JULY 1, 2024

Disk Storage Space Evaluation The storage space requirements for a GBase cluster are calculated based on the data volume of the business system, the choice of compression algorithm, and the number of cluster replicas. The calculation method for the physical disk capacity of the GBase cluster is as follows:

Storage 144

Storage Systems 144

Scalegrid

SEPTEMBER 10, 2024

Customer-Requested Features We’re always listening to your feedback, so we added the ability to access additional storage without upgrading to a larger plan. This update allows customers to scale storage more flexibly, based on their current needs, without incurring unnecessary costs.

Latency 130

Latency AWS Storage Tuning 130

DZone

AUGUST 13, 2024

But this also caused storage challenges like disk failures and data recovery. As our data grew, we had problems with AWS Redshift which was slow and expensive. Changing to ClickHouse made our query performance faster and greatly cut our costs. To avoid extensive maintenance, we adopted JuiceFS, a distributed file system with high performance.

Artificial Intelligence 223

Artificial Intelligence Architecture AWS Storage 223

DZone

JUNE 24, 2024

Objectives Modern AI innovations require proper infrastructure, especially concerning data throughput and storage capabilities. While GPUs drive faster results, legacy storage solutions often lag behind, causing inefficient resource utilization and extended times in completing the project.

Systems 130

Systems Software Software Storage 130

n0derunner

APRIL 7, 2020

Often, when doing storage performance testing, we are interested in stressing the underlying storage devices, not the OS filesystem. Part 2, Accurately testing storage devices. appeared first on n0derunner.

Storage 52

Storage Testing Performance Testing Performance 52

Dynatrace

SEPTEMBER 5, 2024

The dashboard tracks a histogram chart of total storage utilized with logs daily. You can see in a table retention periods by the number of logs and storage they consumed. The dashboard also breaks down log volume by Grail buckets, showing you what buckets consume the most storage.

Innovation 257

Innovation AWS Analytics Storage 257

DZone

NOVEMBER 7, 2023

This is about how a cyber security service provider built its log storage and analysis system (LSAS) and realized 3X data writing speed, 7X query execution speed, and visualized management. Log Storage and Analysis Platform In this use case, the LSAS collects system logs from its enterprise users, scans them, and detects viruses.

Storage 219

Storage Speed Systems 219

DZone

JUNE 3, 2024

This is especially the case when it comes to taking advantage of vast amounts of data stored in cloud platforms like Amazon S3 - Simple Storage Service, which has become a central repository of data types ranging from the content of web applications to big data analytics.

Big Data 273

Big Data AWS Analytics Storage 273

The Morning Paper

NOVEMBER 5, 2019

File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution Aghayev et al., In this case, the assumption that a distributed storage backend should clearly be layered on top of a local file system. What is a distributed storage backend? SOSP’19. This is not surprising in hindsight.

Storage 64

Storage Systems Hardware Efficiency 64

n0derunner

JUNE 28, 2019

The post View from Nutanix storage during Postgres DB benchmark appeared first on n0derunner. Even though the log writes are sequential, they are low-concurrency and small size (looks like mostly 16K-32K). This write pattern is also a good candidate for oplog. These low-concurrency log writes also hit oplog.

Benchmarking 75

Benchmarking Storage Cache Database 75

Percona Community

JUNE 1, 2022

An important role is played by the time of their storage. Introduction Nowadays, the main tools for monitoring the operation of any application are metrics and logs. Often, in order to understand certain processes and predict their development in the future, we need to analyze metrics over a fairly long period of time.

Storage 52

Storage Metrics Monitoring Processing 52

All Things Distributed

JANUARY 23, 2012

Expanding the Cloud - The AWS Storage Gateway. Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customersâ?? With the launch of the AWS Storage Gateway our customers can now integrate their on-premises IT environment with AWSâ??s

Storage 111

Storage AWS Virtualization Cloud 111

DZone

MAY 3, 2023

In the previous posts, we covered things we had to do to upload files on the front end, things we had to do on the back end, and optimizing costs by moving file uploads to object storage.

Latency 190

Latency Speed Performance Storage 190

SQL Shack

JULY 14, 2020

This article explores the configuration of Windows failover clusters, storage controllers and quorum configurations for SQL Server Always On Availability Groups. Prerequisites In this series of articles, we will configure the SQL Server always on availability groups from end to end for your learning purpose.

Storage 75

Storage Availability Servers 75

Scalegrid

FEBRUARY 8, 2024

A distributed storage system is foundational in today’s data-driven landscape, ensuring data spread over multiple servers is reliable, accessible, and manageable. Understanding distributed storage is imperative as data volumes and the need for robust storage solutions rise.

Storage 130

Storage Systems Big Data Azure 130

DZone

FEBRUARY 5, 2023

Cosmos DB is a multimodal database in Azure that supports schema-less storage. For key object storage, RU tends to be less, but it still depends on the payload size. Cosmos DB can be a good candidate for a key-value store. By default, Cosmos DB containers tend to index all the fields of a document uploaded.

Azure 243

Azure Storage C++ Performance Testing 243

The Polyglot Developer

FEBRUARY 10, 2017

So I was researching object storage and I came across the open source distributed object storage software, Minio. After all they are both object storage solutions. Object storage is great for data backups so it gave me the idea to turn my Raspberry Pi into an object storage backup solution.

Storage 72

Storage Open Source Hardware Architecture 72

n0derunner

DECEMBER 22, 2020

The post A Generalized workload generator for storage IO appeared first on n0derunner.

Storage 52

Storage Benchmarking Virtualization Servers 52