Uber Engineering

JULY 6, 2024

Kafka Tiered Storage, developed in collaboration with the Apache Kafka community, introduces the separation of storage and processing in brokers, significantly improving the scalability, reliability, and efficiency of Kafka clusters.

Storage 75

Storage Scalability Efficiency Processing 75

Percona

DECEMBER 11, 2023

There are a wealth of options on how you can approach storage configuration in Percona Operator for PostgreSQL , and in this blog post, we review various storage strategies — from basics to more sophisticated use cases. For example, you can choose the public cloud storage type – gp3, io2, etc, or set file system.

Storage 96

Storage Strategy Cloud Azure 96

DZone

JUNE 28, 2023

However, lurking beneath the surface lies a complex web of data storage and retrieval. That's why knowing how to debug mobile app database problems and optimize data storage performance is essential for developers seeking excellence. In the dynamic realm of mobile app development , a flawless user experience is the ultimate goal.

Storage 203

Storage Mobile Database Performance 203

Dynatrace

JULY 29, 2021

Using existing storage resources optimally is key to being able to capture the right data over time. Increased storage space availability. The compression of transaction data older than three days can free up to 50% more storage space in your Dynatrace Managed Cluster. Data compression is completed on June 12.

Storage 199

Storage Virtualization Infrastructure Availability 199

All Things Distributed

JULY 27, 2023

Three distinct perspectives on scale that come along with building and operating a storage system the size of S3.

Storage 147

Storage Systems 147

DZone

AUGUST 6, 2021

On a small scale, this isn’t problematic but when dealing with large-scale deployments, organizations end up using lots of computing, storage, and human resources just to manage their indexes. As the database grows in size, so does the index management cost.

Storage 189

Storage Database Systems Efficiency 189

InfoQ

MAY 15, 2023

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud. By Facundo Agriel

Storage 112

Storage Systems Scalability Cloud 112

Percona

FEBRUARY 10, 2023

Today along with their team, we will see how pvc-autoresizer can automate storage scaling for MongoDB clusters on Kubernetes. Our goal is to automate storage scaling when our disk reaches a certain threshold of use and simultaneously reduce the amount of alert noise related to that. kubectl annotate pvc --all resize.topolvm.io/storage_limit="100Gi"

Storage 102

Storage Blockchain AWS Cloud 102

Percona Community

JANUARY 9, 2024

Automated storage scaling stands out as a critical challenge. In the world of Kubernetes, where everything evolves quickly. Members of the Data on Kubernetes Community have proposed a solution to address this issue for Kubernetes operators.

Storage 67

Storage 67

Percona

JANUARY 4, 2024

Managing storage and performance efficiently in your MySQL database is crucial, and general tablespaces offer flexibility in achieving this. In contrast to the single system tablespace that holds system tables by default, general tablespaces are user-defined storage containers for multiple InnoDB tables.

Storage 82

Storage Engineering Database Hardware 82

InfoQ

OCTOBER 14, 2023

times the storage per node than before” without a price change. Google recently announced various improvements to Cloud Spanner, its distributed, decoupled relational database service with a “50% increase in throughput and 2.5 By Steef-Jan Wiggers

Google 101

Google Storage Cloud Database 101

DZone

JUNE 25, 2019

It’s really scary knowing that such corruptions are happening in the memory of our computers and servers – that is before they even reach the network and storage portions of the stack. That data must then be safely transported over a network to the storage system where it is written to disk.

Storage 145

Storage Systems Transportation Network 145

DZone

JUNE 28, 2024

At first, data tiering was a tactic used by storage systems to reduce data storage costs. This involved grouping data that was not accessed as often into more affordable, if less effective, storage array choices. Even though they are quite costly, SSDs and flash can be categorized as high-performance storage classes.

Azure 223

Azure Storage Performance Cloud 223

n0derunner

SEPTEMBER 15, 2023

Background For the past few months engineers from Nutanix have been participating in the MLPerftm Storage benchmark which is designed to measure the storage performance required for ML training workloads. appeared first on n0derunner.

Benchmarking 59

Benchmarking Storage Servers Engineering 59

DZone

JANUARY 29, 2020

To resolve the problem it was suggested to find more suitable data storage. There are several limitations to store and fetch such data (all restrictions could be found in official documentation ). For some internal reasons well known Amazon S3 bucket was chosen for this purpose. The choice affected the project's unit test base.

Storage 124

Storage Efficiency Testing Database 124

DZone

JUNE 4, 2019

NVMe Storage Use Cases. NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. There are several AI/ML focused use cases to highlight.

Storage 100

Storage Artificial Intelligence FinTech Performance 100

DZone

JUNE 3, 2019

For example, one well-respected vendor's standard solution is limited to 7.5TB of internal storage, and it can only scale to 30TB. Normally, GPU nodes don't have much room for SSDs, which limits the opportunity to train very deep neural networks that need more data.

Storage 100

Storage Performance Network Scalability 100

DZone

MAY 22, 2019

However, Hive cannot access a single table directly using a single query with the data of this Hive table across different mediums of storage and different clusters. In this regard, data will always reside in the under-storage system as the source of truth and can be residing temporarily in the Alluxio file system.

Storage 100

Storage Systems Virtualization Cloud 100

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. Snowflake is a data warehouse designed to overcome these limitations, and the fundamental mechanism by which it achieves this is the decoupling (disaggregation) of compute and storage. joins) during query processing. Disaggregation (or not).

Storage 112

Storage Engineering Cache Serverless 112

Uber Engineering

SEPTEMBER 9, 2021

Uber deploys a few storage technologies to store business data based on their application model.

Storage 134

Storage Systems Engineering Technology 134

Percona

FEBRUARY 1, 2023

This post will look at using The Oversized-Attribute Storage Technique (TOAST) to improve performance and scalability. Therefore, TOAST is a storage technique used in PostgreSQL to handle large data objects such as images, videos, and audio files.

Storage 85

Storage Scalability Strategy Performance 85

SQL Shack

JULY 28, 2020

In this article, we will learn how to test our storage subsystems performance using Diskspd. The storage subsystem is one of the key performance factors for SQL Server because SQL Server storage engine stores database objects, tables, and indexes on the physical files.

Storage 82

Storage Servers Testing Engineering 82

DZone

JULY 1, 2024

Disk Storage Space Evaluation The storage space requirements for a GBase cluster are calculated based on the data volume of the business system, the choice of compression algorithm, and the number of cluster replicas. The calculation method for the physical disk capacity of the GBase cluster is as follows:

Storage 147

Storage Systems 147

n0derunner

JULY 1, 2022

I was recently asked to investigate why Nutanix storage was not as fast as a competing solution in a PoC environment. One thing that seemed really odd was that the working set size for the tests were in the order of … The post Beware of tiny working-set-sizes when testing storage performance. appeared first on n0derunner.

Storage 52

Storage Testing Performance Benchmarking 52

The Morning Paper

JANUARY 28, 2020

Narrowing the gap between serverless and its state with storage functions , Zhang et al., Shredder is " a low-latency multi-tenant cloud store that allows small units of computation to be performed directly within storage nodes. " In from of them is a networking layer, and the in-memory storage layer holds the actual data.

Serverless 99

Serverless Storage Latency Cloud 99

Particular Software

APRIL 12, 2021

Azure Storage Queues is a basic yet robust queueing service available on the Azure platform. For example, it lacks concepts like topics and subscriptions to build publisher and subscriber topologies—these are capabilities NServiceBus adds on top of Azure Storage Queues. Here’s how it works.

Azure 52

Azure Storage Transportation Java 52

DZone

SEPTEMBER 25, 2023

It is the second of a series of articles that is built on top of that project, representing experiments with various statistical and machine learning models, data pipelines implemented using existing DAG tools, and storage services, both cloud-based and alternative on-premises solutions.

Storage 264

Storage AWS Architecture Cloud 264

DZone

JUNE 24, 2024

Objectives Modern AI innovations require proper infrastructure, especially concerning data throughput and storage capabilities. While GPUs drive faster results, legacy storage solutions often lag behind, causing inefficient resource utilization and extended times in completing the project.

Systems 130

Systems Software Software Storage 130

Percona Community

OCTOBER 23, 2020

In this blog post we’re going to see those technologies at work to give us awesome block storage performance with flexibility and simple operations. It’s a new generation in storage software, designed for super high speed low latency NVMe devices. Why is SPDK exciting?

Storage 52

Storage Latency Speed Database 52

DZone

JUNE 3, 2024

This is especially the case when it comes to taking advantage of vast amounts of data stored in cloud platforms like Amazon S3 - Simple Storage Service, which has become a central repository of data types ranging from the content of web applications to big data analytics.

Big Data 278

Big Data AWS Analytics Storage 278

n0derunner

APRIL 7, 2020

Often, when doing storage performance testing, we are interested in stressing the underlying storage devices, not the OS filesystem. Part 2, Accurately testing storage devices. appeared first on n0derunner.

Storage 52

Storage Testing Performance Testing Performance 52

DZone

NOVEMBER 7, 2023

This is about how a cyber security service provider built its log storage and analysis system (LSAS) and realized 3X data writing speed, 7X query execution speed, and visualized management. Log Storage and Analysis Platform In this use case, the LSAS collects system logs from its enterprise users, scans them, and detects viruses.

Storage 222

Storage Speed Systems 222

Dynatrace

MAY 21, 2024

Masking at storage: Data is persistently masked upon ingestion into Dynatrace. Leverage three masking layers Masking at capture and masking at storage operations exclude targeted sensitive data points. When using Dynatrace OneAgent ® , captured data doesn’t leave the monitored environment.

Government 210

Government Storage Tuning Monitoring 210

The Morning Paper

NOVEMBER 5, 2019

File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution Aghayev et al., In this case, the assumption that a distributed storage backend should clearly be layered on top of a local file system. What is a distributed storage backend? SOSP’19. This is not surprising in hindsight.

Storage 64

Storage Systems Hardware Efficiency 64

n0derunner

JUNE 28, 2019

The post View from Nutanix storage during Postgres DB benchmark appeared first on n0derunner. Even though the log writes are sequential, they are low-concurrency and small size (looks like mostly 16K-32K). This write pattern is also a good candidate for oplog. These low-concurrency log writes also hit oplog.

Benchmarking 75

Benchmarking Storage Cache Database 75

Percona Community

JUNE 1, 2022

An important role is played by the time of their storage. Introduction Nowadays, the main tools for monitoring the operation of any application are metrics and logs. Often, in order to understand certain processes and predict their development in the future, we need to analyze metrics over a fairly long period of time.

Storage 52

Storage Metrics Monitoring Processing 52

DZone

JUNE 13, 2024

For instance, for a storage account named "Foo", its connection string might be "Bar". While it encompasses many functionalities, the article will focus on Managed Identities. Why Managed Identities? Initially, Azure resources were accessed using connecting strings--keys tied to specific resources.

Azure 162

Azure Storage 162