InfoQ

OCTOBER 14, 2023

times the storage per node than before” without a price change. Google recently announced various improvements to Cloud Spanner, its distributed, decoupled relational database service with a “50% increase in throughput and 2.5 By Steef-Jan Wiggers

Storage 98

Storage Cloud Google Database 98

InfoQ

MAY 15, 2023

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud. By Facundo Agriel

Storage 110

Storage Systems Scalability Cloud 110

DZone

AUGUST 6, 2021

On a small scale, this isn’t problematic but when dealing with large-scale deployments, organizations end up using lots of computing, storage, and human resources just to manage their indexes. As the database grows in size, so does the index management cost.

Storage 189

Storage Database Systems Monitoring 189

n0derunner

SEPTEMBER 15, 2023

Background For the past few months engineers from Nutanix have been participating in the MLPerftm Storage benchmark which is designed to measure the storage performance required for ML training workloads. appeared first on n0derunner.

Benchmarking 59

Benchmarking Storage Servers Engineering 59

DZone

SEPTEMBER 25, 2023

It is the second of a series of articles that is built on top of that project, representing experiments with various statistical and machine learning models, data pipelines implemented using existing DAG tools, and storage services, both cloud-based and alternative on-premises solutions.

Storage 257

Storage AWS Servers Architecture 257

Percona

FEBRUARY 1, 2023

This post will look at using The Oversized-Attribute Storage Technique (TOAST) to improve performance and scalability. Therefore, TOAST is a storage technique used in PostgreSQL to handle large data objects such as images, videos, and audio files.

Storage 94

Storage Scalability Performance Database 94

DZone

JUNE 25, 2019

It’s really scary knowing that such corruptions are happening in the memory of our computers and servers – that is before they even reach the network and storage portions of the stack. That data must then be safely transported over a network to the storage system where it is written to disk.

Storage 129

Storage Systems Servers Transportation 129

DZone

JUNE 4, 2019

NVMe Storage Use Cases. NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. There are several AI/ML focused use cases to highlight.

Storage 100

Storage Performance Artificial Intelligence FinTech 100

Uber Engineering

SEPTEMBER 9, 2021

Uber deploys a few storage technologies to store business data based on their application model.

Storage 134

Storage Systems Engineering Technology 134

DZone

MAY 22, 2019

However, Hive cannot access a single table directly using a single query with the data of this Hive table across different mediums of storage and different clusters. In this regard, data will always reside in the under-storage system as the source of truth and can be residing temporarily in the Alluxio file system.

Storage 100

Storage Systems Performance Cloud 100

Dynatrace

DECEMBER 5, 2023

Streamline privacy requirements with flexible retention periods Data retention is a critical aspect of data handling, and it’s not just about privacy compliance—it’s about having the flexibility to optimize data storage times in Grail for your Dynatrace use cases. Other data types will be available soon). What’s next?

Storage 162

Storage Availability Best Practices Healthcare 162

DZone

JANUARY 29, 2020

To resolve the problem it was suggested to find more suitable data storage. There are several limitations to store and fetch such data (all restrictions could be found in official documentation ). For some internal reasons well known Amazon S3 bucket was chosen for this purpose. The choice affected the project's unit test base.

Storage 112

Storage Testing Efficiency Database 112

DZone

JUNE 3, 2019

For example, one well-respected vendor's standard solution is limited to 7.5TB of internal storage, and it can only scale to 30TB. Normally, GPU nodes don't have much room for SSDs, which limits the opportunity to train very deep neural networks that need more data.

Storage 100

Storage Performance Scalability Network 100

DZone

NOVEMBER 7, 2023

This is about how a cyber security service provider built its log storage and analysis system (LSAS) and realized 3X data writing speed, 7X query execution speed, and visualized management. Log Storage and Analysis Platform In this use case, the LSAS collects system logs from its enterprise users, scans them, and detects viruses.

Storage 196

Storage Speed Systems 196

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. Snowflake is a data warehouse designed to overcome these limitations, and the fundamental mechanism by which it achieves this is the decoupling (disaggregation) of compute and storage. joins) during query processing. Disaggregation (or not).

Storage 112

Storage Cache Engineering Serverless 112

SQL Shack

JULY 28, 2020

In this article, we will learn how to test our storage subsystems performance using Diskspd. The storage subsystem is one of the key performance factors for SQL Server because SQL Server storage engine stores database objects, tables, and indexes on the physical files.

Servers 82

Servers Storage Testing Database 82

n0derunner

JULY 1, 2022

I was recently asked to investigate why Nutanix storage was not as fast as a competing solution in a PoC environment. One thing that seemed really odd was that the working set size for the tests were in the order of … The post Beware of tiny working-set-sizes when testing storage performance. appeared first on n0derunner.

Storage 52

Storage Testing Performance Benchmarking 52

The Morning Paper

JANUARY 28, 2020

Narrowing the gap between serverless and its state with storage functions , Zhang et al., Shredder is " a low-latency multi-tenant cloud store that allows small units of computation to be performed directly within storage nodes. " In from of them is a networking layer, and the in-memory storage layer holds the actual data.

Serverless 99

Serverless Storage Latency Servers 99

DZone

NOVEMBER 25, 2023

We often dwell on the technical aspects of database selection, focusing on performance metrics , storage capacity, and querying capabilities. Yet, the impact of choosing the right NoSQL database goes beyond these parameters; it affects your business outcomes.

Database 266

Database Latency Metrics Speed 266

Particular Software

APRIL 12, 2021

Azure Storage Queues is a basic yet robust queueing service available on the Azure platform. For example, it lacks concepts like topics and subscriptions to build publisher and subscriber topologies—these are capabilities NServiceBus adds on top of Azure Storage Queues. Here’s how it works.

Azure 52

Azure Storage Transportation Java 52

DZone

OCTOBER 10, 2023

How do you make Jaeger even more effective for monitoring and troubleshooting distributed applications, especially in high-traffic, demanding environments where a high-performance storage solution is critical? Use the best-performing Jaeger storage backend that you can find.

Performance 158

Performance Storage Traffic Cloud 158

DZone

NOVEMBER 20, 2023

They are characterized by resources such as CPU, RAM, storage capacity, or even bandwidth. EC2 instances on AWS are virtual servers that can be used to run applications and services on the AWS cloud. Before you even begin exploring the different AWS EC2 instances , it is necessary to know your needs and your use cases.

AWS 174

AWS Servers Storage Cloud 174

DZone

OCTOBER 24, 2023

Logstash: a log-processing tool that collects logs from various sources, parses them, and sends them to Elasticsearch for storage and analysis. Kibana: A powerful visualization tool that allows you to explore and analyze the data stored in Elasticsearch using interactive charts, graphs, and dashboards.

Scalability 235

Scalability Infrastructure Storage Database 235

DZone

NOVEMBER 27, 2023

An open-source distributed SQL query engine, Trino is widely used for data analytics on distributed data storage. Optimizing Trino to make it faster can help organizations achieve quicker insights and better user experiences, as well as cut costs and improve infrastructure efficiency and scalability. But how do we do that?

Tuning 150

Tuning Performance Speed Open Source 150

Percona Community

OCTOBER 23, 2020

In this blog post we’re going to see those technologies at work to give us awesome block storage performance with flexibility and simple operations. It’s a new generation in storage software, designed for super high speed low latency NVMe devices. Why is SPDK exciting?

Storage 52

Storage Latency Database Speed 52

DZone

MAY 3, 2023

In the previous posts, we covered things we had to do to upload files on the front end, things we had to do on the back end, and optimizing costs by moving file uploads to object storage.

Latency 180

Latency Performance Speed Storage 180

DZone

AUGUST 29, 2023

It is the first of a series of articles that will be built on top of that project, representing experiments with various statistical and machine learning models, data pipelines implemented using existing DAG tools, and storage services, both cloud-based and alternative on-premises solutions.

Storage 257

Storage Cloud 257

DZone

FEBRUARY 5, 2023

Cosmos DB is a multimodal database in Azure that supports schema-less storage. For key object storage, RU tends to be less, but it still depends on the payload size. Cosmos DB can be a good candidate for a key-value store. By default, Cosmos DB containers tend to index all the fields of a document uploaded.

Azure 242

Azure Storage Performance Testing Database 242

Dynatrace

NOVEMBER 9, 2023

Storage container metrics Track the usage and performance of storage containers to optimize resource allocation. Disk metrics Monitor the performance of disks to ensure efficient data storage and retrieval. Virtual disk metrics Keep tabs on virtual disk performance, identifying any potential storage bottlenecks.

Metrics 190

Metrics Monitoring Storage Infrastructure 190

n0derunner

APRIL 7, 2020

Often, when doing storage performance testing, we are interested in stressing the underlying storage devices, not the OS filesystem. Part 2, Accurately testing storage devices. appeared first on n0derunner.

Storage 52

Storage Testing Performance Testing Performance 52

DZone

JUNE 6, 2023

When dealing with IoT, one of the first things that come to mind is the limited processing, networking, and storage capabilities these devices operate with. A messaging protocol is a set of rules and formats that are agreed upon among entities that want to communicate with each other.

Tuning 290

Tuning IoT Servers Storage 290

Dynatrace

NOVEMBER 11, 2022

Therefore, they need an environment that offers scalable computing, storage, and networking. Hyperconverged infrastructure (HCI) is an IT architecture that combines servers, storage, and networking functions into a unified, software-centric platform to streamline resource management. What is hyperconverged infrastructure?

Infrastructure 199

Infrastructure Servers Storage Cloud 199

DZone

MARCH 29, 2023

Data migration involves transferring data from on-premise storage to the cloud. Data migration is the process of moving data from one location to another, which is an essential aspect of cloud migration. With the rapid adoption of cloud computing , businesses are moving their IT infrastructure to the cloud.

Best Practices 301

Best Practices Cloud Data Engineering Storage 301

Dynatrace

OCTOBER 23, 2023

Secondly, determining the correct allocation of resources (CPU, memory, storage) to each virtual machine to ensure optimal performance without over-provisioning can be difficult. Firstly, managing virtual networks can be complex as networking in a virtual environment differs significantly from traditional networking.

Efficiency 212

Efficiency Performance Hardware Virtualization 212

Dynatrace

APRIL 19, 2023

ALLOW storage:buckets:read WHERE storage:bucket-name startsWith “default_”;ALLOW storage:events:read, storage:logs:read, storage:metrics:read, storage:entities:read, storage:bizevents:read, storage:spans:read; Storage Read per table Each table includes a policy that combines table and bucket access called Storage <tablename> Read.

Storage 206

Storage Systems Architecture Monitoring 206

Percona Community

JUNE 1, 2022

An important role is played by the time of their storage. Introduction Nowadays, the main tools for monitoring the operation of any application are metrics and logs. Often, in order to understand certain processes and predict their development in the future, we need to analyze metrics over a fairly long period of time.

Metrics 52

Metrics Storage Monitoring Processing 52

Dynatrace

OCTOBER 4, 2022

Data warehouses offer a single storage repository for structured data and provide a source of truth for organizations. Unlike data warehouses, however, data is not transformed before landing in storage. A data lakehouse provides a cost-effective storage layer for both structured and unstructured data. Data management.

Storage 206

Storage Analytics Architecture Artificial Intelligence 206