DZone

JUNE 4, 2019

NVMe Storage Use Cases. NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. There are several AI/ML focused use cases to highlight.

Storage 100

Storage Performance Artificial Intelligence FinTech 100

DZone

JUNE 25, 2019

It’s really scary knowing that such corruptions are happening in the memory of our computers and servers – that is before they even reach the network and storage portions of the stack. That data must then be safely transported over a network to the storage system where it is written to disk.

Storage 120

Storage Systems Servers Transportation 120

DZone

MAY 22, 2019

However, Hive cannot access a single table directly using a single query with the data of this Hive table across different mediums of storage and different clusters. In this regard, data will always reside in the under-storage system as the source of truth and can be residing temporarily in the Alluxio file system.

Storage 100

Storage Systems Performance Cloud 100

Uber Engineering

SEPTEMBER 9, 2021

Uber deploys a few storage technologies to store business data based on their application model.

Storage 124

Storage Systems Engineering Technology 124

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. Snowflake is a data warehouse designed to overcome these limitations, and the fundamental mechanism by which it achieves this is the decoupling (disaggregation) of compute and storage. joins) during query processing. Disaggregation (or not).

Storage 112

Storage Cache Engineering Serverless 112

DZone

JUNE 3, 2019

For example, one well-respected vendor's standard solution is limited to 7.5TB of internal storage, and it can only scale to 30TB. Normally, GPU nodes don't have much room for SSDs, which limits the opportunity to train very deep neural networks that need more data.

Storage 100

Storage Performance Scalability Network 100

DZone

MAY 3, 2023

In the previous posts, we covered things we had to do to upload files on the front end, things we had to do on the back end, and optimizing costs by moving file uploads to object storage.

Latency 190

Latency Performance Speed Storage 190

DZone

JANUARY 29, 2020

To resolve the problem it was suggested to find more suitable data storage. There are several limitations to store and fetch such data (all restrictions could be found in official documentation ). For some internal reasons well known Amazon S3 bucket was chosen for this purpose. The choice affected the project's unit test base.

Storage 108

Storage Testing Efficiency Database 108

n0derunner

JULY 1, 2022

I was recently asked to investigate why Nutanix storage was not as fast as a competing solution in a PoC environment. One thing that seemed really odd was that the working set size for the tests were in the order of … The post Beware of tiny working-set-sizes when testing storage performance. appeared first on n0derunner.

Storage 52

Storage Testing Performance Benchmarking 52

SQL Shack

JULY 28, 2020

In this article, we will learn how to test our storage subsystems performance using Diskspd. The storage subsystem is one of the key performance factors for SQL Server because SQL Server storage engine stores database objects, tables, and indexes on the physical files.

Servers 82

Servers Storage Testing Database 82

The Morning Paper

JANUARY 28, 2020

Narrowing the gap between serverless and its state with storage functions , Zhang et al., Shredder is " a low-latency multi-tenant cloud store that allows small units of computation to be performed directly within storage nodes. " In from of them is a networking layer, and the in-memory storage layer holds the actual data.

Serverless 99

Serverless Storage Latency Servers 99

Particular Software

APRIL 12, 2021

Azure Storage Queues is a basic yet robust queueing service available on the Azure platform. For example, it lacks concepts like topics and subscriptions to build publisher and subscriber topologies—these are capabilities NServiceBus adds on top of Azure Storage Queues. Here’s how it works.

Azure 52

Azure Storage Transportation Java 52

Dynatrace

APRIL 19, 2023

ALLOW storage:buckets:read WHERE storage:bucket-name startsWith “default_”;ALLOW storage:events:read, storage:logs:read, storage:metrics:read, storage:entities:read, storage:bizevents:read, storage:spans:read; Storage Read per table Each table includes a policy that combines table and bucket access called Storage <tablename> Read.

Storage 210

Storage Systems Architecture Monitoring 210

DZone

MARCH 29, 2023

Data migration involves transferring data from on-premise storage to the cloud. Data migration is the process of moving data from one location to another, which is an essential aspect of cloud migration. With the rapid adoption of cloud computing , businesses are moving their IT infrastructure to the cloud.

Best Practices 313

Best Practices Cloud Data Engineering Storage 313

DZone

MAY 5, 2023

The core of Kafka is messaging at any scale in combination with a distributed storage (= commit log) for reliable durability, decoupling of applications, and replayability of historical data. The Tremendous Adoption of Apache Kafka and Apache Flink Apache Kafka became the de facto standard for data streaming.

Open Source 244

Open Source Storage Engineering Innovation 244

DZone

FEBRUARY 5, 2023

Cosmos DB is a multimodal database in Azure that supports schema-less storage. For key object storage, RU tends to be less, but it still depends on the payload size. Cosmos DB can be a good candidate for a key-value store. By default, Cosmos DB containers tend to index all the fields of a document uploaded.

Azure 239

Azure Storage Performance Testing Database 239

DZone

MAY 16, 2023

Virtualization is a technology that can create servers, storage devices, and networks all in virtual space. One area that virtualization technology is making a huge impact is the security sector. How Is Virtualization Technology Used? Devices connect to a virtual network to share data and resources.

Virtualization 196

Virtualization Hardware Servers Storage 196

Percona Community

OCTOBER 23, 2020

In this blog post we’re going to see those technologies at work to give us awesome block storage performance with flexibility and simple operations. It’s a new generation in storage software, designed for super high speed low latency NVMe devices. Why is SPDK exciting?

Storage 52

Storage Latency Database Speed 52

Dynatrace

NOVEMBER 11, 2022

Therefore, they need an environment that offers scalable computing, storage, and networking. Hyperconverged infrastructure (HCI) is an IT architecture that combines servers, storage, and networking functions into a unified, software-centric platform to streamline resource management. What is hyperconverged infrastructure?

Infrastructure 201

Infrastructure Storage Servers Cloud 201

Dynatrace

APRIL 28, 2023

This means compromising between keeping data available as long as possible for analysis while juggling the costs and overhead of storage, archiving, and retrieval. For example, suppose data has to be retained for a longer period because of legal or business reasons.

Storage 214

Storage Analytics Infrastructure Metrics 214

Percona

APRIL 20, 2023

In this blog post, you will learn: how to leverage FDE on Kubernetes with Percona Operator for PostgreSQL how to start using encrypted storage for already running cluster Prepare In most public clouds, block storage is not encrypted by default. The configuration of the storage class depends on your storage plugin.

Storage 120

Storage AWS Database Cloud 120

DZone

MAY 7, 2023

By the end of the tutorial, you’ll have a running Spring Boot app that serves as an HTTP API server with ScyllaDB as the underlying data storage. Additionally, an HTTP API tool, Postman, is used to interact with our application and to test the API functionality. And you’ll learn how ScyllaDB can be used to store time series data.

Servers 259

Servers Storage Testing 259

n0derunner

APRIL 7, 2020

Often, when doing storage performance testing, we are interested in stressing the underlying storage devices, not the OS filesystem. Part 2, Accurately testing storage devices. appeared first on n0derunner.

Storage 52

Storage Testing Performance Testing Performance 52

DZone

FEBRUARY 23, 2023

Any real-time analytics provider or batching/storage adaptor can transform and store data supplied to an event hub. Introduction With big data streaming platform and event ingestion service Azure Event Hubs , millions of events can be received and processed in a single second.

Azure 307

Azure Scalability Big Data Storage 307

DZone

APRIL 11, 2023

It provides built-in connectors for various data sources such as databases, file systems, cloud storage, and more. Ingesting Data With Azure Data Factory Azure Data Factory is a cloud-based data integration service enabling you to ingest data from various sources into a cloud-based data lake or warehouse.

Azure 246

Azure Analytics Cloud Scalability 246

Dynatrace

OCTOBER 4, 2022

Data warehouses offer a single storage repository for structured data and provide a source of truth for organizations. Unlike data warehouses, however, data is not transformed before landing in storage. A data lakehouse provides a cost-effective storage layer for both structured and unstructured data. Data management.

Storage 210

Storage Analytics Architecture Artificial Intelligence 210

Percona Community

JUNE 1, 2022

An important role is played by the time of their storage. Introduction Nowadays, the main tools for monitoring the operation of any application are metrics and logs. Often, in order to understand certain processes and predict their development in the future, we need to analyze metrics over a fairly long period of time.

Metrics 52

Metrics Storage Monitoring Processing 52

Dynatrace

AUGUST 1, 2022

You quickly realize that it will take ages to fill up the overprovisioned database storage. Two days later, your database runs out of storage in the middle of the night. Therefore, you don’t know your current growth rate and can’t estimate the required storage for keeping the database up and running for the next month.

Storage 188

Storage Database Cache Metrics 188

The Netflix TechBlog

APRIL 27, 2023

While Atlas is architected around compute & storage separation, and we could theoretically just scale the query layer to meet the increased query demand, every query, regardless of its type, has a data component that needs to be pushed down to the storage layer.

Cache 253

Cache Metrics Database Storage 253

Percona

MARCH 1, 2023

Inputs These are the following instances that we will start with: AWS RDS for MySQL in us-east-1 10 x db.r5.4xlarge 200 GB storage each The cost of RDS consists mostly of two things – compute and storage. year For ten instances, it will be $168,192 per year Default gp2 storage is $0.115 per GB-month, for 200 GB, it is $22.50/month

Cloud 131

Cloud AWS Storage Database 131

The Morning Paper

NOVEMBER 5, 2019

File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution Aghayev et al., In this case, the assumption that a distributed storage backend should clearly be layered on top of a local file system. What is a distributed storage backend? SOSP’19. This is not surprising in hindsight.

Storage 64

Storage Systems Hardware Cache 64

DZone

SEPTEMBER 22, 2022

Users can allocate different storage tiers as the resources for Alluxio workers, including MEM/SSD/HDD, which are further composed of directories. Alluxio workers are responsible for managing local resources, and they store data as blocks.

Code 246

Code Virtualization Storage Systems 246

SQL Shack

JULY 14, 2020

This article explores the configuration of Windows failover clusters, storage controllers and quorum configurations for SQL Server Always On Availability Groups. Prerequisites In this series of articles, we will configure the SQL Server always on availability groups from end to end for your learning purpose.

Servers 75

Servers Storage Availability 75

The Netflix TechBlog

SEPTEMBER 24, 2021

From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step. Since not all projects are terabytes projects, allocating the largest cloud storage to all packager instances is not an efficient use of cloud resources.

Cloud 243

Cloud Cache Storage Latency 243

The Netflix TechBlog

FEBRUARY 13, 2023

Media Feature Storage: Amber Storage Media feature computation tends to be expensive and time-consuming. This feature store is equipped with a data replication system that enables copying data to different storage solutions depending on the required access patterns.

Media 299

Media Infrastructure Storage Systems 299

DZone

MARCH 2, 2023

Not only has near-infinitely scalable cloud storage reduced the burden of storing large video files, but CDNs (content delivery networks) deployed by video streaming and social media giants in this timeframe have all but eliminated those slow server-to-client buffering times, which initially plagued the user experience.

Java 272

Java Scalability Traffic Social Media 272

Dynatrace

OCTOBER 4, 2022

Teams have introduced workarounds to reduce storage costs. Additionally, efforts such as lowered data retention times, two-tiered storage systems, shaky index management, sampled data, and data pipelines reduce the overall amount of stored data. Stop worrying about log data ingest and storage — start creating value instead.

Analytics 235

Analytics Storage Serverless Architecture 235