The Netflix TechBlog

DECEMBER 21, 2020

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203

Storage Latency Efficiency Data Engineering 203

DZone

JUNE 28, 2023

However, lurking beneath the surface lies a complex web of data storage and retrieval. That's why knowing how to debug mobile app database problems and optimize data storage performance is essential for developers seeking excellence. In the dynamic realm of mobile app development , a flawless user experience is the ultimate goal.

Storage 199

Storage Mobile Database Performance 199

Dynatrace

JULY 29, 2021

Using existing storage resources optimally is key to being able to capture the right data over time. Increased storage space availability. The compression of transaction data older than three days can free up to 50% more storage space in your Dynatrace Managed Cluster. Data compression is completed on June 12.

Storage 206

Storage Virtualization Infrastructure Availability 206

Percona

FEBRUARY 10, 2023

Today along with their team, we will see how pvc-autoresizer can automate storage scaling for MongoDB clusters on Kubernetes. Our goal is to automate storage scaling when our disk reaches a certain threshold of use and simultaneously reduce the amount of alert noise related to that. kubectl annotate pvc --all resize.topolvm.io/storage_limit="100Gi"

Storage 112

Storage Blockchain AWS Cloud 112

All Things Distributed

JULY 27, 2023

Three distinct perspectives on scale that come along with building and operating a storage system the size of S3.

Storage 147

Storage Systems 147

Percona

JANUARY 4, 2024

Managing storage and performance efficiently in your MySQL database is crucial, and general tablespaces offer flexibility in achieving this. In contrast to the single system tablespace that holds system tables by default, general tablespaces are user-defined storage containers for multiple InnoDB tables.

Storage 90

Storage Engineering Database Hardware 90

InfoQ

MAY 15, 2023

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud. By Facundo Agriel

Storage 115

Storage Systems Scalability Cloud 115

Percona Community

JANUARY 9, 2024

Automated storage scaling stands out as a critical challenge. In the world of Kubernetes, where everything evolves quickly. Members of the Data on Kubernetes Community have proposed a solution to address this issue for Kubernetes operators.

Storage 67

Storage 67

DZone

AUGUST 6, 2021

On a small scale, this isn’t problematic but when dealing with large-scale deployments, organizations end up using lots of computing, storage, and human resources just to manage their indexes. As the database grows in size, so does the index management cost.

Storage 189

Storage Database Systems Efficiency 189

InfoQ

OCTOBER 14, 2023

times the storage per node than before” without a price change. Google recently announced various improvements to Cloud Spanner, its distributed, decoupled relational database service with a “50% increase in throughput and 2.5 By Steef-Jan Wiggers

Google 110

Google Storage Cloud Database 110

The Netflix TechBlog

MARCH 6, 2019

Mounting object storage in Netflix’s media processing platform By Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. Our object storage service splits objects into many parts and stores them in S3.

Media 214

Media Storage Processing Cache 214

DZone

JUNE 25, 2019

It’s really scary knowing that such corruptions are happening in the memory of our computers and servers – that is before they even reach the network and storage portions of the stack. That data must then be safely transported over a network to the storage system where it is written to disk.

Storage 145

Storage Systems Transportation Network 145

n0derunner

SEPTEMBER 15, 2023

Background For the past few months engineers from Nutanix have been participating in the MLPerftm Storage benchmark which is designed to measure the storage performance required for ML training workloads. appeared first on n0derunner.

Benchmarking 59

Benchmarking Storage Servers Engineering 59

DZone

JANUARY 29, 2020

To resolve the problem it was suggested to find more suitable data storage. There are several limitations to store and fetch such data (all restrictions could be found in official documentation ). For some internal reasons well known Amazon S3 bucket was chosen for this purpose. The choice affected the project's unit test base.

Storage 124

Storage Efficiency Testing Database 124

DZone

JUNE 4, 2019

NVMe Storage Use Cases. NVMe storage's strong performance, combined with the capacity and data availability benefits of shared NVMe storage over local SSD, makes it a strong solution for AI/ML infrastructures of any size. There are several AI/ML focused use cases to highlight.

Storage 100

Storage FinTech Artificial Intelligence Performance 100

DZone

JUNE 3, 2019

For example, one well-respected vendor's standard solution is limited to 7.5TB of internal storage, and it can only scale to 30TB. Normally, GPU nodes don't have much room for SSDs, which limits the opportunity to train very deep neural networks that need more data.

Storage 100

Storage Performance Network Scalability 100

DZone

MAY 22, 2019

However, Hive cannot access a single table directly using a single query with the data of this Hive table across different mediums of storage and different clusters. In this regard, data will always reside in the under-storage system as the source of truth and can be residing temporarily in the Alluxio file system.

Storage 100

Storage Systems Virtualization Cloud 100

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. Snowflake is a data warehouse designed to overcome these limitations, and the fundamental mechanism by which it achieves this is the decoupling (disaggregation) of compute and storage. joins) during query processing. Disaggregation (or not).

Storage 112

Storage Engineering Cache Serverless 112

Uber Engineering

SEPTEMBER 9, 2021

Uber deploys a few storage technologies to store business data based on their application model.

Storage 134

Storage Systems Engineering Technology 134

Percona

FEBRUARY 1, 2023

This post will look at using The Oversized-Attribute Storage Technique (TOAST) to improve performance and scalability. Therefore, TOAST is a storage technique used in PostgreSQL to handle large data objects such as images, videos, and audio files.

Storage 92

Storage Scalability Strategy Performance 92

SQL Shack

JULY 28, 2020

In this article, we will learn how to test our storage subsystems performance using Diskspd. The storage subsystem is one of the key performance factors for SQL Server because SQL Server storage engine stores database objects, tables, and indexes on the physical files.

Storage 82

Storage Servers Testing Engineering 82

n0derunner

JULY 1, 2022

I was recently asked to investigate why Nutanix storage was not as fast as a competing solution in a PoC environment. One thing that seemed really odd was that the working set size for the tests were in the order of … The post Beware of tiny working-set-sizes when testing storage performance. appeared first on n0derunner.

Storage 52

Storage Testing Performance Benchmarking 52

The Morning Paper

JANUARY 28, 2020

Narrowing the gap between serverless and its state with storage functions , Zhang et al., Shredder is " a low-latency multi-tenant cloud store that allows small units of computation to be performed directly within storage nodes. " In from of them is a networking layer, and the in-memory storage layer holds the actual data.

Serverless 99

Serverless Storage Latency Hardware 99

DZone

SEPTEMBER 25, 2023

It is the second of a series of articles that is built on top of that project, representing experiments with various statistical and machine learning models, data pipelines implemented using existing DAG tools, and storage services, both cloud-based and alternative on-premises solutions.

Storage 258

Storage AWS Architecture Cloud 258

Particular Software

APRIL 12, 2021

Azure Storage Queues is a basic yet robust queueing service available on the Azure platform. For example, it lacks concepts like topics and subscriptions to build publisher and subscriber topologies—these are capabilities NServiceBus adds on top of Azure Storage Queues. Here’s how it works.

Azure 52

Azure Storage Transportation Java 52

Percona Community

OCTOBER 23, 2020

In this blog post we’re going to see those technologies at work to give us awesome block storage performance with flexibility and simple operations. It’s a new generation in storage software, designed for super high speed low latency NVMe devices. Why is SPDK exciting?

Storage 52

Storage Latency Speed Database 52

DZone

NOVEMBER 7, 2023

This is about how a cyber security service provider built its log storage and analysis system (LSAS) and realized 3X data writing speed, 7X query execution speed, and visualized management. Log Storage and Analysis Platform In this use case, the LSAS collects system logs from its enterprise users, scans them, and detects viruses.

Storage 216

Storage Speed Systems 216

n0derunner

APRIL 7, 2020

Often, when doing storage performance testing, we are interested in stressing the underlying storage devices, not the OS filesystem. Part 2, Accurately testing storage devices. appeared first on n0derunner.

Storage 52

Storage Testing Performance Testing Performance 52

Dynatrace

DECEMBER 5, 2023

Streamline privacy requirements with flexible retention periods Data retention is a critical aspect of data handling, and it’s not just about privacy compliance—it’s about having the flexibility to optimize data storage times in Grail for your Dynatrace use cases. Other data types will be available soon). What’s next?

Storage 172

Storage Healthcare Best Practices Speed 172

Percona Community

JUNE 1, 2022

An important role is played by the time of their storage. Introduction Nowadays, the main tools for monitoring the operation of any application are metrics and logs. Often, in order to understand certain processes and predict their development in the future, we need to analyze metrics over a fairly long period of time.

Storage 52

Storage Metrics Monitoring Processing 52

The Morning Paper

NOVEMBER 5, 2019

File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution Aghayev et al., In this case, the assumption that a distributed storage backend should clearly be layered on top of a local file system. What is a distributed storage backend? SOSP’19. This is not surprising in hindsight.

Storage 64

Storage Systems Hardware Efficiency 64

n0derunner

JUNE 28, 2019

The post View from Nutanix storage during Postgres DB benchmark appeared first on n0derunner. Even though the log writes are sequential, they are low-concurrency and small size (looks like mostly 16K-32K). This write pattern is also a good candidate for oplog. These low-concurrency log writes also hit oplog.

Benchmarking 75

Benchmarking Storage Cache Database 75

Dynatrace

JANUARY 17, 2024

AI requires more compute and storage. Training AI data is resource-intensive and costly, again, because of increased computational and storage requirements. As a result, AI observability supports cloud FinOps efforts by identifying how AI adoption spikes costs because of increased usage of storage and compute resources.

Strategy 229

Strategy Artificial Intelligence Storage Cloud 229

Scalegrid

FEBRUARY 1, 2024

The biggest drawbacks are that a full backup can be time-consuming, and they require a significant amount of storage space. This backup type can save on storage space and reduce backup time by only capturing the changes made since the last backup. Therefore, it’s not the ideal choice if there are time or storage limitations.

Best Practices 130

Best Practices Storage Strategy Database 130

All Things Distributed

JANUARY 23, 2012

Expanding the Cloud - The AWS Storage Gateway. Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customersâ?? With the launch of the AWS Storage Gateway our customers can now integrate their on-premises IT environment with AWSâ??s

Storage 116

Storage AWS Virtualization Cloud 116

DZone

OCTOBER 24, 2023

Logstash: a log-processing tool that collects logs from various sources, parses them, and sends them to Elasticsearch for storage and analysis. Kibana: A powerful visualization tool that allows you to explore and analyze the data stored in Elasticsearch using interactive charts, graphs, and dashboards.

Analytics 255

Analytics Storage Infrastructure Scalability 255

DZone

MAY 3, 2023

In the previous posts, we covered things we had to do to upload files on the front end, things we had to do on the back end, and optimizing costs by moving file uploads to object storage.

Latency 183

Latency Speed Performance Storage 183