article thumbnail

Article: Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System

InfoQ

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud. By Facundo Agriel

Storage 111
article thumbnail

The Block Allocation Policy of Virtual Distributed File System at the Source Code Level

DZone

Users can allocate different storage tiers as the resources for Alluxio workers, including MEM/SSD/HDD, which are further composed of directories. In this article, we analyze the policies of block allocation from the source code.

Code 246
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Leveraging Infrastructure as Code for Data Engineering Projects: A Comprehensive Guide

DZone

Data engineering projects often require the setup and management of complex infrastructures that support data processing, storage, and analysis. In this article, we will explore the benefits of leveraging IaC for data engineering projects and provide detailed implementation steps to get started.

article thumbnail

Low Overhead Continuous Contextual Production Profiling

DZone

However, it does not provide visibility into the operations taking place at the code level, such as method, socket, and thread states. Moreover, the process of collecting these profiles introduces overhead during application runtime and necessitates the storage and visualization of significantly large datasets.

Latency 253
article thumbnail

Medallion Architecture: Efficient Batch and Stream Processing Data Pipelines With Azure Databricks and Delta Lake

DZone

This article explores the concepts of Medallion Architecture and demonstrates how to implement batch and stream processing pipelines using Azure Databricks and Delta Lake. Detailed code samples and explanations will be provided to illustrate each implementation step.

Azure 246
article thumbnail

Why growing AI adoption requires an AI observability strategy

Dynatrace

And an O’Reilly Media survey indicated that two-thirds of survey respondents have already adopted generative AI —a form of AI that uses training data to create text, images, code, or other types of content that reflect its users’ natural language queries. AI requires more compute and storage. AI performs frequent data transfers.

Strategy 226
article thumbnail

Redis vs Memcached in 2024

Scalegrid

This article will explore how they handle data storage and scalability, perform in different scenarios, and, most importantly, how these factors influence your choice. It uses a hash table to manage these pairs, divided into fixed-size buckets with linked lists for key-value storage.

Cache 130