article thumbnail

Medallion Architecture: Efficient Batch and Stream Processing Data Pipelines With Azure Databricks and Delta Lake

DZone

Medallion Architecture provides a framework for organizing data processing workflows into different zones, enabling optimized batch and stream processing. This article explores the concepts of Medallion Architecture and demonstrates how to implement batch and stream processing pipelines using Azure Databricks and Delta Lake.

Azure 246
article thumbnail

Article: Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System

InfoQ

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud. By Facundo Agriel

Storage 111
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Pioneering customer-centric pricing models: Decoding ingest-centric vs. answer-centric pricing

Dynatrace

The rise of cloud-native microservice architectures further exacerbates this change. Dynatrace has developed the purpose-built data lakehouse, Grail , eliminating the need for separate management of indexes and storage. All data is readily accessible without storage tiers, such as costly solid-state drives (SSDs).

Retail 237
article thumbnail

Serverless Reference Architectures with AWS Lambda

All Things Distributed

If you are looking for more examples there are the Lambda Serverless Reference Architectures that can serve as the blueprint for building your own serverless applications. Mobile Backend Serverless Reference Architecture. Real-time File Processing Serverless Reference Architecture. IoT Backend Serverless Reference Architecture.

Lambda 107
article thumbnail

Tailored access management, Part 2: Onboard users to Grail and AppEngine

Dynatrace

The new architecture enables more granularity in permission management and provides the dynamics necessary to serve modern access management use cases. ALLOW storage:system:read; The Storage All System Data Read policy grants access to Dynatrace internal data such as auditing events and query execution events.

Storage 209
article thumbnail

Boost DevOps maturity with observability and a data lakehouse

Dynatrace

Research has found that 99% of organizations have embraced a multicloud architecture. When data storage strategies become problematic to DevOps maturity Data warehouse-based approaches add cost and time to analytics projects. Cloud complexity leads to data silos Most organizations are battling cloud complexity.

DevOps 184
article thumbnail

Serverless Reference Architecture with AWS Lambda

All Things Distributed

If you are looking for more examples there are the Lambda Serverless Reference Architectures that can serve as the blueprint for building your own serverless applications. Mobile Backend Serverless Reference Architecture. Real-time File Processing Serverless Reference Architecture. IoT Backend Serverless Reference Architecture.

Lambda 105