article thumbnail

Optimizing data warehouse storage

The Netflix TechBlog

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203
article thumbnail

Netflix Cloud Packaging in the Terabyte Era

The Netflix TechBlog

Our previous tech blog Packaging award-winning shows with award-winning technology detailed our packaging technology deployed on the streaming side. From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step.

Cloud 237
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Migrating Critical Traffic At Scale with No Downtime?—?Part 1

The Netflix TechBlog

This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal. This blog post will provide a detailed analysis of replay traffic testing, a versatile technique we have applied in the preliminary validation phase for multiple migration initiatives.

Traffic 339
article thumbnail

Mayastor: Lightning Fast Storage for Kubernetes

Percona Community

In this blog post we’re going to see those technologies at work to give us awesome block storage performance with flexibility and simple operations. It’s a new generation in storage software, designed for super high speed low latency NVMe devices. Why is SPDK exciting?

Storage 52
article thumbnail

Uber’s Big Data Platform: 100+ Petabytes with Minute Latency

Uber Engineering

To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog.

Big Data 109
article thumbnail

Improved Alerting with Atlas Streaming Eval

The Netflix TechBlog

While Atlas is architected around compute & storage separation, and we could theoretically just scale the query layer to meet the increased query demand, every query, regardless of its type, has a data component that needs to be pushed down to the storage layer. The internals here are outside the scope of this blog post.

Storage 288
article thumbnail

Reducing Your Database Hosting Costs: DigitalOcean vs. AWS vs. Azure

Scalegrid

Since database hosting is more dependent on memory (RAM) than storage, we are going to compare various instance sizes ranging from just 1GB of RAM up to 64GB of RAM so you can see how costs vary across different application workloads. Does it affect latency? Yes, you can see an increase in latency. EC2 instances. VM instances.

Azure 344