article thumbnail

Rebuilding Netflix Video Processing Pipeline with Microservices

The Netflix TechBlog

Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. The Netflix video processing pipeline went live with the launch of our streaming service in 2007. The Netflix video processing pipeline went live with the launch of our streaming service in 2007.

article thumbnail

Incremental Processing using Netflix Maestro and Apache Iceberg

The Netflix TechBlog

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

What are quality gates? How to use quality gates to deliver better software at speed and scale

Dynatrace

Open vulnerability on process group: The total number of currently high-profile vulnerabilities related to a process group. Vulnerability score: The highest vulnerability risk score for a process group. Quality gates after load/performance testing Teams can use quality gates to evaluate performance metrics.

Speed 216
article thumbnail

Migrating Critical Traffic At Scale with No Downtime?—?Part 1

The Netflix TechBlog

This blog post will provide a detailed analysis of replay traffic testing, a versatile technique we have applied in the preliminary validation phase for multiple migration initiatives. In this testing strategy, we execute a copy (replay) of production traffic against a system’s existing and new versions to perform relevant validations.

Traffic 339
article thumbnail

The Three Cs: Concatenate, Compress, Cache

CSS Wizardry

In this post, I’m going to break these processes down into each of: ? In one test, I concatenated it all into one big file, and the other had the library split into 12 files. Read the complete test methodology. 4,362ms of cumulative latency; 240ms of cumulative download. This will be referred to as css_time.

Cache 291
article thumbnail

Consistent caching mechanism in Titus Gateway

The Netflix TechBlog

We introduce a caching mechanism in the API gateway layer, allowing us to offload processing from singleton leader elected controllers without giving up strict data consistency and guarantees clients observe. cell): Titus Job Coordinator is a leader elected process managing the active state of the system.

Cache 224
article thumbnail

MezzFS?—?Mounting object storage in Netflix’s media processing platform

The Netflix TechBlog

—?Mounting object storage in Netflix’s media processing platform By Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. Encoding is not a one-time process?—?large We have one file?—?the

Media 214