article thumbnail

Rebuilding Netflix Video Processing Pipeline with Microservices

The Netflix TechBlog

Since then, the video pipeline has undergone substantial improvements and broad expansions: Starting with Standard Dynamic Range (SDR) at Standard-Definitions , we expanded the encoding pipeline to 4K and High Dynamic Range (HDR) which enabled support for our premium offering. The requests from the studio side are generally latency-sensitive.

article thumbnail

Evolving from Rule-based Classifier: Machine Learning Powered Auto Remediation in Netflix Data…

The Netflix TechBlog

We have deployed Auto Remediation in production for handling memory configuration errors and unclassified errors of Spark jobs and observed its efficiency and effectiveness (e.g., For efficient error handling, Netflix developed an error classification service, called Pensive, which leverages a rule-based classifier for error classification.

Tuning 210
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Practical API Design at Netflix, Part 1: Using Protobuf FieldMask

The Netflix TechBlog

Remote calls are never free; they impose extra latency, increase probability of an error, and consume network bandwidth. By default, gRPC uses protobuf as its IDL (interface definition language) and data serialization protocol. Our protobuf message definition (.proto FieldMask is a protobuf message. Field names are not included.

Design 245
article thumbnail

Automated observability, security, and reliability at scale

Dynatrace

However, scaling up software development requires more tools along the software product lifecycle, which must be configured promptly and efficiently. Efficient environment configuration at scale One of software engineers’ most significant challenges is managing the numerous tools and technologies required for the software product lifecycle.

article thumbnail

What is a data lakehouse? Combining data lakes and warehouses for the best of both worlds

Dynatrace

While data lakes and data warehousing architectures are commonly used modes for storing and analyzing data, a data lakehouse is an efficient third way to store and analyze data that unifies the two architectures while preserving the benefits of both. This data lands in its original, raw form without requiring schema definition.

article thumbnail

Data Movement in Netflix Studio via Data Mesh

The Netflix TechBlog

Operational Reporting is a reporting paradigm specialized in covering high-resolution, low-latency data sets, serving detailed day-to-day activities¹ and processes of a business domain. Most of the business views created on top of the Iceberg tables can tolerate a few minutes of latency. The audits check for equality (i.e.

Big Data 253
article thumbnail

Migrating Critical Traffic At Scale with No Downtime?—?Part 2

The Netflix TechBlog

Migrating Critical Traffic At Scale with No Downtime — Part 2 Shyam Gala , Javier Fernandez-Ivern , Anup Rokkam Pratap , Devang Shah Picture yourself enthralled by the latest episode of your beloved Netflix series, delighting in an uninterrupted, high-definition streaming experience.

Traffic 279