The Netflix TechBlog

Optimized shot-based encodes for 4K: Now streaming!

The Netflix TechBlog

by Aditya Mavlankar , Liwei Guo , Anush Moorthy and Anne Aaron Netflix has an ever-expanding collection of titles which customers can enjoy in 4K resolution with a suitable device and subscription plan.

Building Netflix’s Distributed Tracing Infrastructure

The Netflix TechBlog

by Maulik Pandey Our Team?—? Kevin Lew , Narayanan Arunachalam , Elizabeth Carretto , Dustin Haffner , Andrei Ushakov, Seth Katz , Greg Burrell , Ram Vaithilingam , Mike Smith and Maulik Pandey “ @Netflixhelps Why doesn’t Tiger King play on my phone?”?—?a

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Edgar: Solving Mysteries Faster with Observability

The Netflix TechBlog

Edgar helps Netflix teams troubleshoot distributed systems efficiently with the help of a summarized presentation of request tracing, logs, analysis, and metadata. by Elizabeth Carretto Everyone loves Unsolved Mysteries. There’s always someone who seems like the surefire culprit.

Seamlessly Swapping the API backend of the Netflix Android app

The Netflix TechBlog

How we migrated our Android endpoints out of a monolith into a new microservice by Rohan Dhruva , Ed Ballot As Android developers, we usually have the luxury of treating our backends as magic boxes running in the cloud, faithfully returning us JSON.

Computational Causal Inference at Netflix

The Netflix TechBlog

Jeffrey Wong , Colin McFarland Every Netflix data scientist, whether their background is from biology, psychology, physics, economics, math, statistics, or biostatistics, has made meaningful contributions to the way Netflix analyzes causal effects.

Analytics at Netflix: Who we are and what we do

The Netflix TechBlog

Analytics at Netflix: Who We Are and What We Do An Introduction to Analytics and Visualization Engineering at Netflix by Molly Jackman & Meghana Reddy Explained: Season 1 (Photo Credit: Netflix) Across nearly every industry, there is recognition that data analytics is key to driving informed business decision-making.

How Our Paths Brought Us to Data and Netflix

The Netflix TechBlog

Part of our series on who works in Analytics at Netflix?—?and

Key Challenges with Quasi Experiments at Netflix

The Netflix TechBlog

Kamer Toker-Yildiz , Colin McFarland , Julia Glick At Netflix, when we can’t run A/B experiments we run quasi experiments !

Machine Learning for a Better Developer Experience

The Netflix TechBlog

Stanislav Kirdey , William High Imagine having to go through 2.5GB of log entries from a failed software build?—?3 3 million lines?—?to to search for a bug or a regression that happened on line 1M. It’s probably not even doable manually!

HDMI?—?Scaling Netflix Certification

The Netflix TechBlog

HDMI?—?Scaling Scaling Netflix Certification Scott Bolter , Matthew Lehman , Akshay Garg ¹ At Netflix, we take the task of preserving the creative vision of our content all the way to a subscriber TV screen very seriously.

Media 146

Improving our video encodes for legacy devices

The Netflix TechBlog

by Mariana Afonso , Anush Moorthy , Liwei Guo , Lishan Zhu , Anne Aaron Netflix has been one of the pioneers of streaming video-on-demand content?—?we we announced our intention to stream video over 13 years ago, in January 2007?—?and

Empowering the Visual Effects Community with the NetFX Platform

The Netflix TechBlog

The cloud-based platform allows vendors, artists and creators to connect and collaborate on visual effects (VFX) from anywhere in the… Continue reading on Netflix TechBlog ». visual-effects

Cloud 159

Unbundling Data Science Workflows with Metaflow and AWS Step Functions

The Netflix TechBlog

by David Berg, Ravi Kiran Chirravuri, Romain Cledat, Jason Ge, Savin Goyal, Ferras Hamad, Ville Tuulos Continue reading on Netflix TechBlog ». infrastructure aws machine-learning

AWS 149

GraphQL Search Indexing

The Netflix TechBlog

by Artem Shtatnov and Ravi Srinivas Ranganathan Almost a year ago we described our learnings from adopting GraphQL on the Netflix Marketing Tech team. We have a lot more to share since then! There are plenty of existing resources describing how to express a search query in GraphQL and paginate the results. This post looks at the other side of search: how to index data and make it searchable.

Byte Down: Making Netflix’s Data Infrastructure Cost-Effective

The Netflix TechBlog

By Torio Risianto, Bhargavi Reddy, Tanvi Sahni, Andrew Park Continue reading on Netflix TechBlog ». cloud-storage data data-infrastructure aws netflix

Open-sourcing Polynote: an IDE-inspired polyglot notebook

The Netflix TechBlog

Jeremy Smith , Jonathan Indig , Faisal Siddiqi We are pleased to announce the open-source launch of Polynote : a new, polyglot notebook with first-class Scala support, Apache Spark integration, multi-language interoperability including Scala, Python, and SQL, as-you-type autocomplete, and more. Polynote provides data scientists and machine learning researchers with a notebook environment that allows them the freedom to seamlessly integrate our JVM-based ML platform ?—?which

Python at Netflix

The Netflix TechBlog

By Pythonistas at Netflix, coordinated by Amjith Ramanujam and edited by Ellen Livengood As many of us prepare to go to PyCon, we wanted to share a sampling of how Python is used at Netflix. We use Python through the full content lifecycle, from deciding which content to fund all the way to operating the CDN that serves the final video to 148 million members. We use and contribute to many open-source Python packages, some of which are mentioned below.

How Netflix brings safer and faster streaming experience to the living room on crowded networks…

The Netflix TechBlog

How Netflix brings safer and faster streaming experience to the living room on crowded networks using TLS 1.3 By Sekwon Choi At Netflix, we are obsessed with the best streaming experiences. We want playback to start instantly and to never stop unexpectedly in any network environment. We are also committed to protecting users’ privacy and service security without sacrificing any part of the playback experience.

Predictive CPU isolation of containers at Netflix

The Netflix TechBlog

By Benoit Rostykus, Gabriel Hartmann Noisy Neighbors We’ve all had noisy neighbors at one point in our life. Whether it’s at a cafe or through a wall of an apartment, it is always disruptive. The need for good manners in shared spaces turns out to be important not just for people, but for your Docker containers too. When you’re running in the cloud your containers are in a shared space; in particular they share the CPU’s memory hierarchy of the host instance.

Cache 212

Bringing 4K and HDR to Anime at Netflix with Sol Levante

The Netflix TechBlog

By Haruka Miyagawa & Kylee Peña Continue reading on Netflix TechBlog ». 4k netflix anime animation hdr

159
159

Engineering a Studio Quality Experience With High-Quality Audio at Netflix

The Netflix TechBlog

by Guillaume du Pontavice, Phill Williams and Kylee Peña (on behalf of our Streaming Algorithms, Audio Algorithms, and Creative Technologies teams) Remember the epic opening sequence of Stranger Things 2 ? The thrill of that car chase through Pittsburgh not only introduced a whole new set of mysteries, but it returned us to a beloved and dangerous world alongside Dustin, Lucas, Mike, Will and Eleven.

Ready for changes with Hexagonal Architecture

The Netflix TechBlog

by Damir Svrtan and Sergii Makagon As the production of Netflix Originals grows each year, so does our need to build apps that enable efficiency throughout the entire creative process. Our wider Studio Engineering Organization has built more than 30 apps that help content progress from pitch (aka screenplay) to playback: ranging from script content acquisition, deal negotiations and vendor management to scheduling, streamlining production workflows, and so on.

How Netflix uses Druid for Real-time Insights to Ensure a High-Quality Experience

The Netflix TechBlog

By Ben Sykes Continue reading on Netflix TechBlog ». apache realtime druid metrics-and-analytics kafka

Reimagining Experimentation Analysis at Netflix

The Netflix TechBlog

Toby Mao , Sri Sri Perangur , Colin McFarland Another day, another custom script to analyze an A/B test. Maybe you’ve done this before and have an old script lying around. If it’s new, it’s probably going to take some time to set up, right? Not at Netflix.

Delta: A Data Synchronization and Enrichment Platform

The Netflix TechBlog

Part I: Overview Andreas Andreakis , Falguni Jhaveri , Ioannis Papapanagiotou , Mark Cho , Poorna Reddy , Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), providing advanced search capabilities (ElasticSearch etc.), caching (Memcached etc.), and more.

Introducing Dispatch

The Netflix TechBlog

By Kevin Glisson, Marc Vilanova, Forest Monsen Netflix is pleased to announce the open-source release of our crisis management orchestration framework: Dispatch! Okay, but what is Dispatch? Put simply, Dispatch is: All of the ad-hoc things you’re doing to manage incidents today, done for you, and a bunch of other things you should’ve been doing, but have not had the time!

AWS 149

Applying Netflix DevOps Patterns to Windows

The Netflix TechBlog

Baking Windows with Packer By Justin Phelps and Manuel Correa Customizing Windows images at Netflix was a manual, error-prone, and time consuming process. In this blog post, we describe how we improved the methodology, which technologies we leveraged, and how this has improved service deployment and consistency.

DevOps 178

Netflix Now Streaming AV1 on Android

The Netflix TechBlog

By Liwei Guo , Vivian Li , Julie Beckley , Venkatesh Selvaraj , and Jeff Watts Today we are excited to announce that Netflix has started streaming AV1 to our Android mobile app. AV1 is a high performance, royalty-free video codec that provides 20% improved compression efficiency over our VP9† encodes. AV1 is made possible by the wide-ranging industry commitment of expertise and intellectual property within the Alliance for Open Media (AOMedia), of which Netflix is a founding member.

Media 151

Netflix Now Streaming AV1 on Android

The Netflix TechBlog

By Liwei Guo , Vivian Li , Julie Beckley , Venkatesh Selvaraj , and Jeff Watts Today we are excited to announce that Netflix has started streaming AV1 to our Android mobile app. AV1 is a high performance, royalty-free video codec that provides 20% improved compression efficiency over our VP9† encodes. AV1 is made possible by the wide-ranging industry commitment of expertise and intellectual property within the Alliance for Open Media (AOMedia), of which Netflix is a founding member.

Media 151

How Netflix microservices tackle dataset pub-sub

The Netflix TechBlog

By Ammar Khaku Introduction In a microservice architecture such as Netflix’s, propagating datasets from a single source to multiple downstream destinations can be challenging. These datasets can represent anything from service configuration to the results of a batch job, are often needed in-memory to optimize access and must be updated as they change over time. One example displaying the need for dataset propagation: at any given time Netflix runs a very large number of A/B tests.

Cache 167

Open-Sourcing Metaflow, a Human-Centric Framework for Data Science

The Netflix TechBlog

by David Berg , Ravi Kiran Chirravuri , Romain Cledat , Savin Goyal , Ferras Hamad , Ville Tuulos tl;dr Metaflow is now open-source! Get started at metaflow.org. Netflix applies data science to hundreds of use cases across the company, including optimizing content delivery and video encoding. Data scientists at Netflix relish our culture that empowers them to work autonomously and use their judgment to solve problems independently.

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

The Netflix TechBlog

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can Can I run a check myself to understand what data is behind this metric?”

Evolving Regional Evacuation

The Netflix TechBlog

Niosha Behnam | Demand Engineering @ Netflix At Netflix we prioritize innovation and velocity in pursuit of the best experience for our 150+ million global customers. This means that our microservices constantly evolve and change, but what doesn’t change is our responsibility to provide a highly available service that delivers 100+ million hours of daily streaming to our subscribers.

Essential Suite?—?Artwork Producer Assistant

The Netflix TechBlog

Essential Suite?—?Artwork Artwork Producer Assistant By: Hamid Shahid & Syed Haq Introduction Netflix continues to invest in content for a global audience with a diverse range of unique tastes and interests. Correspondingly, the member experience must also evolve to connect this global audience to the content that most appeals to each of them.

Design 138

DBLog: A Generic Change-Data-Capture Framework

The Netflix TechBlog

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4].

Netflix Hack Day?—?November 2019

The Netflix TechBlog

Netflix Hack Day?—?Fall Fall 2019 By Tom Richards , Carenina Garcia Motion , and Leslie Posada Hack Day at Netflix is an opportunity to build and show off a feature, tool, or quirky app. The goal is simple: experiment with new ideas/technologies, engage with colleagues across different disciplines, and have fun! We know even the silliest idea can spur something more. The most important value of our Hack Days is that they support a culture of innovation.

Open-Sourcing riskquant, a library for quantifying risk

The Netflix TechBlog

Netflix has a program in our Information Security department for quantifying the risk of deliberate (attacker-driven) and accidental… Continue reading on Netflix TechBlog ». security risk information-security

DBLog: A Generic Change-Data-Capture Framework

The Netflix TechBlog

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4].

Extending Vector with eBPF to inspect host and container performance

The Netflix TechBlog

by Jason Koch , with Martin Spier , Brendan Gregg , Ed Hunter Improving the tools available to our engineers to help them diagnose, triage, and work through software performance challenges in the cloud is a key goal for the cloud performance engineering team at Netflix. Today we are excited to announce latency heatmaps and improved container support for our on-host monitoring solution?—?Vector?—?to Vector?—?to to the broader community. Vector is open source and in use by multiple companies.

Evolution of Netflix Conductor:

The Netflix TechBlog

and beyond By Anoop Panicker and Kishore Banala Conductor is a workflow orchestration engine developed and open-sourced by Netflix. If you’re new to Conductor, this earlier blogpost and the documentation should help you get started and acclimatized to Conductor. Netflix Conductor: A microservices orchestrator In the last two years since inception, Conductor has seen wide adoption and is instrumental in running numerous core workflows at Netflix.

Lambda 163

Netflix Studio Engineering Overview

The Netflix TechBlog

By Steve Urban , Sridhar Seetharaman , Shilpa Motukuri , Tom Mack , Erik Strauss , Hema Kannan , CJ Barker Netflix is revolutionizing the way a modern studio operates. Our mission in Studio Engineering is to build a unified, global, and digital studio that powers the effective production of amazing content. link] Netflix produces some of the world’s most beloved and award-winning films and series, including The Irishman, The Crown, La Casa de Papel, Ozark, and Tiger King.

Spinnaker Sets Sail to the Continuous Delivery Foundation

The Netflix TechBlog

Author: Andy Glover Since releasing Spinnaker to the open source community in 2015 , the platform has flourished with the addition of new cloud providers, triggers, pipeline stages, and much more. Myriad new features, improvements, and innovations have been added by an ever growing, actively engaged community. Each new innovation has been a step towards an even better Continuous Delivery platform that facilitates rapid, reliable, safe delivery of flexible assets to pluggable deployment targets.

Hyper Scale VPC Flow Logs enrichment to provide Network Insight

The Netflix TechBlog

How Netflix is able to enrich VPC Flow Logs at Hyper Scale to provide Network Insight By Hariharan Ananthakrishnan and Angela Ho The Cloud Network Infrastructure that Netflix utilizes today is a large distributed ecosystem that consists of specialized functional tiers and services such as DirectConnect, VPC Peering, Transit Gateways, NAT Gateways, etc.