article thumbnail

Data Engineers of Netflix?—?Interview with Pallavi Phadnis

The Netflix TechBlog

Data Engineers of Netflix?—?Interview Interview with Pallavi Phadnis This post is part of our “ Data Engineers of Netflix ” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Pallavi Phadnis is a Senior Software Engineer at Netflix.

article thumbnail

3. Psyberg: Automated end to end catch up

The Netflix TechBlog

Audit Run various quality checks on the staged data. Several audits, such as verifying source and target counts, are performed on this batch of data. Publish If the audits are successful, cherry-pick the staging snapshot to publish the data to production.

Tuning 244
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Experimentation is a major focus of Data Science across Netflix

The Netflix TechBlog

Here we describe the role of Experimentation and A/B testing within the larger Data Science and Engineering organization at Netflix, including how our platform investments support running tests at scale while enabling innovation. Curious to learn about what it’s like to be a Data Engineer at Netflix?

article thumbnail

How Data Inspires Building a Scalable, Resilient and Secure Cloud Infrastructure At Netflix

The Netflix TechBlog

Netflix’s engineering culture is predicated on Freedom & Responsibility, the idea that everyone (and every team) at Netflix is entrusted with a core responsibility and they are free to operate with freedom to satisfy their mission. All these micro-services are currently operated in AWS cloud infrastructure.

article thumbnail

Reimagining Experimentation Analysis at Netflix

The Netflix TechBlog

You can look at ABlaze (our centralized A/B testing platform) and take a quick look at how it’s performing. Note that the new encodes perform well in the lower quantiles but worse in the higher ones You notice that the first new encode (Cell 2?—?Encode Simulated dataset that shows what the distribution of play delay may look like.

Metrics 215
article thumbnail

What is IT automation?

Dynatrace

This requires significant data engineering efforts, as well as work to build machine-learning models. An IT automation strategy should start by breaking down the workflow, the types of operations they will perform, and how teams can best monitor and optimize them in production. The goal of automation is to reduce IT complexity.

article thumbnail

5 key areas for tech leaders to watch in 2020

O'Reilly

The results for data-related topics are both predictable and—there’s no other way to put it—confusing. Starting with data engineering, the backbone of all data work (the category includes titles covering data management, i.e., relational databases, Spark, Hadoop, SQL, NoSQL, etc.). This follows a 3% drop in 2018.