The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

MARCH 4, 2024

Operational automation–including but not limited to, auto diagnosis, auto remediation, auto configuration, auto tuning, auto scaling, auto debugging, and auto testing–is key to the success of modern data platforms. Auto Remediation generates recommendations by considering both performance (i.e., Multi-objective optimizations.

Tuning 210

Tuning Efficiency Big Data Engineering 210

Dynatrace

JULY 6, 2022

While automating IT practices can save administrators a lot of time, without AIOps, the system is only as intelligent as the humans who program it. This requires significant data engineering efforts, as well as work to build machine-learning models. Monitoring automation is ongoing. Digital process automation tools.

Artificial Intelligence 198

Artificial Intelligence Tuning Strategy Big Data 198

The Netflix TechBlog

MARCH 5, 2019

As a micro-service owner, a Netflix engineer is responsible for its innovation as well as its operation, which includes making sure the service is reliable, secure, efficient and performant. In the Performance space, our data teams currently focus on the quality of experience on Netflix-enabled devices.

Infrastructure 182

Infrastructure Cloud Scalability AWS 182

The Netflix TechBlog

JANUARY 11, 2022

To learn about Analytics and Viz Engineering, have a look at Analytics at Netflix: Who We Are and What We Do by Molly Jackman & Meghana Reddy and How Our Paths Brought Us to Data and Netflix by Julie Beckley & Chris Pham. Curious to learn about what it’s like to be a Data Engineer at Netflix?

Innovation 243

Innovation Metrics Engineering Testing 243

The Netflix TechBlog

OCTOBER 18, 2022

Due to its popularity, the number of workflows managed by the system has grown exponentially. The scheduler on-call has to closely monitor the system during non-business hours. As the usage increased, we had to vertically scale the system to keep up and were approaching AWS instance type limits.

Java 202

Java Scalability Traffic Architecture 202

The Netflix TechBlog

SEPTEMBER 10, 2019

You can look at ABlaze (our centralized A/B testing platform) and take a quick look at how it’s performing. Note that the new encodes perform well in the lower quantiles but worse in the higher ones You notice that the first new encode (Cell 2?—?Encode Simulated dataset that shows what the distribution of play delay may look like.

Metrics 215

Metrics Architecture Infrastructure Innovation 215

O'Reilly

FEBRUARY 18, 2020

Infrastructure and ops usage was the fastest growing sub-topic under the generic systems administration topic. DevOps aims to produce programmers who can work competently in each of the layers in a system “ stack.” The results for data-related topics are both predictable and—there’s no other way to put it—confusing.

Software Architecture 132

Software Architecture DevOps Data Engineering Architecture 132

The Netflix TechBlog

NOVEMBER 22, 2019

December 2 1pm-2pm CMP 326-R Capacity Management Made Easy with Amazon EC2 Auto Scaling Vadim Filanovsky , Senior Performance Engineer & Anoop Kapoor, AWS Abstract :Amazon EC2 Auto Scaling offers a hands-free capacity management experience to help customers maintain a healthy fleet, improve application availability, and reduce costs.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

December 2 1pm-2pm CMP 326-R Capacity Management Made Easy with Amazon EC2 Auto Scaling Vadim Filanovsky , Senior Performance Engineer & Anoop Kapoor, AWS Abstract :Amazon EC2 Auto Scaling offers a hands-free capacity management experience to help customers maintain a healthy fleet, improve application availability, and reduce costs.

AWS 100

AWS Entertainment Open Source Benchmarking 100

High Scalability

APRIL 14, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! Who's Hiring? InterviewCamp.io Try out their platform. Cool Products and Services.

Education 105

Education Software Engineering Scalability Engineering 105

High Scalability

MAY 12, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! Who's Hiring? InterviewCamp.io Try out their platform. Cool Products and Services.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

APRIL 28, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! Who's Hiring? InterviewCamp.io Try out their platform. Cool Products and Services.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

MARCH 30, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! Who's Hiring? InterviewCamp.io Try out their platform. Advertise your job here!

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

MAY 26, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! Created by former senior-level AWS engineers of 15 years. Who's Hiring?

Education 100

Education AWS Software Engineering Scalability 100

High Scalability

MARCH 24, 2020

Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc.

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

MARCH 17, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 105

Education Engineering Java Software Engineering 105

High Scalability

MARCH 3, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 114

Education Engineering Games Java 114

High Scalability

FEBRUARY 18, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 113

Education Engineering Games Java 113

High Scalability

FEBRUARY 9, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 108

Education Engineering Games Java 108

InfoQ

JULY 20, 2023

Microsoft recently announced the general availability (GA) of Azure Managed Lustre, a managed file system for high-performance computing (HPC) and AI workloads. By Steef-Jan Wiggers

Azure 40

Azure Availability Systems Performance 40

Abhishek Tiwari

DECEMBER 27, 2017

Unified data management architecture. A unified data management (UDM) system combines the best of data warehouses, data lakes, and streaming without expensive and error-prone ETL. In contrast, Alluxio a middleware for data access - think Alluxio storage layer as fast cache.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

O'Reilly

FEBRUARY 2, 2020

The same is true for the models themselves; if you don’t have the artifacts you produced, you won’t be able to make statements about how they performed. Given source code and the training data, you could re-produce a model, but it almost certainly wouldn’t be the same because of randomization in the training process.

Software Architecture 67

Software Architecture Monitoring Software Engineering Architecture 67

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Performance.

Big Data 63

Big Data Storage Benchmarking Hardware 63

Adrian Cockcroft

FEBRUARY 13, 2023

Discussion of the water positive goal and optimizations that let AWS return water directly for irrigation rather than to the sewage system. STP213 Scaling global carbon footprint management — Blake Blackwell Persefoni Manager Data Engineering and Michael Floyd AWS Head of Sustainability Solutions.

AWS 64

AWS Energy Architecture Programming 64

Abhishek Tiwari

JULY 12, 2018

Unfortunately, building data pipelines remains a daunting, time-consuming, and costly activity. Not everyone is operating at Netflix or Spotify scale data engineering function. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines.

Latency 63

Latency Analytics Scalability Engineering 63

The Netflix TechBlog

MAY 26, 2020

The enriched data allows us to analyze networks across a variety of dimensions (e.g. availability, performance, and security), to ensure applications can effectively deliver their data payload across a globally dispersed cloud-based ecosystem.

Network 150

Network Tuning AWS Big Data 150

The Netflix TechBlog

NOVEMBER 22, 2019

December 2 1pm-2pm CMP 326-R Capacity Management Made Easy with Amazon EC2 Auto Scaling Vadim Filanovsky , Senior Performance Engineer & Anoop Kapoor, AWS Abstract :Amazon EC2 Auto Scaling offers a hands-free capacity management experience to help customers maintain a healthy fleet, improve application availability, and reduce costs.

AWS 37

AWS Entertainment Open Source Benchmarking 37

Abhishek Tiwari

JULY 20, 2019

Store engineering squad - focus on software and systems required for the storefront including point-of-sales system, promotions, etc. ERP engineering squad - supply chain planning, purchase order management, product lifecycle management, merchandise planning, etc. How is that even possible? Probably yes.

Engineering 61

Engineering Retail Airlines Healthcare 61

High Scalability

NOVEMBER 12, 2019

Grokking the System Design Interview is a popular course on Educative.io (taken by 20,000+ people) that's widely considered the best System Design interview resource on the Internet. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes.

Education 22

Education Java Software Engineering Engineering 22

High Scalability

OCTOBER 29, 2019

Grokking the System Design Interview is a popular course on Educative.io (taken by 20,000+ people) that's widely considered the best System Design interview resource on the Internet. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes.

Education 22

Education Java Software Engineering Engineering 22

High Scalability

JANUARY 7, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 21

Education Java Software Engineering Engineering 21

High Scalability

DECEMBER 12, 2019

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 21

Education Java Software Engineering Engineering 21

High Scalability

JANUARY 22, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 22

Education Java Software Engineering Engineering 22

Testsigma

NOVEMBER 28, 2019

The best courses relevant to Automation Testing on Edureka are: Selenium Certification Training Performance Testing using Jmeter Ruby with Cucumber Certification Testing. They have an affiliation with universities like MIT, Harvard University, Boston University, The University of Texas System and more. Now that’s something cool!

Website 53

Website Testing Programming Automotive 53

High Scalability

FEBRUARY 6, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 21

Education Java Software Engineering Engineering 21