The Netflix TechBlog

JANUARY 11, 2022

Curious to learn about what it’s like to be a Data Engineer at Netflix? Hear directly from Samuel Setegne , Dhevi Rajendran , Kevin Wylie , and Pallavi Phadnis in our “Data Engineers of Netflix” interview series. In advertising, the treatments (the ads that we purchase) have a direct monetary cost to Netflix.

Innovation 243

Innovation Metrics Engineering Testing 243

Dynatrace

JULY 6, 2022

Developing automation takes time. While automating IT practices can save administrators a lot of time, without AIOps, the system is only as intelligent as the humans who program it. This requires significant data engineering efforts, as well as work to build machine-learning models. Monitoring automation is ongoing.

Artificial Intelligence 194

Artificial Intelligence Tuning Strategy Big Data 194

The Netflix TechBlog

JULY 21, 2022

Since memory management is not something one usually associates with classification problems, this blog focuses on formulating the problem as an ML problem and the data engineering that goes along with it. for us at Netflix, this is a combination of the device type, app session ID and software development kit version (SDK version).

Big Data 179

Big Data Cache Engineering Data Engineering 179

The Netflix TechBlog

MARCH 5, 2019

As a micro-service owner, a Netflix engineer is responsible for its innovation as well as its operation, which includes making sure the service is reliable, secure, efficient and performant. How can we develop templated detection modules (rules- and ML-based) and data streams to increases speed of development?

Infrastructure 182

Infrastructure Cloud Scalability AWS 182

The Netflix TechBlog

MARCH 25, 2019

Finally, imagine yourself in the role of a data platform reliability engineer tasked with providing advanced lead time to data pipeline (ETL) owners by proactively identifying issues upstream to their ETL jobs. Let’s review a few of these principles: Ensure data integrity ?—?Accurately Design a flexible data model ?

Infrastructure 219

Infrastructure Big Data Transportation Architecture 219

O'Reilly

FEBRUARY 18, 2020

There’s plenty of security risks for business executives, sysadmins, DBAs, developers, etc., The laggard use case was Python-based web development frameworks, which grew by just 3% in usage, year over year. there’s a Python library for virtually anything a developer or data scientist might need to do. to be wary of.

Software Architecture 132

Software Architecture DevOps Data Engineering Architecture 132

The Netflix TechBlog

DECEMBER 21, 2020

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. Use cases We found several use cases where a system like AutoOptimize can bring tons of value.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

SEPTEMBER 10, 2019

Our data scientists faced numerous challenges in our previous infrastructure. Complex business logic was embedded directly into the ETL pipelines by data engineers. In order to replicate results, scientists had to delve deep into the data, code, and documentation.

Metrics 215

Metrics Architecture Infrastructure Innovation 215

High Scalability

APRIL 14, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! this is going to be a challenging journey for any backend engineer! Who's Hiring?

Education 105

Education Software Engineering Scalability Engineering 105

High Scalability

MAY 12, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! this is going to be a challenging journey for any backend engineer! Who's Hiring?

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

APRIL 28, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! this is going to be a challenging journey for any backend engineer! Who's Hiring?

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

MAY 26, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! this is going to be a challenging journey for any backend engineer! Who's Hiring?

Education 100

Education AWS Software Engineering Scalability 100

The Netflix TechBlog

MARCH 4, 2024

Given the extensive scope and intricate complexity inherent to such a distributed, large-scale system, even if the failed jobs account for a tiny portion of the total workload, diagnosing and remediating job failures can cause considerable operational burdens. This work would have not been possible without the solid, in-depth collaborations.

Tuning 210

Tuning Efficiency Big Data Engineering 210

High Scalability

MARCH 30, 2020

has hours of system design content. They also do live system design discussions every week. Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). this is going to be a challenging journey for any backend engineer! this is going to be a challenging journey for any backend engineer! Who's Hiring?

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

MARCH 24, 2020

Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc.

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

MARCH 17, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 105

Education Engineering Java Software Engineering 105

The Netflix TechBlog

NOVEMBER 22, 2019

4:45pm-5:45pm NFX 202 A day in the life of a Netflix Engineer Dave Hahn , SRE Engineering Manager Abstract : Netflix is a large, ever-changing ecosystem serving millions of customers across the globe through cloud-based systems and a globally distributed CDN.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

4:45pm-5:45pm NFX 202 A day in the life of a Netflix Engineer Dave Hahn , SRE Engineering Manager Abstract : Netflix is a large, ever-changing ecosystem serving millions of customers across the globe through cloud-based systems and a globally distributed CDN.

AWS 100

AWS Entertainment Open Source Benchmarking 100

High Scalability

MARCH 3, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 114

Education Engineering Games Java 114

High Scalability

FEBRUARY 18, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 113

Education Engineering Games Java 113

High Scalability

FEBRUARY 9, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 108

Education Engineering Games Java 108

The Agile Manager

JULY 31, 2022

There are shadow IT teams of developers or data engineers that spring up in areas like operations or marketing because the captive IT function is slow, if not outright incapable, of responding to internal customer demand. IT shadows appear in a lot of different forms. The scope taken out of the 1.0

Programming 59

Programming Engineering Data Engineering Innovation 59

O'Reilly

JULY 15, 2020

Almost one-third (29%) of respondents say their employers are migrating or implementing a majority of their systems (over 50%) using microservices. Software engineers comprise the survey audience’s single largest cluster, over one quarter (27%) of respondents (Figure 1). Adopters are betting big on microservices.

Database 134

Database Architecture Education Systems 134

O'Reilly

FEBRUARY 2, 2020

Source code is relatively less important compared to typical applications; the training data is what determines how the model behaves, and the training process is all about tweaking parameters in the application so that it delivers correct results most of the time. You need a repository for models and for the training data.

Software Architecture 67

Software Architecture Monitoring Software Engineering Architecture 67

Adrian Cockcroft

FEBRUARY 13, 2023

Discussion of the water positive goal and optimizations that let AWS return water directly for irrigation rather than to the sewage system. STP213 Scaling global carbon footprint management — Blake Blackwell Persefoni Manager Data Engineering and Michael Floyd AWS Head of Sustainability Solutions.

AWS 64

AWS Energy Architecture Programming 64

InfoQ

JULY 20, 2023

Microsoft recently announced the general availability (GA) of Azure Managed Lustre, a managed file system for high-performance computing (HPC) and AI workloads. By Steef-Jan Wiggers

Azure 40

Azure Availability Systems Performance 40

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Key challenges. Performance.

Big Data 63

Big Data Storage Benchmarking Hardware 63

Abhishek Tiwari

DECEMBER 27, 2017

Unified data management architecture. A unified data management (UDM) system combines the best of data warehouses, data lakes, and streaming without expensive and error-prone ETL. Machine learning meets data integration. Overall, less complexity to deal with.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

Abhishek Tiwari

JULY 12, 2018

Data Pipeline A data pipeline is a software that ingests data from multiple sources, transforms it and finally makes it available to internal or external products. Unfortunately, building data pipelines remains a daunting, time-consuming, and costly activity. Depending on frameworks, data processing units (a.k.a

Latency 63

Latency Analytics Scalability Engineering 63

The Netflix TechBlog

NOVEMBER 22, 2019

4:45pm-5:45pm NFX 202 A day in the life of a Netflix Engineer Dave Hahn , SRE Engineering Manager Abstract : Netflix is a large, ever-changing ecosystem serving millions of customers across the globe through cloud-based systems and a globally distributed CDN.

AWS 37

AWS Entertainment Open Source Benchmarking 37

Testsigma

NOVEMBER 28, 2019

SoftwareTestingHelp has been around since 2006 and has a large number of blogs, articles, courses and ebooks covering a wide variety of topics on testing, automation and development. They have an affiliation with universities like MIT, Harvard University, Boston University, The University of Texas System and more.

Website 53

Website Testing Programming Automotive 53

O'Reilly

SEPTEMBER 15, 2021

In June 2021, we asked the recipients of our Data & AI Newsletter to respond to a survey about compensation. The results gave us insight into what our subscribers are paid, where they’re located, what industries they work for, what their concerns are, and what sorts of career development opportunities they’re pursuing.

Azure 145

Azure Social Media Programming AWS 145

High Scalability

JANUARY 22, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 22

Education Java Software Engineering Engineering 22

High Scalability

NOVEMBER 12, 2019

Grokking the System Design Interview is a popular course on Educative.io (taken by 20,000+ people) that's widely considered the best System Design interview resource on the Internet. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes.

Education 22

Education Java Software Engineering Engineering 22

High Scalability

OCTOBER 29, 2019

Grokking the System Design Interview is a popular course on Educative.io (taken by 20,000+ people) that's widely considered the best System Design interview resource on the Internet. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes.

Education 22

Education Java Software Engineering Engineering 22

Abhishek Tiwari

JULY 20, 2019

Store engineering squad - focus on software and systems required for the storefront including point-of-sales system, promotions, etc. ERP engineering squad - supply chain planning, purchase order management, product lifecycle management, merchandise planning, etc.

Engineering 61

Engineering Retail Airlines Healthcare 61

High Scalability

FEBRUARY 6, 2020

Level up on in-demand technologies and prep for your interviews on Educative.io, featuring popular courses like the bestselling Grokking the System Design Interview. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Advertise your job here!

Education 21

Education Java Software Engineering Engineering 21