The Netflix TechBlog

MARCH 5, 2019

As a micro-service owner, a Netflix engineer is responsible for its innovation as well as its operation, which includes making sure the service is reliable, secure, efficient and performant. How can we develop templated detection modules (rules- and ML-based) and data streams to increases speed of development?

Infrastructure 182

Infrastructure Cloud Scalability AWS 182

Dynatrace

JULY 6, 2022

Developing automation takes time. This requires significant data engineering efforts, as well as work to build machine-learning models. To learn more about how Dynatrace drives more efficient IT operations by automating IT processes, read the ebook, Developing an AIOps strategy for cloud observability. Read eBook now!

Artificial Intelligence 185

Artificial Intelligence Tuning Strategy Big Data 185

The Netflix TechBlog

MARCH 2, 2021

At Netflix, our data scientists span many areas of technical specialization, including experimentation, causal inference, machine learning, NLP, modeling, and optimization. Together with data analytics and data engineering, we comprise the larger, centralized Data Science and Engineering group.

Analytics 207

Analytics C++ Innovation Engineering 207

The Netflix TechBlog

MARCH 4, 2024

For efficient error handling, Netflix developed an error classification service, called Pensive, which leverages a rule-based classifier for error classification. To address these challenges, we have developed a new feature, called Auto Remediation , which integrates the rule-based classifier with an ML service.

Tuning 210

Tuning Efficiency Big Data Engineering 210

High Scalability

APRIL 14, 2020

You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Please apply here. Join more than 265,000 other learners.

Education 105

Education Software Engineering Scalability Engineering 105

High Scalability

MAY 12, 2020

You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Please apply here. Join more than 265,000 other learners.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

APRIL 28, 2020

You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Please apply here. Join more than 265,000 other learners.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

MARCH 24, 2020

You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Please apply here. Try the 30-day free trial!

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

MAY 26, 2020

You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Please apply here. Join more than 300,000 other learners.

Education 100

Education AWS Software Engineering Scalability 100

The Agile Manager

JULY 31, 2022

There are shadow IT teams of developers or data engineers that spring up in areas like operations or marketing because the captive IT function is slow, if not outright incapable, of responding to internal customer demand. IT shadows appear in a lot of different forms. The scope taken out of the 1.0

Programming 59

Programming Engineering Data Engineering Innovation 59

High Scalability

MARCH 17, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 105

Education Engineering Java Software Engineering 105

High Scalability

MARCH 3, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 114

Education Games Engineering Java 114

High Scalability

MARCH 30, 2020

You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Please apply here. Try the 30-day free trial!

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

FEBRUARY 18, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 113

Education Games Engineering Java 113

High Scalability

FEBRUARY 9, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 108

Education Games Engineering Java 108

Abhishek Tiwari

FEBRUARY 26, 2020

Containers for local development and probably in UAT. Explore serverless functions to create Skills++: Induct Technical Architects, Developer Experience (DevX) 50-100 Engineers Focus: Finding new ways to add more value quickly for your customers by exploiting data. Test coverage (50-70%).

Technology 81

Technology Technology Architecture Engineering 81

InfoQ

AUGUST 3, 2023

The GA is a follow-up to the earlier announcement of the development of the infrastructure. AWS recently announced the general availability (GA) of Amazon EC2 P5 instances powered by the latest NVIDIA H100 Tensor Core GPUs suitable for users that require high performance and scalability in AI/ML and HPC workloads. By Steef-Jan Wiggers

AWS 78

AWS Availability Scalability Infrastructure 78

The Netflix TechBlog

NOVEMBER 22, 2019

3:15pm-4:15pm OPN 209 Netflix’s application deployment at scale Andy Glover , Director Delivery Engineering & Paul Roberts, AWS Abstract : Spinnaker is an open-source continuous-delivery platform created by Netflix to improve its developers’ efficiency and reduce the time it takes to get an application into production.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

3:15pm-4:15pm OPN 209 Netflix’s application deployment at scale Andy Glover , Director Delivery Engineering & Paul Roberts, AWS Abstract : Spinnaker is an open-source continuous-delivery platform created by Netflix to improve its developers’ efficiency and reduce the time it takes to get an application into production.

AWS 100

AWS Entertainment Open Source Benchmarking 100

O'Reilly

FEBRUARY 2, 2020

Source code is relatively less important compared to typical applications; the training data is what determines how the model behaves, and the training process is all about tweaking parameters in the application so that it delivers correct results most of the time. You need a repository for models and for the training data.

Software Architecture 68

Software Architecture Monitoring Software Engineering Architecture 68

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Key challenges. Performance.

Big Data 63

Big Data Storage Benchmarking Hardware 63

Adrian Cockcroft

FEBRUARY 13, 2023

STP213 Scaling global carbon footprint management — Blake Blackwell Persefoni Manager Data Engineering and Michael Floyd AWS Head of Sustainability Solutions. Good example, well presented interesting new material.

AWS 64

AWS Energy Architecture Programming 64

O'Reilly

JULY 15, 2020

Adding architects and engineers, we see that roughly 55% of the respondents are directly involved in software development. Technical roles represented in the “Other” category include IT managers, data engineers, DevOps practitioners, data scientists, systems engineers, and systems administrators.

Database 135

Database Architecture Education Systems 135

The Netflix TechBlog

SEPTEMBER 18, 2020

My work is typically developed in R or Python. I focus on improving experimentation methodology to test how well the newest files are working: do they need less bits to stream while providing a higher video quality? Do they cause less errors?

Analytics 223

Analytics Education Innovation Engineering 223

Abhishek Tiwari

DECEMBER 27, 2017

Machine learning meets data integration. Data solution vendors like SnapLogic and Informatica are already developing machine learning and artificial intelligence (AI) based smart data integration assistants.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

Abhishek Tiwari

JULY 12, 2018

Data Pipeline A data pipeline is a software that ingests data from multiple sources, transforms it and finally makes it available to internal or external products. Unfortunately, building data pipelines remains a daunting, time-consuming, and costly activity. Depending on frameworks, data processing units (a.k.a

Latency 63

Latency Analytics Scalability Engineering 63

InfoQ

DECEMBER 29, 2023

Zendesk reduced its data storage costs by over 80% by migrating from DynamoDB to a tiered storage solution using MySQL and S3. The company considered different storage technologies and decided to combine the relational database and the object store to strike a balance between querybility and scalability while keeping the costs down.

Storage 134

Storage Scalability Database Technology 134

Testsigma

NOVEMBER 28, 2019

SoftwareTestingHelp has been around since 2006 and has a large number of blogs, articles, courses and ebooks covering a wide variety of topics on testing, automation and development. Udacity Udacity provides nanodegree programs on all automation languages like C++, Machine Learning, Data engineer, Robotics and more.

Website 53

Website Testing Programming Automotive 53

The Netflix TechBlog

DECEMBER 3, 2022

Dataflow Dataflow is a command line utility built to improve experience and to streamline the data pipeline development at Netflix. The most commonly used one is dataflow project , which helps folks in managing their data pipeline repositories through creation, testing, deployment and few other activities. test_sparksql_write.py

Best Practices 358

Best Practices Code Testing Data Engineering 358

The Netflix TechBlog

SEPTEMBER 18, 2020

Full ownership often means building new data pipelines, navigating complex schemas and large data sets, developing or improving metrics for business performance, and creating intuitive visualizations and dashboards?—?always Others have grown into new areas as part of their professional development at Netflix.

Analytics 240

Analytics Engineering Data Engineering Tuning 240

InfoQ

SEPTEMBER 25, 2023

When it comes to deploying Large Language Models (LLMs) in production, the two major challenges originate from the huge amount of parameters they require and the necessity of handling very long input sequences to represent contextual information.

Data Engineering 94

Data Engineering Scalability Engineering Development 94

InfoQ

NOVEMBER 29, 2023

HubSpot adopted routing messages over multiple Kafka topics (called swimlanes) for the same producer to avoid the build-up in the consumer group lag and prioritize the processing of real-time traffic.

Processing 77

Processing Traffic Data Engineering Scalability 77

The Netflix TechBlog

FEBRUARY 9, 2022

Let’s define some requirements that we are interested in delivering to the Netflix data engineers or anyone who would like to schedule a workflow with some external assets in it. Conclusions This new method available for Netflix data engineers makes workflow management easier, more transparent and more reliable.

Storage 201

Storage Data Engineering Testing Code 201

High Scalability

JANUARY 22, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 22

Education Java Software Engineering Engineering 22

The Netflix TechBlog

NOVEMBER 22, 2019

3:15pm-4:15pm OPN 209 Netflix’s application deployment at scale Andy Glover , Director Delivery Engineering & Paul Roberts, AWS Abstract : Spinnaker is an open-source continuous-delivery platform created by Netflix to improve its developers’ efficiency and reduce the time it takes to get an application into production.

AWS 37

AWS Entertainment Open Source Benchmarking 37

High Scalability

FEBRUARY 6, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 21

Education Java Software Engineering Engineering 21

InfoQ

JULY 3, 2023

LinkedIn introduced Couchbase as a centralized caching tier for scaling member profile reads to handle increasing traffic that has outgrown their existing database cluster. The new solution achieved over 99% hit rate, helped reduce tail latencies by more than 60% and costs by 10% annually. By Rafal Gancarz

Cache 83

Cache Latency Traffic Database 83