The Netflix TechBlog

OCTOBER 18, 2022

Meson was based on a single leader architecture with high availability. increasing at > 100% a year, the need for a scalable data workflow orchestrator has become paramount for Netflix’s business needs. Figure 1 shows the high-level architecture. We developed our own secured expression language (SEL) to ensure security.

Java 202

Java Scalability Traffic Architecture 202

Dynatrace

JULY 6, 2022

As organizations continue to adopt multicloud strategies, the complexity of these environments grows, increasing the need to automate cloud engineering operations to ensure organizations can enforce their policies and architecture principles. Developing automation takes time. Read eBook now! The post What is IT automation?

Artificial Intelligence 194

Artificial Intelligence Tuning Strategy Big Data 194

The Netflix TechBlog

MARCH 5, 2019

As a micro-service owner, a Netflix engineer is responsible for its innovation as well as its operation, which includes making sure the service is reliable, secure, efficient and performant. How can we develop templated detection modules (rules- and ML-based) and data streams to increases speed of development?

Infrastructure 182

Infrastructure Cloud Scalability AWS 182

The Netflix TechBlog

NOVEMBER 20, 2023

These challenges are currently addressed in suboptimal and less cost efficient ways by individual local teams to fulfill the needs, such as Lookback: This is a generic and simple approach that data engineers use to solve the data accuracy problem. Users configure the workflow to read the data in a window (e.g.

Processing 210

Processing Big Data Efficiency Engineering 210

The Netflix TechBlog

JANUARY 11, 2022

Curious to learn about what it’s like to be a Data Engineer at Netflix? Hear directly from Samuel Setegne , Dhevi Rajendran , Kevin Wylie , and Pallavi Phadnis in our “Data Engineers of Netflix” interview series. Say we are developing a new way to generate a piece of evidence, such as a trailer.

Innovation 243

Innovation Metrics Engineering Testing 243

The Netflix TechBlog

DECEMBER 21, 2020

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. Some of the optimizations are prerequisites for a high-performance data warehouse.

Storage 203

Storage Latency Efficiency Data Engineering 203

Adrian Cockcroft

FEBRUARY 13, 2023

STP213 Scaling global carbon footprint management — Blake Blackwell Persefoni Manager Data Engineering and Michael Floyd AWS Head of Sustainability Solutions. SUS302 Optimizing architectures for sustainability — Katja Philipp AWS SA and Szymon Kochanski AWS SA. SUS209 — there was no talk with this code.

AWS 64

AWS Energy Architecture Programming 64

High Scalability

APRIL 14, 2020

You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Please apply here. Need excellent people?

Education 105

Education Software Engineering Scalability Engineering 105

High Scalability

MAY 12, 2020

You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Please apply here. Need excellent people?

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

APRIL 28, 2020

You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Please apply here. Need excellent people?

Education 102

Education Software Engineering Scalability Engineering 102

The Netflix TechBlog

MARCH 4, 2024

For efficient error handling, Netflix developed an error classification service, called Pensive, which leverages a rule-based classifier for error classification. To address these challenges, we have developed a new feature, called Auto Remediation , which integrates the rule-based classifier with an ML service.

Tuning 210

Tuning Efficiency Big Data Engineering 210

High Scalability

MAY 26, 2020

You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc. Created by former senior-level AWS engineers of 15 years.

Education 100

Education AWS Software Engineering Scalability 100

O'Reilly

JULY 15, 2020

Software engineers comprise the survey audience’s single largest cluster, over one quarter (27%) of respondents (Figure 1). If you combine the different architectural roles—i.e., Adding architects and engineers, we see that roughly 55% of the respondents are directly involved in software development.

Database 134

Database Architecture Education Systems 134

Abhishek Tiwari

DECEMBER 27, 2017

A common theme across all these trends is to remove the complexity by simplifying data management as a whole. In 2018, we anticipate that ETL will either lose relevance or the ETL process will disintegrate and be consumed by new data architectures. Unified data management architecture.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

O'Reilly

FEBRUARY 2, 2020

Source code is relatively less important compared to typical applications; the training data is what determines how the model behaves, and the training process is all about tweaking parameters in the application so that it delivers correct results most of the time. You need a repository for models and for the training data.

Software Architecture 67

Software Architecture Monitoring Software Engineering Architecture 67

Abhishek Tiwari

JULY 20, 2019

The engineering organisation described may not work for you because of a team of 8-10 people is still a very big overhead. In this model, software architecture and code ownership is a reflection of the organisational model. Thirdly, let engineers themselves choose the delivery teams and organise them around the initiative.

Engineering 61

Engineering Retail Airlines Healthcare 61

Abhishek Tiwari

JULY 12, 2018

Data Pipeline A data pipeline is a software that ingests data from multiple sources, transforms it and finally makes it available to internal or external products. Unfortunately, building data pipelines remains a daunting, time-consuming, and costly activity. Depending on frameworks, data processing units (a.k.a

Latency 63

Latency Analytics Scalability Engineering 63

All Things Distributed

NOVEMBER 15, 2016

Put simply, data is not always readily available and accessible to organizational end users. The data infrastructure to collect, store, and process data is geared primarily towards developers and IT professionals whereas insights need to be derived by not just technical professionals but also non-technical business users.

Analytics 152

Analytics Availability Media Social Media 152

InfoQ

AUGUST 3, 2023

The GA is a follow-up to the earlier announcement of the development of the infrastructure. AWS recently announced the general availability (GA) of Amazon EC2 P5 instances powered by the latest NVIDIA H100 Tensor Core GPUs suitable for users that require high performance and scalability in AI/ML and HPC workloads. By Steef-Jan Wiggers

AWS 77

AWS Availability Scalability Infrastructure 77

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Key challenges. Performance.

Big Data 63

Big Data Storage Benchmarking Hardware 63

InfoQ

DECEMBER 29, 2023

Zendesk reduced its data storage costs by over 80% by migrating from DynamoDB to a tiered storage solution using MySQL and S3. The company considered different storage technologies and decided to combine the relational database and the object store to strike a balance between querybility and scalability while keeping the costs down.

Storage 134

Storage Scalability Database Technology 134

Abhishek Tiwari

DECEMBER 23, 2017

Data integration generally requires in-depth domain knowledge, a strong understanding of data schemas and underlying relationships. This can be time-consuming and bit challenging if you are dealing with hundreds of data sources and thousands of event types (see my recent article on ELT architecture ). Tip of the iceberg.

Architecture 40

Architecture Engineering Processing Code 40

InfoQ

NOVEMBER 29, 2023

HubSpot adopted routing messages over multiple Kafka topics (called swimlanes) for the same producer to avoid the build-up in the consumer group lag and prioritize the processing of real-time traffic.

Processing 79

Processing Traffic Data Engineering Scalability 79

O'Reilly

MARCH 2, 2020

The one thing I don’t see, and the one thing that more than anything else captures the value in Agile, is the ongoing conversation between the customer (however that’s conceived) and the developer. Agile is not, and never was, about getting developers to write software faster. This is important. Neckbeards? Geeks and nerds?

Artificial Intelligence 20

Artificial Intelligence Software Architecture Programming Infrastructure 20

InfoQ

JULY 3, 2023

LinkedIn introduced Couchbase as a centralized caching tier for scaling member profile reads to handle increasing traffic that has outgrown their existing database cluster. The new solution achieved over 99% hit rate, helped reduce tail latencies by more than 60% and costs by 10% annually. By Rafal Gancarz

Cache 83

Cache Latency Traffic Database 83

InfoQ

JULY 20, 2023

Microsoft recently announced the general availability (GA) of Azure Managed Lustre, a managed file system for high-performance computing (HPC) and AI workloads. By Steef-Jan Wiggers

Azure 40

Azure Availability Systems Performance 40

InfoQ

APRIL 7, 2021

Recently, Google announced A2 Virtual Machines (VMs)' general availability based on the NVIDIA Ampere A100 Tensor Core GPUs in Compute Engine.

Virtualization 56

Virtualization Google Availability Engineering 56