Data Engineering, Performance and Tuning - Technology Performance Pulse

Evolving from Rule-based Classifier: Machine Learning Powered Auto Remediation in Netflix Data…

The Netflix TechBlog

MARCH 4, 2024

Operational automation–including but not limited to, auto diagnosis, auto remediation, auto configuration, auto tuning, auto scaling, auto debugging, and auto testing–is key to the success of modern data platforms. Auto Remediation generates recommendations by considering both performance (i.e., Multi-objective optimizations.

Tuning

Tuning Efficiency Big Data Engineering

3. Psyberg: Automated end to end catch up

The Netflix TechBlog

NOVEMBER 14, 2023

Audit Run various quality checks on the staged data. Several audits, such as verifying source and target counts, are performed on this batch of data. Publish If the audits are successful, cherry-pick the staging snapshot to publish the data to production. Stay tuned for a new post on this!

Tuning

Tuning Processing C++ Efficiency

What is IT automation?

Dynatrace

JULY 6, 2022

Expect to spend time fine-tuning automation scripts as you find the right balance between automated and manual processing. This requires significant data engineering efforts, as well as work to build machine-learning models. By tuning workflows, you can increase their efficiency and effectiveness.

Artificial Intelligence

Artificial Intelligence Tuning Strategy Big Data

Presentation: Modern Compute Stack for Scaling Large AI/ML/LLM Workloads

InfoQ

MAY 8, 2024

Jules Damji discusses which infrastructure should be used for distributed fine-tuning and training, how to scale ML workloads, how to accommodate large models, and how can CPUs and GPUs be utilized? By Jules Damji

Tuning

Tuning Infrastructure Artificial Intelligence Data Engineering

Friends don't let friends build data pipelines

Abhishek Tiwari

JULY 12, 2018

Unfortunately, building data pipelines remains a daunting, time-consuming, and costly activity. Not everyone is operating at Netflix or Spotify scale data engineering function. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines.

Latency

Latency Analytics Scalability Engineering

QCon London: Lessons Learned From Building LinkedIn’s AI/ML Data Platform

InfoQ

APRIL 15, 2024

He specifically delved into Venice DB, the NoSQL data store used for feature persistence. At the QCon London 2024 conference, Félix GV from LinkedIn discussed the AI/ML platform powering the company’s products. By Rafal Gancarz

Artificial Intelligence

Artificial Intelligence Big Data Data Engineering Latency

Optimizing data warehouse storage

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage

Storage Latency Efficiency Data Engineering

Analytics at Netflix: Who we are and what we do

The Netflix TechBlog

SEPTEMBER 18, 2020

Full ownership often means building new data pipelines, navigating complex schemas and large data sets, developing or improving metrics for business performance, and creating intuitive visualizations and dashboards?—?always and technical excellence ??. always with an eye towards actionable insights.

Analytics

Analytics Engineering Data Engineering Tuning

Hyper Scale VPC Flow Logs enrichment to provide Network Insight

The Netflix TechBlog

MAY 26, 2020

And in order to gain visibility into these logs, we need to somehow ingest and enrich this data. It is easier to tune a large Spark job for a consistent volume of data. In other words, we are able to ensure that our Spark app does not “eat” more data than it was tuned to handle. We named this library Sqooby.

Network

Network Tuning AWS Big Data

Incremental Processing using Netflix Maestro and Apache Iceberg

The Netflix TechBlog

NOVEMBER 20, 2023

These challenges are currently addressed in suboptimal and less cost efficient ways by individual local teams to fulfill the needs, such as Lookback: This is a generic and simple approach that data engineers use to solve the data accuracy problem. Users configure the workflow to read the data in a window (e.g.

Processing

Processing Big Data Efficiency Engineering

Formulating ‘Out of Memory Kill’ Prediction on the Netflix App as a Machine Learning Problem

The Netflix TechBlog

JULY 21, 2022

Specifically, if we are able to predict or analyze the Out of Memory kills, we can take device specific actions to pre-emptively lower the performance in favor of not crashing?—?aiming aiming to give the user the ultimate Netflix Experience within the “performance vs pre-emptive action” tradeoff limitations.

Big Data

Big Data Cache Engineering Data Engineering

Orchestrating Data/ML Workflows at Scale With Netflix Maestro

The Netflix TechBlog

OCTOBER 18, 2022

Therefore, the orchestrator has to manage a workflow consisting of hundreds of thousands of jobs in a performant way, which is also quite challenging. Usability Netflix is a data-driven company, where key decisions are driven by data insights, from the pixel color used on the landing page to the renewal of a TV-series.

Java

Java Scalability Traffic Architecture

Organise your engineering teams around the work by reteaming

Abhishek Tiwari

JULY 20, 2019

For instance, if you are fast-growing VC funded e-commerce startup and your number one business priority is multiplying current growth and performing exceptionally well on key financial metrics charted out by your investors. Is it possible to draw inspiration from outside of software engineering? How is that even possible?

Engineering

Engineering Retail Airlines Healthcare

Technology Performance Pulse

Evolving from Rule-based Classifier: Machine Learning Powered Auto Remediation in Netflix Data…

3. Psyberg: Automated end to end catch up

Trending Sources

What is IT automation?

Presentation: Modern Compute Stack for Scaling Large AI/ML/LLM Workloads

Friends don't let friends build data pipelines

QCon London: Lessons Learned From Building LinkedIn’s AI/ML Data Platform

Optimizing data warehouse storage

Analytics at Netflix: Who we are and what we do

Hyper Scale VPC Flow Logs enrichment to provide Network Insight

Incremental Processing using Netflix Maestro and Apache Iceberg

Formulating ‘Out of Memory Kill’ Prediction on the Netflix App as a Machine Learning Problem

Orchestrating Data/ML Workflows at Scale With Netflix Maestro

Organise your engineering teams around the work by reteaming

Stay Connected