Dynatrace

JULY 6, 2022

And what are the best strategies to reduce manual labor so your team can focus on more mission-critical issues? Expect to spend time fine-tuning automation scripts as you find the right balance between automated and manual processing. This requires significant data engineering efforts, as well as work to build machine-learning models.

Artificial Intelligence 188

Artificial Intelligence Tuning Strategy Big Data 188

The Netflix TechBlog

JULY 21, 2022

We at Netflix, as a streaming service running on millions of devices, have a tremendous amount of data about device capabilities/characteristics and runtime data in our big data platform. With large data, comes the opportunity to leverage the data for predictive and classification based analysis.

Big Data 179

Big Data Cache Engineering Data Engineering 179

The Netflix TechBlog

DECEMBER 21, 2020

Some of the optimizations are prerequisites for a high-performance data warehouse. Sometimes Data Engineers write downstream ETLs on ingested data to optimize the data/metadata layouts to make other ETL processes cheaper and faster. Orient: Gather tuning parameters for a particular table that changed.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

NOVEMBER 20, 2023

For example, a job would reprocess aggregates for the past 3 days because it assumes that there would be late arriving data, but data prior to 3 days isn’t worth the cost of reprocessing. Backfill: Backfilling datasets is a common operation in big data processing. data arrives too late to be useful).

Processing 210

Processing Big Data Efficiency Engineering 210