The Netflix TechBlog

MAY 26, 2020

It is easier to tune a large Spark job for a consistent volume of data. As you may know, S3 can emit messages when events (such as a file creation events) occur which can be directed into an AWS SQS queue. In other words, we are able to ensure that our Spark app does not “eat” more data than it was tuned to handle.

Network 150

Network Tuning AWS Big Data 150

Rigor

SEPTEMBER 11, 2019

Jake is a frequent speaker at many popular conferences and events, such as 100 Days of Google Dev , JAMstakConf , JSConf , SmashingConf , and dozens of others. Sergey is a principal engineer at Meetup and a well-known performance educator who regularly runs monthly hands-on Meet4SPEED events in New York City. Sergey Chernyshev.

Performance 89

Performance Education Google Website 89

Dynatrace

MARCH 11, 2021

In the fourth part of the series, I’ll show you how I used Dynatrace’s raw problem and event data to find the best fit for optimized anomaly detection settings. I took a big-data-analysis approach, which started with another problem visualization. Statistically analyzing Dynatrace’s event and problem data.

Tuning 263

Tuning Architecture Monitoring Big Data 263

The Netflix TechBlog

JULY 26, 2021

With this batch style approach, several issues have surfaced like data movement is tightly coupled with database tables, database schema is not an exact mapping of business data model, and data being stale given it is not real time etc. As of now, CDC sources have been implemented for data stores at Netflix (MySQL, Postgres).

Big Data 253

Big Data Government Analytics Processing 253

Dynatrace

APRIL 20, 2023

Further, business leaders must often determine whether the data is relevant for the business and if they can afford it. Logs are automatically produced and time-stamped documentation of events relevant to cloud architectures. Business leaders can decide which logs they want to use and tune storage to their data needs.

Analytics 187

Analytics Infrastructure Storage Efficiency 187

Dynatrace

FEBRUARY 16, 2023

As teams try to gain insight into this data deluge, they have to balance the need for speed, data fidelity, and scale with capacity constraints and cost. To solve this problem, Dynatrace launched Grail, its causational data lakehouse , in 2022. But logs are just one pillar of the observability triumvirate.

Analytics 186

Analytics Innovation Metrics Database 186

The Netflix TechBlog

APRIL 29, 2019

Our Infrastructure Security team leverages Python to help with IAM permission tuning using Repokid. Orchestration The Big Data Orchestration team is responsible for providing all of the services and tooling to schedule and execute ETL and Adhoc pipelines. We leverage Python to protect our SSH resources using Bless.

Open Source 279

Open Source Network Infrastructure Big Data 279

The Netflix TechBlog

DECEMBER 21, 2020

These principles reduce resource usage by being more efficient and effective while lowering the end-to-end latency in data processing. Both automatic (event-driven) as well as manual (ad-hoc) optimization. It decides what to do and when to do in response to an incoming event. Transparency to end-users.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

OCTOBER 18, 2022

by Jun He , Akash Dwivedi , Natallia Dzenisenka , Snehal Chennuru , Praneeth Yenugutala , Pawan Dixit At Netflix, Data and Machine Learning (ML) pipelines are widely used and have become central for the business, representing diverse use cases that go beyond recommendations, predictions and data transformations.

Java 202

Java Scalability Traffic Architecture 202

The Netflix TechBlog

MARCH 25, 2019

In the push model paradigm, various platform tools such as the data transportation layer, reporting tools, and Presto will publish lineage events to a set of lineage related Kafka topics, therefore, making data ingestion relatively easy to scale improving scalability for the data lineage system.

Infrastructure 219

Infrastructure Big Data Transportation Architecture 219

The Netflix TechBlog

NOVEMBER 20, 2023

For example, a job would reprocess aggregates for the past 3 days because it assumes that there would be late arriving data, but data prior to 3 days isn’t worth the cost of reprocessing. Backfill: Backfilling datasets is a common operation in big data processing.

Processing 210

Processing Big Data Efficiency Engineering 210

The Netflix TechBlog

NOVEMBER 3, 2023

Stay tuned for more updates! Streaming SQL in Data Mesh was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story. We’ve been working with our partner teams to prioritize and build the next set of features to extend the SQL Processor.

Processing 265

Processing Engineering Infrastructure Latency 265