The Netflix TechBlog

JUNE 7, 2021

After several iterations of the architecture and some tuning, the solution has proven to be able to scale. Summary Providing network insight into the cloud network infrastructure using eBPF flow logs at scale is made possible with eBPF and a highly scalable and efficient flow collection pipeline.

Network 325

Network Transportation AWS Cloud 325

Dynatrace

SEPTEMBER 15, 2020

If you want to see a more hands-on approach, I encourage you to watch the recording as Stefano did a live demo of Akamas’s integration with Dynatrace, showing how to minimize the footprint of a Java application with automated JVM tuning.

Performance 185

Performance Java Metrics Cloud 185

The Netflix TechBlog

OCTOBER 15, 2019

High Level Architecture of Delta Stay Tuned We will publish follow-up blogs about technical details of the key components such as Delta-Connector and Delta Stream Processing Framework. Please stay tuned. Below is a view of the high level architecture of the Delta platform.

Transportation 192

Transportation Architecture Processing Storage 192

Percona

JANUARY 19, 2023

It was developed for optimizing data storage and access for big data sets. There is a cool blog post from Vadim covering big data sets in MyRocks: MyRocks Use Case: Big Dataset Query tuning: It is common to find applications that at the beginning perform very well, but as data grows the performance starts to decrease.

Open Source 85

Open Source Storage Database Big Data 85

Rigor

JANUARY 13, 2020

Take, for example, The Web Almanac , the golden collection of Big Data combined with the collective intelligence from most of the authors listed below, brilliantly spearheaded by Google’s @rick_viscomi. Web Performance Tuning. Professional Website Performance. Professional Website Performance. Website Optimization.

Performance 75

Performance Social Media Website Website Performance 75

InfoQ

APRIL 15, 2024

He specifically delved into Venice DB, the NoSQL data store used for feature persistence. At the QCon London 2024 conference, Félix GV from LinkedIn discussed the AI/ML platform powering the company’s products. By Rafal Gancarz

Artificial Intelligence 105

Artificial Intelligence Big Data Data Engineering Latency 105

Brendan Gregg

APRIL 30, 2018

In this year's CFP we’re looking for topics covering the latest trends and best practices in cloud computing, containerization, machine learning, big data, infrastructure, scalability, DevOps, IT management, automation, reliability, monitoring, performance tuning, security, databases, programming, datacenters, and more.

DevOps 43

DevOps Network Best Practices Programming 43

The Netflix TechBlog

MAY 26, 2020

And in order to gain visibility into these logs, we need to somehow ingest and enrich this data. It is easier to tune a large Spark job for a consistent volume of data. In other words, we are able to ensure that our Spark app does not “eat” more data than it was tuned to handle. We named this library Sqooby.

Network 150

Network Tuning AWS Big Data 150

Brendan Gregg

APRIL 29, 2018

In this year's CFP we’re looking for topics covering the latest trends and best practices in cloud computing, containerization, machine learning, big data, infrastructure, scalability, DevOps, IT management, automation, reliability, monitoring, performance tuning, security, databases, programming, datacenters, and more.

DevOps 40

DevOps Network Best Practices Programming 40

SQL Server According to Bob

FEBRUARY 15, 2018

Best practices on Building a Big Data Analytics Solution – Michael Rys. If you want to learn about Azure Data Lake, there is no one better. Friday Sessions: SQL Intelligence excels your tuning and security expertise – Veljko Vasic, Ron Matchoro and Frans Lytzen. I’ve known Michael for a very long time.

Engineering 40

Engineering Azure Best Practices Servers 40

The Netflix TechBlog

JULY 21, 2022

We at Netflix, as a streaming service running on millions of devices, have a tremendous amount of data about device capabilities/characteristics and runtime data in our big data platform. With large data, comes the opportunity to leverage the data for predictive and classification based analysis.

Big Data 179

Big Data Cache Engineering Data Engineering 179

The Netflix TechBlog

JULY 26, 2021

However, it is paramount that we validate the complete set of identifiers such as a list of movie ids across producers and consumers for higher overall confidence in the data transport layer of choice. Please stay tuned! Data Mesh: Delivering Data-driven Value at Scale , O’Reilly Media, Inc., Endnotes ¹ Inmon, Bill.

Big Data 253

Big Data Government Analytics Processing 253

Dynatrace

MARCH 11, 2021

I took a big-data-analysis approach, which started with another problem visualization. I wanted to understand how I could tune Dynatrace’s problem detection, but to do that I needed to understand the situation first. To achieve that I took two approaches: Visualizing historic problem data via a “Swimlane Visualization”.

Tuning 263

Tuning Architecture Monitoring Big Data 263

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3.

Latency 243

Latency Storage Big Data Tuning 243

Rigor

SEPTEMBER 11, 2019

He has a keen interest in web technologies, performance tuning, security, and the practical use of technology. Doug is a freelance mobile performance expert, a popular speaker – particularly on the topic of web tuning and image optimization – and the author of High Performance Android Apps. Doug Sillars. Doug Sillars.

Performance 89

Performance Education Google Website 89

Dynatrace

APRIL 20, 2023

Using Grail to heal observability pains Grail logs not only store big data, but also map out dependencies to enable fast analytics and data reasoning. Business leaders can decide which logs they want to use and tune storage to their data needs. Seamless integration.

Analytics 187

Analytics Infrastructure Storage Efficiency 187

Dynatrace

FEBRUARY 16, 2023

As teams try to gain insight into this data deluge, they have to balance the need for speed, data fidelity, and scale with capacity constraints and cost. To solve this problem, Dynatrace launched Grail, its causational data lakehouse , in 2022.

Analytics 185

Analytics Innovation Metrics Database 185

The Netflix TechBlog

APRIL 29, 2019

Our Infrastructure Security team leverages Python to help with IAM permission tuning using Repokid. Orchestration The Big Data Orchestration team is responsible for providing all of the services and tooling to schedule and execute ETL and Adhoc pipelines. We leverage Python to protect our SSH resources using Bless.

Open Source 279

Open Source Network Infrastructure Big Data 279

The Netflix TechBlog

NOVEMBER 20, 2023

For example, a job would reprocess aggregates for the past 3 days because it assumes that there would be late arriving data, but data prior to 3 days isn’t worth the cost of reprocessing. Backfill: Backfilling datasets is a common operation in big data processing.

Processing 210

Processing Big Data Efficiency Engineering 210

The Netflix TechBlog

MARCH 25, 2019

Netflix’s diverse data landscape made it challenging to capture all the right data and conforming it to a common data model. Spark is the primary big-data compute engine at Netflix and with pretty much every upgrade in Spark, the spark plan changed as well springing continuous and unexpected surprises for us.

Infrastructure 219

Infrastructure Big Data Transportation Architecture 219

The Netflix TechBlog

DECEMBER 21, 2020

Orient: Gather tuning parameters for a particular table that changed. AutoAnalyze In short, AutoAnalyze finds the best tuning/configuration parameters for a table. The work done in the service can be further broken down into the following 3 steps: Observe: Listen to changes in the warehouse in near real-time.

Storage 203

Storage Latency Efficiency Data Engineering 203

The Netflix TechBlog

OCTOBER 18, 2022

by Jun He , Akash Dwivedi , Natallia Dzenisenka , Snehal Chennuru , Praneeth Yenugutala , Pawan Dixit At Netflix, Data and Machine Learning (ML) pipelines are widely used and have become central for the business, representing diverse use cases that go beyond recommendations, predictions and data transformations.

Java 202

Java Scalability Traffic Architecture 202

O'Reilly

SEPTEMBER 19, 2023

Each time, the underlying implementation changed a bit while still staying true to the larger phenomenon of “Analyzing Data for Fun and Profit.” ” They weren’t quite sure what this “data” substance was, but they’d convinced themselves that they had tons of it that they could monetize.

Hardware 99

Hardware Storage Big Data Blockchain 99

All Things Distributed

MARCH 28, 2011

We see that with our Amazon customers; when they hear a great tune on a radio they may identify it using the Shazam or Soundhound apps on their mobile phone and buy that song instantly from the Amazon MP3 store. Driving down the cost of Big-Data analytics. Introducing the AWS South America (Sao Paulo) Region.

AWS 138

AWS Cloud Storage Internet 138

All Things Distributed

NOVEMBER 30, 2016

Effectively applying AI involves extensive manual effort to develop and tune many different types of machine learning and deep learning algorithms (e.g. automatic speech recognition, natural language understanding, image classification), collect and clean the training data, and train and tune the machine learning models.

AWS 166

AWS Lambda Artificial Intelligence Mobile 166

The Netflix TechBlog

NOVEMBER 3, 2023

Stay tuned for more updates! Streaming SQL in Data Mesh was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story. We’ve been working with our partner teams to prioritize and build the next set of features to extend the SQL Processor.

Processing 265

Processing Engineering Infrastructure Latency 265

Percona

JANUARY 14, 2019

Currently, an issue has been opened to make the “tailing” based on the primary key much faster: slow order by primary key with small limit on big data.

Database 40

Database Analytics Blockchain Healthcare 40