The Netflix TechBlog

OCTOBER 27, 2020

By Tianlong Chen and Ioannis Papapanagiotou Netflix has more than 195 million subscribers that generate petabytes of data everyday. Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy.

Latency 243

Latency Storage Big Data Tuning 243

Percona

OCTOBER 18, 2023

Migrating a proprietary database to open source is a major decision that can significantly affect your organization. Today, we’ll be taking a deep dive into the intricacies of database migration, along with specific solutions to help make the process easier.

Open Source 89

Open Source Database Hardware Strategy 89

Percona

NOVEMBER 3, 2023

Sure, database migration is complex, particularly when you’re looking to migrate from a proprietary database to an open source one. Database migration is almost always time-consuming, tedious, and full of potential pitfalls. Database migration is complex Let’s start here. Have you tuned your environment?

Database 81

Database Open Source Benchmarking Strategy 81

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 203

Storage Latency Efficiency Data Engineering 203

DZone

MAY 23, 2019

How do you tune the Snowflake data warehouse when there are no indexes, and few options available to tune the database itself? Snowflake was designed for simplicity, with few performance tuning options. This article summarizes the top five best practices to maximize query performance.

Best Practices 100

Best Practices Tuning Performance Database 100

Scalegrid

JULY 13, 2020

Oracle Database is a commercial, proprietary multi-model database management system produced by Oracle Corporation, and the largest relational database management system (RDBMS) in the world. While Oracle remains the #1 database on the market, its popularity has steadily declined by over 18% since 2013. Not available.

Open Source 244

Open Source Tuning C++ Database 244

The Netflix TechBlog

MARCH 25, 2019

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can

Infrastructure 219

Infrastructure Big Data Transportation Architecture 219

Dynatrace

MAY 17, 2023

OpenTelemetry , the open source observability tool, has become the go-to standard for instrumenting custom applications to collect observability telemetry data. For this third and final part of our series, we saved the best for last: How you can enhance telemetry data even more and with less effort on your end with Dynatrace OneAgent.

Metrics 176

Metrics Monitoring Database Network 176

The Netflix TechBlog

DECEMBER 17, 2019

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events.

Database 212

Database Traffic Transportation Open Source 212

The Netflix TechBlog

DECEMBER 17, 2019

Andreas Andreakis , Ioannis Papapanagiotou Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. In databases like MySQL and PostgreSQL, transaction logs are the source of CDC events.

Database 197

Database Traffic Transportation Open Source 197

All Things Distributed

MARCH 19, 2018

For example, training on more data means more accurate models. At AWS, we continue to strive to enable builders to build cutting-edge technologies faster in a secure, reliable, and scalable fashion. Now, thousands of customers are trying Amazon SageMaker and building ML models on top of their data lakes in AWS.

Scalability 159

Scalability Hardware AWS Tuning 159

Percona

JANUARY 4, 2024

Managing storage and performance efficiently in your MySQL database is crucial, and general tablespaces offer flexibility in achieving this. They provide flexibility in data organization and performance optimization compared to the default setup. What are MySQL general tablespaces?

Storage 85

Storage Engineering Database Hardware 85

Percona

NOVEMBER 2, 2023

MongoDB is a dynamic database system continually evolving to deliver optimized performance, robust security, and limitless scalability. Our new eBook, “ From Planning to Performance: MongoDB Upgrade Best Practices ,” guides you through the entire process to ensure your database’s long-term success.

Best Practices 84

Best Practices Hardware Tuning Scalability 84

The Netflix TechBlog

FEBRUARY 9, 2021

Below is a visual representation of the various systems involved in retrieving plan and offer data. Moving forward, we’ll refer to the combination of plan and offer data simply as SKU (Stock Keeping Unit) data. Notice the difference in data structures from the legacy implementation.

Engineering 142

Engineering Scalability Architecture Innovation 142

Scalegrid

MARCH 22, 2024

This covers everything from tech specs and what you expect from the service to security, how you’ll handle data, and how services will be managed. Hosting either on-premises or with third-party data center providers and deciding whether management services should be conducted internally or outsourced.

Strategy 130

Strategy Cloud Government Innovation 130

Dynatrace

DECEMBER 18, 2023

For busy site reliability engineers, ensuring system reliability, scalability, and overall health is an imperative that’s getting harder to achieve in ever-expanding, cloud-native, container-based environments. To get a more granular look into telemetry data, many analysts rely on custom metrics using Prometheus. What is Prometheus?

Metrics 183

Metrics Engineering Energy Tuning 183

Percona

MARCH 23, 2023

Hardware Memory The amount of RAM to be provisioned for database servers can vary greatly depending on the size of the database and the specific requirements of the company. Storage The type of storage and disk used for database servers can have a significant impact on performance and reliability. Benchmark before you decide.

Performance 123

Performance Hardware Tuning Storage 123

Dynatrace

JUNE 16, 2022

These application security testing approaches often do not have enough insight into real-time data and event flows to prevent vulnerabilities from slipping through the review process. An ideal RASP technology does not need training or fine-tuning to learn what bad application behavior looks like. The benefits of RASP.

Tuning 228

Tuning Cloud Open Source Network 228

Percona

FEBRUARY 15, 2023

Bloom filters are an essential component of an LSM-based database engine like MyRocks. With MyRocks/RocksDB, data is stored in a set of large SST files. Tuning In terms of tuning, two parameters can be tuned, the size of the bitmap and the number of bits set by every value.

Storage 125

Storage Tuning Cache Engineering 125

Percona

OCTOBER 10, 2023

Databases are different from a lot of software. This is not a general rule, but as databases are responsible for a core layer of any IT system – data storage and processing — they require reliability. Stay tuned for more news about MongoDB offerings. For one, they often favor stability over innovation.

Availability 96

Availability Servers Database Open Source 96

Software Architecture

AUGUST 24, 2017

Source: [link] Cassandra has tunable consistency which means that not only on the database level, you can tune the immediate and eventual consistency of your data per query/operation by setting the read CL (consistency level) and write CL. W= number of nodes your data is written to. But, you are NOT getting a guarantee.

Tuning 100

Tuning Software Architecture Database C++ 100

The Netflix TechBlog

JANUARY 10, 2024

Reloaded was well-architected, providing good stability, scalability, and a reasonable level of flexibility. In addition to the scalability and the stability that the developers already enjoyed in Reloaded, Cosmos aimed to significantly increase system flexibility and feature development velocity. 264, AV1, etc.).

Processing 229

Processing Media Latency Innovation 229

Percona

JUNE 2, 2023

WordPress also benefited from and popularized MySQL, introducing the LAMP stack – and open source – to an audience that might never have touched a database before or had any intro to open source. MySQL was the database for phpWebLog, for example, way back in 2000. With a little tuning, though, MySQL could hold its own.

Open Source 92

Open Source Traffic Tuning Database 92

Percona

AUGUST 7, 2023

In InnoDB, the undo log and the redo log are two indispensable components that play a vital role in maintaining data integrity and ensuring transactional consistency. Both logs are crucial in the ACID (Atomicity, Consistency, Isolation, Durability) properties of database systems.

Database 94

Database Tuning Storage Open Source 94

Dynatrace

NOVEMBER 10, 2022

Lifting and shifting applications from the data center to the cloud delivers only marginal benefits. To take full advantage of the scalability, flexibility, and resilience of cloud platforms, organizations need to build or rearchitect applications around a cloud-native architecture. Stateless whenever possible.

Cloud 162

Cloud Architecture Serverless Infrastructure 162

Dynatrace

OCTOBER 2, 2023

Because monolithic applications combine database, client-side interfaces, and server-side application elements in a single executable, they’re difficult to understand, even for their own administrators. Utilize observability data to monitor and improve digital experiences and analyze data that can affect the business.

Architecture 237

Architecture Artificial Intelligence Cloud Open Source 237

Dynatrace

SEPTEMBER 29, 2021

This opens the door to auto-scalable applications, which effortlessly matches the demands of rapidly growing and varying user traffic. For a deeper look into how to gain end-to-end observability into Kubernetes environments, tune into the on-demand webinar Harness the Power of Kubernetes Observability. What is Docker? Observability.

Open Source 307

Open Source Traffic DevOps Cloud 307

Software Architecture

AUGUST 24, 2017

Consistency The topic and concept of consistency is very important when you work with a distributed database like Cassandra. When you’re working with a database which runs on only one server, consistency is a non-issue. Read Consistency Read consistency refers to having same data on all replica nodes for any read request.

Software Architecture 100

Software Architecture Database Tuning Servers 100

Percona

DECEMBER 27, 2023

In our conversations with developers and engineers during the conference, we confirmed the CNCF GItOps Microsurvey data – 91% are already using GitOps. of respondents are currently utilizing databases in Kubernetes (k8s). Developers will receive the endpoint without needing to consider resources and tuning at all.

Cloud 77

Cloud Open Source Strategy Best Practices 77

Percona

MARCH 6, 2023

MySQL uses Redo Logs internally during crash recovery to correct data written by incomplete transactions. If you want an answer based on detailed information about your database usage and patterns, it’s recommended to use Percona Management and Monitoring. So you won’t have downtime if you need to tune this. GB per hour).

Tuning 87

Tuning Open Source Database Engineering 87

The Netflix TechBlog

OCTOBER 19, 2020

Traces collected from various microservices are ingested in a stream processing manner into the data store. Edgar uses this infrastructure tagging schema to query and join traces with log data for troubleshooting streaming sessions. Stream Processing: to sample or not to sample trace data?

Infrastructure 233

Infrastructure Transportation Storage Open Source 233

Percona

AUGUST 17, 2023

PostgreSQL is open source relational database management software. It’s used for data management (shocker), application development, and data analytics. In another survey, using multiple metrics for overall popularity, PostgreSQL ranks as the fourth most popular database management system ( DB-Engines, March 2023 ).

Open Source 76

Open Source Healthcare Social Media Database 76

HammerDB

APRIL 19, 2023

Among the different components of modern software solutions, the database is one of the most critical. There are many times we get asked why some cloud instance performed poorly for their database application and almost always turned out to be some configuration error. TB)) for storage of database tablespaces and logging.

Performance 54

Performance Benchmarking Cache Storage 54

The Netflix TechBlog

NOVEMBER 2, 2021

As VMAF evolves and is integrated with more encoding and streaming workflows within Netflix, we need scalable ways of fostering video quality innovations. The Reloaded system is a well-matured and scalable system, but its monolithic architecture can slow down rapid innovation. via bug fixes). The workflow is initiated.

Media 171

Media Innovation Metrics Latency 171

O'Reilly Software

APRIL 4, 2018

Considerations for setting the architectural foundations for a fast data platform. Google was among the pioneers that created “web scale” architectures to analyze the massive data sets that resulted from “crawling” the web that gave birth to Apache Hadoop, MapReduce, and NoSQL databases. Back in the days of Web 1.0,

Architecture 80

Architecture Scalability Google Operating System 80

HammerDB

JANUARY 5, 2019

Some opinions claim that “Benchmarks are meaningless”, “benchmarks are irrelevant” or “benchmarks are nothing like your real applications” However for others “Benchmarks matter,” as they “account for the processing architecture and speed, memory, storage subsystems and the database engine.”

Benchmarking 40

Benchmarking Database Social Media Scalability 40

Percona

MARCH 27, 2023

On MySQL and Percona Server for MySQL , there is a schema called information_schema (I_S) which provides information about database tables, views, indexes, and more. You can read some optimization examples in the official doc: [link]. Results for Percona Server for MySQL 8.0 Conclusion In conclusion, checking table information from 5.7

Cache 98

Cache Hardware Servers Benchmarking 98