The Netflix TechBlog

NOVEMBER 14, 2023

This table’s workflow runs hourly, with the main input source being an Iceberg table storing all raw signup events partitioned by landing date, hour, and batch id. Similar to the approach in stateless data processing, Psyberg uses the provided inputs to parse out the partition information for each Iceberg snapshot of the source table.

Processing 226

Processing Data Engineering Efficiency Analytics 226

Percona

MAY 7, 2024

Last week, we were assisting a client who needed to resolve an issue related to partitioned tables.Often, a table can include dozens or even hundreds of partitions. As a result, we […]

Speed 78

Speed Tuning 78

DZone

SEPTEMBER 14, 2023

While this format may not be ideal when you only need to retrieve a few rows from a large partition, it truly excels in analytical use cases. Optimizing Data Input Make Use of Data Forma t In most cases, the data being processed is stored in a columnar format.

Big Data 269

Big Data Processing Games Open Source 269

The Netflix TechBlog

NOVEMBER 14, 2023

Psyberg’s metadata session table is used to identify the partitions included in a batch run. The ETL then appends this data to the corresponding partitions within the fact tables. The ETL process uses this time range to compute the state in the changed partitions and overwrite them in the target table.

Tuning 244

Tuning Processing C++ Efficiency 244

DZone

JUNE 28, 2023

It involves partitioning a large database into smaller, more manageable parts, known as shards. Database sharding is a powerful technique employed to manage large databases more effectively.

Database 228

Database Scalability Servers Performance 228

DZone

FEBRUARY 5, 2023

Performance tests conducted on 100 partitions and 100K records gave a P95 between 25-30ms for read and write. Prohibiting other fields to be indexed increases performance and reduces RU to query point records. Cosmos DB cost is measured in RU.) For key object storage, RU tends to be less, but it still depends on the payload size.

Azure 243

Azure Storage C++ Performance Testing 243

Scalegrid

SEPTEMBER 12, 2021

PostgreSQL also introduced many other breakthrough features in versions 10 through 11, 12 and 13 that make it a real competitor against Oracle, such as partitioning enhancements, parallel query and logical replication. New […].

240

240

DZone

AUGUST 26, 2021

To get rid of it, it is suggested to increase the C: drive partition size on Windows 10. When the C: drive runs out of space, it turns red and your system constantly sends messages for insufficient disk space on the local disk (C:). There are multiple ways through which you can extend C: drive with and without using software.

C++ 246

C++ Software Software Systems 246

DZone

DECEMBER 15, 2021

In short, the CAP theorem is a mathematical theorem describing how our application will behave in the event of network partitioning. It is one of the most important laws currently in existence. Through the course of this text, I will share more information on this theorem and why it is important.

Network 237

Network Systems Availability 237

Scalegrid

JANUARY 19, 2024

Key Takeaways Rollbacks in MongoDB are triggered by disruptions in the replication process due to primary node crashes, network partitions, or other failures, which can lead to substantial data loss and inconsistencies. This failure in replication could happen due to crashes, network partitions, or other situations where failover occurs.

Database 130

Database Network Servers Monitoring 130

SQL Performance

MAY 24, 2022

The WINDOW clause has the following syntax: WINDOW window_name AS ( [ reference_window_name ]. [ <window partition clause> ]. [ <window order clause> ]. [ <window frame clause> ] ). ORDER BY orderdate , orderid ROWS UNBOUNDED PRECEDING ) AS runsumqty , SUM ( val ) OVER ( PARTITION BY custid.

Servers 145

Servers C++ Code Database 145

SQL Performance

MAY 11, 2022

By a supporting index, I mean one with the window partitioning and ordering elements as the index key, and the rest of the columns that appear in the query as the index included columns. I often refer to such an indexing pattern as a POC index as an acronym for partitioning , ordering, and covering.

C++ 133

C++ Database Servers Code 133

SQL Performance

APRIL 13, 2022

This index can potentially support window functions partitioned by grp and ordered by ord. In both cases, the operators handling the window function’s calculation need to ingest the data ordered by the window partitioning and ordering elements. ( SELECT *, ROW_NUMBER ( ) OVER ( PARTITION BY grp ORDER BY ord ) -.

C++ 136

C++ Strategy Testing Servers 136

DZone

JANUARY 20, 2023

To consider as an example, Apache Kafka uses ZooKeeper for choosing their leader node for the topic partitions. It holds responsibility for providing configuration information, naming, synchronization, and group services over large clusters in distributed systems.

Servers 152

Servers Systems 152

All Things Distributed

NOVEMBER 30, 2012

Back-to-Basics Weekend Reading - Sparse Partitions. I am carrying with me a more theoretical paper on the principles of distributed computing: Sparse Partitions by Awerbug and Peleg. I am carrying with me a more theoretical paper on the principles of distributed computing: Sparse Partitions by Awerbug and Peleg. Comments ().

AWS 75

AWS Scalability Network Systems 75

DZone

MARCH 24, 2023

Say you want to create a Linux file system but do not have a free disk partition available. A loop device is a virtual or pseudo-device which enables a regular file to be accessed as a block device. In such a case, you can create a regular file on the disk and create a loop device using this file.

Virtualization 130

Virtualization Operating System Systems Availability 130

High Scalability

APRIL 16, 2019

Scenario 3 – Network Partition – Network Connectivity Breaks Down Between Master and Slave Nodes. We assume that due to network issues, the cluster has partitioned into two groups – master in one group and 2 slaves in the other group, and we will denote this as [(M), (S1,S2)].

Availability 149

Availability Network Azure AWS 149

Percona

JANUARY 19, 2023

Partitioning: Partitioning is a feature that allows dividing a large table into smaller sub-tables based on a partition key. The most common use case for table partitioning is to divide the data by date. This can help to split large data sets into smaller ones stored in multiple servers.

Open Source 93

Open Source Storage Database Big Data 93

Percona

DECEMBER 1, 2023

The PostgreSQL partition manager pg_partman is an open source extension widely supported and actively maintained by the PostgreSQL community. pg_partman is an extension that streamlines the creation and management of table partition sets, supporting both time-based and serial-based partitioning approaches.

Open Source 81

Open Source Storage Performance Design 81

InfoQ

JUNE 26, 2023

These features are burst capacity, hierarchical partition keys, serverless container storage of 1 TB, and priority-based execution. Microsoft has recently unveiled several new features for Azure Cosmos DB to enhance cost efficiency, boost performance, and increase elasticity. By Steef-Jan Wiggers

Azure 87

Azure Serverless Performance Storage 87

Percona

SEPTEMBER 27, 2023

PostgreSQL Partition Manager — ‘pg_partman’ — is an open source extension freely available, widely supported, and actively maintained by the PostgreSQL community. pg_partman creates and manages both time-based and serial-based table partition sets.

Storage 105

Storage Open Source Database Efficiency 105

Percona

MARCH 29, 2023

A partitioned table in MySQL has its data separated into different tablespaces while still being viewed as a single table. Partitioning can be a useful approach in some cases when handling huge sets of data. In this example, we use a table partitioned based on a date range. mysql> show create table salariesG 1.

Engineering 106

Engineering Open Source Database Scalability 106

Dynatrace

MAY 24, 2021

Dynatrace leverages AWS resource tags for analytics and also for Dynatrace management zones, which are a powerful information-partitioning mechanism that accelerates collaboration and the sharing of relevant team-specific data while simultaneously ensuring secure access control.

Lambda 298

Lambda AWS Serverless Java 298

Testlodge

FEBRUARY 15, 2023

Equivalence Partitioning Equivalence partitioning divides data into sections where results are either valid or invalid. Equivalence partitioning would look like this: To test the field, we would enter at least one value from each partition. All of them can help make a user’s experience more pleasant.

Testing 69

Testing Social Media Open Source Media 69

The Netflix TechBlog

AUGUST 30, 2021

Since device information updates published on MQTT contain the device_session_id in the topic, this means that all device information updates for a given device session will effectively appear on the same Kafka partition, thus giving us a well-defined message order for consumption.

Latency 213

Latency Traffic Transportation Hardware 213

Particular Software

JUNE 20, 2022

The key to a successful Cosmos DB system is its data partitioning strategy. Like the rows of shrubs in a hedge maze, the logical partitions that divide data must be carefully planned, because that affects the scalability of the system and defines the boundaries for logical transactions. In version 1.1

Azure 52

Azure Storage Strategy Tuning 52

Dynatrace

JANUARY 29, 2024

The system executes an automated remediation workflow that dynamically adjusts the disk size by adding space or redistributing partitions. Upon detection, the platform triggers an automated alert. On completing these actions, the observability platform confirms that the remediation effort was successful, closes the loop, and notifies teams.

Best Practices 191

Best Practices DevOps Energy Innovation 191

Percona

JULY 3, 2023

Partitioning is the concept of splitting large tables logically into smaller pieces for better performance of the database. The exact point at which a table will benefit from partitioning depends on the application. Here, as an example, I created a table with range partitioning and partition tables for each quarter on a Date column.

Storage 85

Storage Database Performance Open Source 85

SQL Performance

DECEMBER 27, 2019

6 more. , this took about 10 seconds per file, and yielded the following: To generate the partitions, I naïvely divided the data up "evenly" – or so I thought. In other words, the values I am specifying are the max values in each partition (with dates, you usually want RANGE RIGHT ). REBUILD PARTITION = ALL WITH. (

Database 70

Database Storage Hardware Efficiency 70

SQL Performance

AUGUST 14, 2019

Consider the following query (call it Query 2), which is the same as Query 1, only using the RANGE option instead of ROWS: SELECT actid , tranid , val , SUM ( val ) OVER ( PARTITION BY actid ORDER BY tranid RANGE UNBOUNDED PRECEDING ) AS balance FROM dbo. Figure 1: Plan for Query 1, row-mode processing.

Best Practices 108

Best Practices Servers Database Azure 108

SQL Performance

AUGUST 1, 2022

For example, the following query computes both the continuous and the discrete median (0.5 75 Test XYZ Student C 55 Test XYZ 77.5 75 Test XYZ Student D 55 Test XYZ 77.5 75 Test XYZ Student H 65 Test XYZ 77.5 75 Test XYZ Student I 75 Test XYZ 77.5 75 Test XYZ Student B 80 Test XYZ 77.5 75 Test XYZ Student F 80 Test XYZ 77.5

Servers 142

Servers Testing C++ Code 142

DZone

OCTOBER 29, 2019

Do less: Less memory, less CPU, less disk, less IO, fewer instructions, fewer partitions, less overflow. Less is more. —. Ludwig Mies van der Rohe. This is no truer statement on the goals of a query optimizer. Less everything for the query plan it creates. This is the guiding light for SQL and NoSQL optimizer.

Performance 100

Performance 100

InfoQ

OCTOBER 10, 2022

DoorDash recently published an article on how it is solving scaling challenges with content discovery using Elasticsearch and H3, a geospatial indexing system that partitions the world into hexagonal cells. By Tanmay Deshpande.

Systems 52

Systems Design Scalability Architecture 52

SQL Performance

JANUARY 7, 2020

Some of the issues that I think affected performance include the following: Bad file layout choices – I put 8 files in one filegroup, with parallelism but no (or suboptimal) partitioning, spraying I/Os across multiple files with reckless abandon. To distribute those exactly evenly across 8 partitions, that would be 469,370,744.25

Database 52

Database Processing Performance Engineering 52

Scalegrid

AUGUST 22, 2019

A new master was elected in the majority partition. A new master was elected in the majority partition. A new master was elected in the majority partition. Network-isolate the master server from other servers. DCS communication was blocked for master node. PostgreSQL was demoted on the master server.

Availability 277

Availability Servers Network Open Source 277

Dynatrace

APRIL 5, 2024

This example shows the SLI of a performance SLO, targeting response times/loading times of a key-user-action/DOMload. (( (builtin:apps.web.action.domInteractive.load.browser :splitBy("dt.entity.application_method") :avg :filter(and(or(in("dt.entity.application_method", entitySelector("type(~"APPLICATION_METHOD~"),entityId(~"APPLICATION_METHOD-XXX~")"))))) (..)

Efficiency 225

Efficiency Traffic Tuning Metrics 225

Percona

AUGUST 5, 2023

So, What is MySQL Partitioning? Partitioning is a way in which a database (MySQL in this case) splits its actual data down into separate tables but still gets treated as a single table by the SQL layer. When partitioning in MySQL, it’s a good idea to find a natural partition key.

Strategy 40

Strategy Storage Database Best Practices 40