All Things Distributed

JULY 17, 2015

Data compression today is still as important as it was in the early days of computing. We have seen a shift from generic compression to compression for specific file types, especially those in images, audio and video. Data compression , D.A. Hirschberg, Data compression, Computing Surveys 19,3 (1987) 261-297.

Storage 104

Storage 104

DZone

OCTOBER 16, 2023

Determining the most appropriate data types to store the information depends on various factors, including the required precision of float-point values, the content of the values (such as text), compressibility, and query speed. Choosing the right data types in PostgreSQL can significantly impact your database's performance and efficiency.

Best Practices 272

Best Practices Speed Database Efficiency 272

n0derunner

FEBRUARY 17, 2019

Compressability. What space savings should you expect when running databases with default compression in a Nutanix cluster? When we ran the TPCx-HCI benchmark on our cluster we realized about 2:1 savings from compression alone. Compression+Encryption. The uncompressed data size was about 45TB. Data generation.

Database 40

Database Benchmarking Engineering Development 40

DZone

AUGUST 25, 2023

This means axum gets timeouts, tracing, compression, authorization, and more, for free. One of the components of my OpenTelemetry demo is a Rust application built with the Axum web framework. In its description, axum mentions: axum doesn't have its own middleware system but instead uses tower::Service.

Systems 231

Systems 231

SQL Performance

JANUARY 7, 2020

In part 1 , I showed how both page and columnstore compression could reduce the size of a 1TB table by 80% or more. Since the regular columnstore index didn't compress better than page compression on this data set , and took 13 hours longer to get there, I'll focus solely on the more advanced solution using COLUMNSTORE_ARCHIVE compression.

Database 52

Database Processing Performance Engineering 52

CSS Wizardry

APRIL 22, 2020

One of the more fundamental rules of building fast websites is to optimise your assets, and where text content such as HTML, CSS, and JS are concerned, we’re talking about compression. For a more detailed breakdown of the numbers, see the Compression section of the Web Almanac. Gzip is tremendously effective. That’s a 2.8× Round Trips.

Latency 269

Latency Servers Website Speed 269

DZone

DECEMBER 18, 2023

Each topic is discussed in detail, covering relevant techniques, challenges, and future directions while highlighting notable research contributions and their implications.

Energy 169

Energy Efficiency Systems IoT 169

Percona

JANUARY 3, 2024

port=3306 gtid_port=0 status="ONLINE" weight=1 compression=0 max_connections=1000 max_replication_lag=600 use_ssl=0 max_latency_ms=0 }, { hostgroup_id=2 hostname="192.0.0.2" Config file backup Mysqldump Physical snapshot Here will consider ProxySQL configured for async replication with Primary and Replica as shown below in topology.

Traffic 107

Traffic Hardware Cache Servers 107

DZone

OCTOBER 6, 2021

Network Compression. But along with advantages and uses, computer vision has its challenges in the department of modern applications, which deep neural networks can address quickly and efficiently. With the soaring demand for computing power and storage, it is challenging to deploy deep neural network applications.

Network 258

Network Storage Efficiency 258

Dynatrace

DECEMBER 13, 2023

This can be achieved by reducing the size of files or images, using caching, and compressing data. Optimizing images and videos by reducing their size, using compression, or using lazy loading techniques can improve app performance. Optimize images and videos. Optimize battery life.

Best Practices 197

Best Practices Mobile Metrics Performance 197

DZone

JANUARY 14, 2021

At the start of the year, I looked into how to better compress the output of a Jekyll site. Here’s a small program to read in a file, compress it and save it as a gzip file. I’ll write up the results to that soon. For now, here’s how to gzip a file using Ruby. Enter zlib. It is used to read and write files in gzip format.

Programming 124

Programming Performance 124

DZone

FEBRUARY 20, 2021

Have you ever wanted to know: How do we compress something, without losing any data? Why do some things compress better than others? Suppose we want to compress a string (Huffman coding can be used with any data, but strings make good examples). Huffman Coding: Why Do I Care? How does GZIP work? In 5 Minutes or Less.

Code 133

Code Programming 133

The Netflix TechBlog

NOVEMBER 9, 2021

To do so, the team evaluated different tools provided by the encoder, with the goal of optimizing the tradeoff between compression efficiency and computational efficiency. With multiple iterations, the team arrived at a recipe that significantly speeds up the encoding with negligible compression efficiency changes.

Media 246

Media Open Source Efficiency Software Engineering 246

The Netflix TechBlog

NOVEMBER 17, 2022

While conventional video codecs remain prevalent, NN-based video encoding tools are flourishing and closing the performance gap in terms of compression efficiency. Since we cannot directly optimize for a conventional video codec, which is non-differentiable, we exclude the effect of lossy compression in the loop.

Network 292

Network Media Innovation Efficiency 292

Percona

SEPTEMBER 25, 2023

ProxSQL stops routing connections to replicas when the lag is greater than 600. Replica_127.0.0.2>stop >stop replica; Query OK, 0 rows affected (0.01 sec) Still, ProxySQL routes connections to replica, though replication is stopped. Replica_127.0.0.2>stop >stop replica; Query OK, 0 rows affected (0.01 Replica_127.0.0.2>start

Traffic 91

Traffic Servers Monitoring Processing 91

n0derunner

FEBRUARY 15, 2023

TL;DR If your storage system implements inline compression, performance results with small IO size random writes with time_based and runtime may be inflated with fio versions < 3.3 due to fio generating unexpectedly compressible data when using fio’s default data pattern.

Performance 59

Performance Storage Systems 59

Percona

SEPTEMBER 19, 2023

Option 1: Login as the MySQL OS user: shell> sudo -H -u mysql bash Execute: shell> xtrabackup --no-lock=1 --compress --parallel=4 --host=localhost --user=root --password='password_string' --backup=1 --target-dir=/Backup/21Aug 2> /tmp/b0-with-redo-archiving-as-mysql-os-user.log 1.2) 29 based on MySQL server 8.0.34

C++ 102

C++ Servers Storage Speed 102

The Netflix TechBlog

FEBRUARY 5, 2020

AV1 is a high performance, royalty-free video codec that provides 20% improved compression efficiency over our VP9† encodes. While our goal is to roll out AV1 on all of our platforms, we see a good fit for AV1’s compression efficiency in the mobile space where cellular networks can be unreliable, and our members have limited data plans.

Media 160

Media Open Source Mobile Hardware 160

The Netflix TechBlog

FEBRUARY 5, 2020

AV1 is a high performance, royalty-free video codec that provides 20% improved compression efficiency over our VP9† encodes. While our goal is to roll out AV1 on all of our platforms, we see a good fit for AV1’s compression efficiency in the mobile space where cellular networks can be unreliable, and our members have limited data plans.

Media 160

Media Open Source Mobile Hardware 160

Percona

JANUARY 19, 2023

Compression: Compression is the process of restructuring the data by changing its encoding in order to store it in fewer bytes. There are many compression tools and algorithms for data out there. MySQL supports native compression for InnoDB tables using the Zlib library with the LZ77 compression algorithm.

Open Source 86

Open Source Storage Database Big Data 86

Dynatrace

NOVEMBER 6, 2023

ActiveGate acts as a secure proxy and compresses and routes observability signals in an optimized manner to Dynatrace servers. The Kubernetes-based app platform can be deployed inside or outside a data center using distributed OpenShift topologies alongside even smaller Red Hat Device Edge.

Retail 259

Retail Analytics Storage Cloud 259

O'Reilly

MARCH 26, 2024

Compression, Transformation, Hallucination, and Generation Training data isn’t stored in the model per se, but large chunks of it are reconstructable given the correct key (“prompt”). There are lots of conversations about whether or not LLMs (and machine learning, more generally) are forms of compression or not.

Artificial Intelligence 110

Artificial Intelligence Social Media Google Internet 110

The Netflix TechBlog

APRIL 17, 2019

A brief look into the history of video standards Historically, video compression standards have been developed by two international standardization organizations, ITU-T and MPEG (ISO). 264/AVC, currently, the most ubiquitous video compression standard supported by modern devices, often in hardware. The success was repeated by H.264/AVC,

Open Source 166

Open Source Scalability Speed Efficiency 166

The Netflix TechBlog

SEPTEMBER 18, 2020

They are continuously innovating compression algorithms to efficiently send high quality audio and video files to our customers over the internet. Chris] There’s a lot of things to consider when we roll out a new compression algorithm. Tell me about some of the exciting projects you’re a part of. Which devices get this treatment?

Analytics 223

Analytics Education Innovation Engineering 223

The Netflix TechBlog

AUGUST 28, 2020

Compression and bitrate ladder improvements Even before testing the new streams in the field, we observe the following advantages of the optimized ladders vs the fixed ladders, evaluated over 100 sample titles: Computing the Bjøntegaard Delta (BD) rate shows 50% gains on average over the fixed-bitrate ladder.

Network 330

Network Storage Innovation Mobile 330

Percona

MARCH 29, 2023

Compression in any database is necessary as it has many advantages, like storage reduction, data transmission time, etc. As the amount of data grows, the need for efficient data compression becomes increasingly important to save storage space, reduce I/O overhead, and improve query performance. provides higher compression rates.

Storage 106

Storage Network Open Source Latency 106

Dynatrace

MARCH 6, 2024

The version numbers of these binaries correlate 1:1 with the container image’s tags.

Cloud 212

Cloud Monitoring Java Infrastructure 212

The Netflix TechBlog

MARCH 5, 2024

Memory overhead We expected that losing compressed references on heaps < 32G, due to colored pointers requiring 64-bits object pointers , would be a major factor in the choice of a garbage collector.

Latency 228

Latency Java Tuning Efficiency 228

CSS Wizardry

SEPTEMBER 28, 2023

Honestly, the filesizes you’re aiming for are so small at this point that you’ll probably struggle to get as low as 0.055BPP anyway, but it just seems wise to build in 10% of buffer in case any intermediaries attempt to compress your images further. Instead, use a service like Cloudinary to size and compress images on the fly.

Games 275

Games Ecommerce Cache Systems 275

Dynatrace

JULY 2, 2020

In combination with ActiveGates, network zones save bandwidth and infrastructure costs and by: compressing OneAgent traffic. OneAgent traffic is compressed before it’s sent to the Dynatrace Cluster. optimizing traffic routing. preventing unrelated traffic between data centers and regions.

Network 163

Network Traffic Tuning Infrastructure 163

KeyCDN

MAY 6, 2020

The feature WebP caching and Brotli compression is now also available for Push Zones! Brotli compression is now also enabled for all Push Zones by default which results in great performance improvements. Brotli compression is now also enabled for all Push Zones by default which results in great performance improvements.

Cache 93

Cache Mobile Website Processing 93

Percona

MARCH 24, 2023

You might want to use binlog compression with MySQL/ Percona Server for MySQL , but it can come with drawbacks. This article discusses the need for using binlog compression and the potential issues it may cause. Binlog compression is a technique used to reduce the size of binary log files, which can become quite large over time.

Open Source 83

Open Source Database C++ Servers 83

Percona

JANUARY 24, 2023

Attempts to compress PostgreSQL WAL at different levels have always been around since the beginning. Some of the built-in features ( wal_compression ) have been there since 2016, and almost all backup tools do the WAL compression before taking it to the backup repository. What is Full Page Writes?

Database 115

Database Benchmarking Open Source Latency 115

Percona

DECEMBER 23, 2022

This is a tribute to the efficient compression capabilities of MyRocks. A keen observer will quickly realize the compressed dataset size with MyRocks is roughly the same as the amount of memory of the virtual machine. I wanted most of the memory to be available for the file cache, where the compressed SST files will be cached.

Storage 55

Storage Cache C++ Virtualization 55

SQL Performance

APRIL 17, 2020

The new abilities apply to row-store tables in the following circumstances: Compression must be enabled: On all indexes and partitions , including the base heap or clustered index. Either ROW or PAGE compression. Indexes and partitions may use a mixture of these compression levels. Compression. Integer Example.

Servers 63

Servers Database Testing Efficiency 63

Percona

DECEMBER 28, 2023

Compress the audit log files to reduce the storage space occupied with log files. Encrypt the audit log files by using AES-256 encryption. Dynamically enable or disable the auditing. A server restart is not required to add or adjust existing filtering rules.

Open Source 102

Open Source Database Servers Storage 102

The Netflix TechBlog

DECEMBER 2, 2019

To make this happen, we developed an effective data compression technique by cleverly bucketing our data. The development of an effective data compression strategy enabled us to deploy bootstrapping methods at dramatically greater scale, allowing experimenters to analyze their A/B test results faster and with clearer insights.

Metrics 143

Metrics Strategy Efficiency Testing 143