Compression in any database is necessary as it has many advantages, like storage reduction, data transmission time, etc.
Storage reduction alone results in significant cost savings, and we can save more data in the same space. As the amount of data grows, the need for efficient data compression becomes increasingly important to save storage space, reduce I/O overhead, and improve query performance.
In this blog, we will discuss both data and network-level compression offered in MongoDB. We will discuss snappy and zstd for data block and zstd compression in a network.
Percona Server for MongoDB (PSMDB) supports all types of compression and enterprise-grade features for free. I am using PSMDB 6.0.4 here.
Data compression
MongoDB offers various block compression methods used by the WiredTiger storage engine, like snappy, zlib, and zstd.
When data is written to disk, MongoDB compresses it with a specified block compression method and then writes it to disk. When this data block is read, it decompresses it in memory and presents it to the incoming request.
Block compression is a type of compression that compresses data in blocks rather than compressing the entire data set at once. Block compression can improve performance by allowing data to be read and written in smaller chunks.
By default, MongoDB provides a snappy block compression method for storage and network communication.
Snappy compression is designed to be fast and efficient regarding memory usage, making it a good fit for MongoDB workloads. Snappy is a compression library developed by Google.
Benefits of snappy compression in MongoDB:
- Fast compression and decompression speeds
- Low CPU usage
- A streamable format that allows for quick processing
- Minimal impact on query performance
Zstandard Compression or zstd, another newer block compression method provided by MongoDB starting for v4.2, provides higher compression rates. Zstd is a compression library that Facebook developed.
Zstd typically offers a higher compression ratio than snappy, meaning that it can compress data more effectively and achieve a smaller compressed size for the same input data.
Benefits of zstd compression in MongoDB:
- Higher compression ratios than Snappy
- Highly configurable compression levels
- Fast compression and decompression speeds
- Minimal impact on query performance
To enable zstd block compression, you need to specify the block compressor as “zstd” in the configuration file:
1 2 3 4 5 6 7 8 | storage: engine: wiredTiger wiredTiger: collectionConfig: blockCompressor: zstd blockCompressorQuality: 6 #(available since v5.0) engineConfig: cacheSizeGB: 4 |
In the above example, blockCompressorQuality is set to 6, which is the default.
blockCompressorQuality specifies the level of compression applied when using the zstd compressor. Values can range from 1 to 22.
The higher the specified value for zstdCompressionLevel, the higher the compression which is applied. So, it becomes very important to test for the optimal required use case before implementing it in production.
Here, we are going to test snappy and zstd compression with the following configurations.
Host config: 4vCPU, 14 GB RAM
DB version: PSMDB 6.0.4
OS: CentOS Linux 7
I’ve used mgenerate command to insert a sample document.
1 | mgeneratejs '{"name": "$name", "age": "$age", "emails": {"$array": {"of": "$email", "number": 3}}}' -n 120000000 | mongoimport --uri mongodb://localhost:27017/<db> --collection <coll_name> --mode insert |
Sample record:
1 2 3 4 | _id: ObjectId("64195975e40cea62af1be510"), name: 'Verna Grant', age: 44, |
I’ve created a collection using the below command with a specific block compression method. This does not affect any existing collection or any new collection being created after this.
1 | db.createCollection("user", {storageEngine: {wiredTiger: {configString: "block_compressor=zstd"}}}) |
If any new collection is created in the default manner, it will always be the default snappy or compression method specified in the mongod config file.
At the time of insert ops, no other queries or DML ops were running in the database.
Snappy
Data size: 14.95GB
Data size after compression: 10.75GB
Avg latency: 12.22ms
Avg cpu usage: 34%
Avg insert ops rate: 16K/s
Time taken to import 120000000 document: 7292 seconds
Zstd (with default compression level 6)
Data size: 14.95GB
Data size after compression: 7.69GB
Avg latency: 12.52ms
Avg cpu usage: 31.72%
Avg insert ops rate: 14.8K/s
Time taken to import 120000000 document: 7412 seconds
We can see from the above comparison that we can save almost 3GB of disk space without impacting the CPU or memory.
Network compression
MongoDB also offers network compression.
This can further reduce the amount of data that needs to be transmitted between server and client over the network. This, in turn, requires less bandwidth and network resources, which can improve performance and reduce costs.
It supports the same compression algorithms for network compression, i.e., snappy, zstd, and zlib. All these compression algorithms have various compression ratios and CPU needs.
To enable network compression in mongod and mongos, you can specify the compression algorithm by adding the following line to the configuration file.
1 2 3 | net: compression: compressors: snappy |
We can also use multiple compression algorithms like
1 2 3 | net: compression: compressors: snappy,zstd,zlib |
The client should also use at least one or the same compression method specified in the config to have data over the network compressed, or the data between the client and server would be uncompressed.
In the below example, I am using a python driver to connect to my server with no compression, and zstd compression algorithm
I am doing simple find ops on the sample record shown above.
This is the outbound data traffic without any compression method
Here we can see data transmitted is around 2.33MB/s:
Now, I’ve enabled zstd compression algorithm in both the server and client
1 | client = pymongo.MongoClient("mongodb://user:[email protected]:27017/?replicaSet=rs1&authSource=admin&compressors=zstd") |
Here we can see data avg outbound transmission is around 1MB/s which is almost a 50% reduction.
Note that network compression can have a significant impact on network performance and CPU usage. In my case, there was hardly anything else running, so I did not see any significant CPU usage.
Conclusion
Choosing between snappy and zstd compression depends on the specific use cases. By understanding the benefits of each algorithm and how they are implemented in MongoDB, you can choose the right compression setting for your specific use case and save some disk space.
Choosing the appropriate compression algorithm is important based on your specific requirements and resources. It’s also important to test your applications with and without network compression to determine the optimal configuration for your use case.
I also recommend using Percona Server for MongoDB, which provides MongoDB enterprise-grade features without any license, as it is free. You can learn more about it in the blog MongoDB: Why Pay for Enterprise When Open Source Has You Covered?
Percona also offers some more great products for MongoDB, like Percona Backup for MongoDB, Percona Kubernetes Operator for MongoDB, and Percona Monitoring and Management.
Percona Distribution for MongoDB is a freely available MongoDB database alternative, giving you a single solution that combines the best and most important enterprise components from the open source community, designed and tested to work together.
Download Percona Distribution for MongoDB Today!
