MongoDB v4.2 was released in August 2019. And like previous versions, there is a time to go — and that End of Life (EOL) date approaches.
MongoDB v4.2 EOL is set for April 30, 2023. Since Percona Server for MongoDB (PSMDB) is a fully compatible replacement for the upstream MongoDB Community Edition, we try to follow the same EOL date as the upstream vendor. So let’s discuss why you should upgrade MongoDB, what the upgrade process looks like, and some of the potential gotchas that you may run into along the way.
First, consider why you even need to upgrade MongoDB.
You may think that everything is fine, your application is humming along, you have had no major problems with this version, and you don’t really see a reason to upgrade. And that’s exactly why you want to upgrade; you want to keep it that way. There are multiple very good reasons to upgrade MongoDB.
Those reasons include the following:
- Security, stability, compliance
- Bug fixes
- New features, enhancements
- 3rd party application requirements, code changes
The two most important reasons to upgrade MongoDB are around security and bug fixes.
The goal is always to ensure that your PSMDB/MongoDB environment stays as stable, secure, and performant as possible.
According to Percona’s Software Release Lifecycle Overview, after a version has gone EOL, no new builds or updates will be provided. We will continue to offer Operation Support on Supported Platforms.
Don’t leave your database vulnerable because this version has gone on to the database software wasteland. Plan and execute your upgrade ASAP.
With new versions come new features that may be beneficial to your application. And some just like shiny new things. Some of the major new features that are available in more recent versions of MongoDB are Time Series Collections and Live Resharding in v5.0.
Between v4.2 and v6.0, there are multiple enhancements to take advantage of.
Those include more resilient chunk migrations and the ability to at least refine your shard keys in v4.4, the above-mentioned v5.0 new features, and many changes under the hood to support those major new features. Then with v6.0, you get sync improvements, mandatory use of the new mongosh shell, many enhancements and tweaks to all of the major changes that v5.0 brought about, and a new default chunk size of 128M to help handle some of the recent changes to the auto-splitter, chunk migration, and balancer processes.
But there are quite a few hops between v4.2 and v6.0. So let’s debunk some of the reasons that you may have for NOT upgrading.
Those main potential blockers could include:
- Lack of resources or staff
- Lack of time/can’t afford the downtime
- Legacy code
- Outdated drivers
- Stack incompatibility
But really, the danger of not having support or bug fixes for your business-critical databases and applications is generally a very good reason to overcome all of those blockers sooner rather than later.
Let’s look at the steps required to upgrade MongoDB and the different types of environments that you may have running.
The types of systems include:
- Standalone
- Replica Sets
- Sharded Clusters
Those systems or architectures look like the below:
Now look at the components that make up those systems.
That gives us an idea of what we will need to upgrade.
But we also have to consider the built-in Replication that MongoDB has. That, too, will impact our upgrade process.
Basic overall MongoDB upgrade steps
- Take a backup
- Check prerequisites, resolve as needed
- Add resources if needed
- Download new desired binaries
- Keep FCV (Feature Compatibility Value) set to Current/Previous version
- Shut down mongo processes in the correct order; rolling fashion according to system type
- Replace current binaries with new binaries
- Start up mongo processes in the correct order; rolling fashion as required by system type
- Wait for an amount of time to ensure no problems
- Set FCV to the new desired version
The Pre-Req Checks generally look like this:
Once those are done, begin the upgrade for the standalone system type.
Upgrade a standalone system
It’s a single host or node with one process, so do the following:
- Take a backup
- Shut down the mongod process
- Replace the current binary with the new binary
- Keep FCV at the previous version until everything checks out after the upgrade
- Restart the mongodb process using the original port (default 27017)
- Check version, data, logs, etc.
- When all is good, update FCV
Now let’s look at the steps for the other two system types.
First, the Replica Set upgrade steps:
*Note* that there is a warning there.
You ALWAYS want to upgrade MongoDB to the latest version available – well, after checking the bug list, of course. We’ll review that warning when discussing the “Gotchas” later.
For Replica Set upgrades – upgrade binaries in a rolling manner. Start with the secondaries, then force the election of a new PRIMARY, and then upgrade the binary for the former PRIMARY.
Basic Replica Set upgrade steps
Upgrade Secondaries in a Rolling Method – SECONDARY 1, SECONDARY 2, force election to new PRIMARY, upgrade old PRIMARY/new SECONDARY.
Step 1 – Upgrade SECONDARY 1
– Shutdown Secondary 1
– Take Secondary 1 out of the replica set by restarting it with another port number (ex. port 3333)
– Change Binaries to new version
– Start Secondary 1 back up with its original replica set port number (ex. 27017)
Step 2 – Upgrade SECONDARY 2 – repeat that process with Secondary 2
Shutdown Secondary 2
– Take Secondary 2 out of the replica set by restarting it with another port number (ex. port 3333)
– Change Binaries to new version
– Start Secondary 2 back up with its original replica set port number (ex. 27017)
Step 3 – Upgrade the current PRIMARY
Step Down the current Primary – force election of a new PRIMARY – make sure state is good
Upgrade old PRIMARY, now new Secondary
Shut down old PRIMARY, now new Secondary 3
– Take new Secondary 3 out of the replica set by restarting it with another port number (ex. port 3333)
– Change Binaries to new version
– Start new Secondary 3 backup with its original replica set port number (ex. 27017)
Wait, check the data and logs. If there are no problems, update FCV to the new version
Done.
Now move on to the Sharded Cluster Upgrade process, where there are more components to consider and upgrade.
Reminder – those components are:
*Note* When Upgrading a Sharded Cluster, the order in which you upgrade the components matters.
You have the additional components (the balancer, config server replica set, the main data bearing mongod shard nodes, and the query router process mongoS’s.)
Order Matters … with this system type upgrade.
Stop Balancer
- Upgrade Config Servers
- Upgrade Shard nodes – MongoD’s
- Upgrade the MongoS’s
Start Balancer
Below is an infogram to show that order when upgrading a sharded cluster:
Downgrades
If, for whatever reason, you run into any problem, you can perform a downgrade.
Downgrades basically go in reverse order – more applicable to sharded clusters.
Stop Balancer
- Downgrade MongoS’s
- Downgrade Shard Nodes – MongoD’s
- Downgrade the Config Servers
ReStart Balancer
* This happens all in a rolling manner at the replset level again.
That covers the basic steps.
I will do a more technical series of blogs in the coming months covering running MongoDB in Kubernetes and using the Percona Operator for MongoDB. Those blogs will contain commands, example results, etc., used when managing in that containerized environment. One of the planned blogs will cover upgrades to that environment.
For now, you can see the actual commands used when upgrading a Percona Server for MongoDB replica set or a sharded cluster in Sudhir’s blog.
Now let’s take a look at some of the potential gotchas.
Gotchas — AKA things to watch out for
Whenever you are changing versions via an upgrade, there are different groups of things to watch out for or be aware of.
Those general buckets are:
- Compatibility issues – whether programming language or driver related
- Deprecated functions or tools
- Behavior changes – ex TTL behavior changed significantly between v4.4 and v6
- Bugs – newly discovered or already reported but not fixed yet
Below are some specific examples for those buckets.
Deprecated – Simple Network Management Protocol (SNMP) v6.0
Starting in MongoDB 6.0, SNMP is deprecated and will be removed in the next release.
Deprecated – old mongo shell “mongo” in v5.0
– Some legacy methods unavailable or replaced …
– mongosh uses the same syntax as old mongo shell
– Beware –check your code
Behavior changes
v4.2 –Faster stepdowns
Autosplitter process moved from mongos to the PRIMARY member of the replica set. This led to knowing more truth about chunk balance, so more chunk splits and more chunk migrations. Many more. It caused added write pressure.
v4.4 –Adjustments to the management of jumbo chunks – no longer getting stuck forever due to memory usage limit. Started adding in some of the changes that would be needed for v5.0 when Live Resharding would be hitting.
v5.0 – Deprecated old shell; major changes to the WiredTiger engine and Core Server to support Live ReSharding and Time Series Collections. It took a while for this major release to be fully baked. 😉
v6.0 – Default chunk size increased to 128M to help against too frequent chunk splitting and chunk moves by the Balancer. Removed the old shell. Changes to Initial sync and resync. New Operators; New Functions.
Those are just some of the changes.
More about some of the negative impacts.
Things to watch out for – bugs!
Along with all of the major changes that went into v5.0, there were many bugs for quite a while. Making this even more impactful – the v5.0 changes were backported in v4.4.
So early on for v5.0.x, the same bugs also broke versions of 4.4 from 4.4.1 up through v4.4.8 but really up into v4.9 and v4.10 if you look at the JIRA tickets closely.
Compatibility for Supported Life Cycle and OS Host Software – ex. MongoDB v5.0.x Community Edition drops support for RHEL/CentOS/Oracle 6 on x86_64 and other OS support changes
Here are some screenshots that I took along the way during the many releases for the version.
Make sure to use the latest minor release version when upgrading MongoDB
For v4.4
These various bugs last pretty much ALL the way Through 4.4.8 – Serious bugs that cause checkpoint issues, possible data inconsistencies, missing documents/data loss, duplicate unique key, problems restarting, omitting a page or pages of data, unclean restarts, data loss, etc.
Examples: SERVER-61483 SERVER-61633 SERVER-61945 SERVER-61950 – problems restarting nodes WT-8104 WT-8204 – race conditions, memory leaks. WT-8395 – upgrade related –data in an “inconsistent state”, missing documents WT-8534 WT-8551
Performance impact? There are also a few postings online talking about the newer versions possibly having a negative impact on performance.
There was a recently reported slowdown due to the new mongosh shell and a bug there, but seems to be with certain combinations and using a ruby driver …? Below are links for that issue and bug:
- https://github.com/bitnami/charts/issues/10316#issuecomment-1131718880
- https://jira.mongodb.org/browse/MONGOSH-1240
Ok, enough of that. What should we do?
Upgrade of course!
Slow and Steady wins the race!
And what else?
Migrations!
Consider moving over to Percona Server for MongoDB
That migration — it’s really just a lift and shift.
Normally a binary change between PSMDB and MGDB CE will take care of it. No need for data type changes, and no loss of triggers and procedures to deal with.
Remember, tick-tock … Plan your MongoDB upgrade and migrations today.
Thanks all!
