Percona Backup for MongoDB (PBM) is an open source, distributed, and low-impact solution for consistent backups of MongoDB sharded clusters and replica sets.
With PBM v2.0.0, the ability to take physical backups, in addition to logical ones, has been made Generally Available. Physical backups were added since v1.7.0 as a technical preview.
In March 2023, we discovered an issue with physical backups, which, under certain conditions, can cause the restore to fail (PBM-1058).
This article includes additional details about the following:
- The conditions under which bug PBM-1058 is triggered.
- The available workarounds.
- How to permanently resolve the issue and our recommendations to affected users and customers.
Issue description
If you use PBM v2.0.4 or earlier, and your dbPath contains a trailing slash (for example dbPath is /var/lib/mongodb/), the backup process fails to correctly trim the dbPath from the file path.
As a result, when attempting to restore a backup taken in the above scenario, the restore process will fail as it will copy the files into the wrong location inside dbPath. For instance, “fileName” will be copied into dbPath/dbPath/fileName instead of dbPath/fileName, as in the following example:
- dbPath:
- /var/lib/mongodb/
- fileName:
- Index-19-2519978004532031907.wt
- Location of fileName after the restore:
- /var/lib/mongodb/var/lib/mongodb/index-19-2519978004532031907.wt
- Expected location of fileName after the restore:
- /var/lib/mongodb/index-19-2519978004532031907.wt
Am I affected?
You are affected by PBM-1058 if all conditions of one of the following two cases apply to you:
Case one
- You are using PBM v2.0.4 or earlier.
- You are taking physical backups.
- Your mongod’s dbPath option has a trailing slash.
- Example: your dbPath is path-to-some-directory/
Case two
- You are using PBM v2.0.4 or earlier.
- You are taking physical or incremental backups.
- replsets[].files[].filename fields in the backup metadata start with a forward slash /.
Available workarounds
We have identified some workarounds that can be used with PBM v2.0.4 or earlier. The permanent solution to the issue described in this article is available with PBM v2.0.5 (see next section for more details).
If you are using PBM v2.0.4 or earlier and you are affected by PBM-1058, we recommend one of the following workarounds, depending on if you can restart your MongoDB and take a new backup or not.
Workaround one
If you can take a new backup, please follow these steps:
- Plan a short downtime window.
- Change your mongod dbpath (in any script you use to start MongoDB and in its configuration file /etc/mongo.conf) to remove any trailing slash.
- Restart your mongod.
- Take a new physical backup.
At this time, your new physical backup will not be affected by PBM-1058, and its restore will work normally.
Workaround two
If taking a new backup is not possible, and you are in the need of restoring an already taken backup that is affected by PBM-1058, please follow the following steps:
- In the backup meta on storage (<backup_name>.pbm.json) trim the “path-to-some-directory/” prefix in all “filename” fields.
- Move these files on the disk accordingly.
- Example: Move “Timestamp/Shard_0/path-to-some-directory/fileName” to “Timestamp/Shard_0/fileName”
- Run the command pbm config –force-resync.
- Restore the backup.
Permanent issue resolution
The permanent fix for PBM-1058 has been merged via Pull Request #802.
PBM v2.0.5 includes this fix and was released on March 23, 2023. Packages can be downloaded from this link, and release notes are available here.
Starting with PBM v2.0.5, any newly taken backup will not be affected by PBM-1058, even if your dbPath contains a trailing slash.
Furthermore, starting with v2.0.5, during the restore process, PBM will identify if the backup (taken with PBM v2.0.4 or earlier) is affected by PBM-1058. If yes, it will attempt to auto-resolve the issue during the restore process (Pull Request #805).
The previously mentioned workaround two will still be useful in all other cases.
Recommendations
We recommend all PBM users and customers affected by PBM-1058 upgrade to PBM v2.0.5 or higher and take a new backup as soon as possible. This will avoid any possible issues in the future.
Additional questions
For any additional questions, Percona customers can open a new support ticket.
Community users can use the usual community support channels to request help.
Percona Distribution for MongoDB is a freely available MongoDB database alternative, giving you a single solution that combines the best and most important enterprise components from the open source community, designed and tested to work together.
Download Percona Distribution for MongoDB Today!
