Since Percona Monitoring and Management (PMM) 2.40.0, Grafana uses PostgreSQL instead of SQLite database to store users, dashboards, and other persistent data. This aids in getting some useful information related to [configured alerts] directly via querying from the Postgres backend.
Sometimes we need to migrate only alert rules from one PMM server to another for some requirement or testing/troubleshooting purpose. So in order to cater to such requirements, we can now directly copy such alerts and restore them to the target PMM instance via using cURL request and the PostgreSQL database.
This will be a short blog post just focusing on the steps to perform the export/import of Grafana alert rules.
Let’s see the steps in detail.
1. We need to connect to the source PMM server and get the alert rule “uid” detail from the Postgres backend to pass to the cURL option. The below single command will do all this.
1 2 3 4 5 6 | shell> docker exec -it pmm-server bash -c 'psql -t -U grafana -c "SELECT uid FROM alert_rule" | while read uid; do curl -k -u admin:admin "https://localhost:443/graph/api/v1/provisioning/alert-rules/${uid}" -H "Content-Type: application/json" -o "/tmp/${uid}.dump.json"; done;' |
So, here we have exported all the alert rules in the *.dump.json file via the cURL GET request.
For each alert rule, we see an individual file with a “uid” name prefix. If we are interested in only a specific alert, then we can modify the Postgres query accordingly OR put only the needed [uid] in the curl option.
1 2 | -rw-r--r--. 1 postgres postgres 909 Feb 14 12:52 9mnCxP2Ik.dump.json -rw-r--r--. 1 postgres postgres 736 Feb 14 12:52 lN95bPhSk.dump.json |
2. Then, from the host machine, we can copy the target file generated inside the PMM container.
1 | shell> docker cp pmm-server:/tmp/9mnCxP2Ik.dump.json . |
3. We can also pass the above fetched [uid] details in the mentioned PMM Url in order to view the alert rules.
1 | https://localhost:8443/graph/api/v1/provisioning/alert-rules/9mnCxP2Ik |
Output:
1 | {"id":11,"uid":"9mnCxP2Ik","orgID":1,"folderUID":"cTAiNxeVz","ruleGroup":"default-alert-group","title":"pmm_mysql_too_many_connections Alerting Rule","condition":"A","data":[{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"PA58DA793C7250F1B","model":{"expr":"max_over_time(mysql_global_status_threads_connected[5m]) / ignoring (job)nmysql_global_variables_max_connectionsn* 100nu003e bool 80","instant":true,"intervalMs":1000,"maxDataPoints":43200,"refId":"A"}}],"updated":"2024-02-14T12:49:41Z","noDataState":"OK","execErrState":"Alerting","for":"5m","annotations":{"description":"{{ $value }}% of connections (more than 80%) are usednby {{ $labels.service_name }} on {{ $labels.node_name }}.","summary":"MySQL too many connections ({{ $labels.service_name }})"},"labels":{"percona_alerting":"1","severity":"warning","template_name":"pmm_mysql_too_many_connections"}} |
4. Finally, to import these rules to the target instance, we can simply run the below cUR POST method. This will upload/provision alerts for all the existing files; however, we can restrict it by defining the exact JSON file if we want to import only a specific file.
1 2 3 4 5 6 | for file in /tmp/*.json; do curl -k -u admin:admin -X POST https://127.0.0.1:443/graph/api/v1/provisioning/alert-rules -H 'Content-Type: application/json' -d @"$file"; done; |
OR
1 2 3 4 5 6 7 | for file in /tmp/*.json; do curl -k -X POST https://127.0.0.1:443/graph/api/v1/provisioning/alert-rules -H 'Authorization: Bearer eyJrIjoiZE9EOTF5UFJZZnJmSThXMnI3TmJHeTYxU0NibHg2c2kiLCJuIjoiYWRtaW5fYXBpIiwiaWQiOjF9' -H 'Content-Type: application/json' -d @"$file"; done; |
For more secure way interaction, you can generate the PMM API Key [eyJrIjoiZE9EOTF5UFJZZnJmSThXMnI3TmJHeTYxU0NibHg2c2kiLCJuIjoiYWRtaW5fYXBpIiwiaWQiOjF9] using this link.
Output:
1 | {"id":11,"uid":"9mnCxP2Ik","orgID":1,"folderUID":"cTAiNxeVz","ruleGroup":"default-alert-group","title":"pmm_mysql_too_many_connections Alerting Rule","condition":"A","data":[{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"PA58DA793C7250F1B","model":{"expr":"max_over_time(mysql_global_status_threads_connected[5m]) / ignoring (job)nmysql_global_variables_max_connectionsn* 100nu003e bool 80","instant":true,"intervalMs":1000,"maxDataPoints":43200,"refId":"A"}}],"updated":"2024-02-14T13:16:31.25588241Z","noDataState":"OK","execErrState":"Alerting","for":"5m","annotations":{"description":"{{ $value }}% of connections (more than 80%) are usednby {{ $labels.service_name }} on {{ $labels.node_name }}.","summary":"MySQL too many connections ({{ $labels.service_name }})"},"labels":{"percona_alerting":"1","severity":"warning","template_name":"pmm_mysql_too_many_connections"},"provenance":"api"}{"id":10,"uid":"lN95bPhSk","orgID":1,"folderUID":"cTAiNxeVz","ruleGroup":"default-alert-group","title":"pmm_mysql_down Alerting Rule","condition":"A","data":[{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"PA58DA793C7250F1B","model":{"expr":"sum by (service_name, node_name) (mysql_up) == bool 0","instant":true,"intervalMs":1000,"maxDataPoints":43200,"refId":"A"}}],"updated":"2024-02-14T13:16:31.31848175Z","noDataState":"OK","execErrState":"Alerting","for":"10s","annotations":{"description":"MySQL {{ $labels.service_name }} on {{ $labels.node_name }} is down.","summary":"MySQL down ({{ $labels.service_name }})"},"labels":{"percona_alerting":"1","severity":"critical","template_name":"pmm_mysql_down"},"provenance":"api"}{"message":"invalid alert rule: no queries or expressions are found","traceID":""}[postgres@6ec3c49fb156 opt] # |
There is one caveat here: provisioned alerts are not editable via PMM UI. That means we can view and export them, however, we won’t be able to perform any changes via UI. We have to rely on doing manual changes in the Alert files and then updating them via a PUT request.
To address this issue and to allow editing for provisioned alerts as well as a Header option [“X-Disable-Provenance”] implemented in [“Grafana v9.3”]:- https://github.com/grafana/grafana/pull/58410 and https://github.com/grafana/grafana/issues/57911 which can be used with below request formats.
1 2 | POST /api/v1/provisioning/alert-rules PUT /api/v1/provisioning/alert-rules/{UID} |
Since Grafana 9.4, it’s possible to export the alert rule for provisioning directly via the UI itself.
Unfortunately, PMM’s latest version [2.41.2] is still based on old Grafana v9.2.20, so such a solution is still not feasible from the PMM UI. However, the Grafana version change is planned for [PMM v3], so hopefully, we will see improvement in this area.
Note: The scenario we have shown above will not copy the [Contact points] and any other policies [notification policies] etc. Only [Alert Rule] will be affected. So, in order to change or migrate other parts of the Alerting process, you can use separate URI/endpoint
E.g.,
For Contact points, the below APIs/Method can be used.
1 2 | GET /api/v1/provisioning/contact-points/export POST /api/v1/provisioning/contact-points |
More references
- https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/export-alerting-resources/#export-api-endpoints
- https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/http-api-provisioning/
Conclusion
So, here we discussed one of the approaches to simply migrate the alert rule from one PMM server to another. However, we can always perform the full migration by using backups from the old instance and restoring them to the new instance. Specifically for the latest PMM Version [2.40.0 or above], we can also use some workaround like copying the alert* specific tables from Postgres and restoring them to the target instance, but this is usually not the best-recommended approach and could affect/break the existing alerting in case using any or if there’s any change in the table formats. So, better to test before using it!!
Percona Monitoring and Management is a best-of-breed open source database monitoring solution tool for use with MySQL, PostgreSQL, MongoDB, and the servers on which they run. Monitor, manage, and improve the performance of your databases no matter where they are located or deployed.
Download Percona Monitoring and Management Today
