Percona Alerting was introduced in Percona Monitoring and Management (PMM) 2.31, and it brought a range of custom alerting templates that makes it easier to create Alert rules to monitor your databases.
In this article, we will go over how to set up Alerting in PMM running on Docker and receive notifications via emails when such alerts are triggered. We will also leverage Grafana’s notification templates to customize the alert emails to reduce noise.
Configuring PMM with Grafana SMTP
To configure SMTP for Grafana, we need to provide the following environment variables:
- GF_SMTP_ENABLED: When true, enables Grafana to send emails.
- GF_SMTP_HOST: Host address of your SMTP server.
- GF_SMTP_USER: Username for SMTP authentication.
- GF_SMTP_PASSWORD: Password for SMTP authentication
- GF_SMTP_SKIP_VERIFY: When true, verifies SSL for the SMTP server.
- GF_SMTP_FROM_ADDRESS: Email address to be used when sending out emails.
- GF_SMTP_FROM_NAME: Name to be used when sending out emails.
Create a .env file in your preferred working directory and add your SMTP credentials as follows:
1 2 3 4 5 6 7 | GF_SMTP_ENABLED=true GF_SMTP_HOST=smtp.gmail.com:587 GF_SMTP_USER=<YOUR_EMAIL@DOMAIN.COM> GF_SMTP_PASSWORD=<YOUR_SMTP_PASSWORD> GF_SMTP_SKIP_VERIFY=false GF_SMTP_FROM_ADDRESS=<YOUR_EMAIL@DOMAIN.COM> GF_SMTP_FROM_NAME=Percona Alerting |
If you are using your Gmail’s SMTP credentials, as shown above, you will have to generate an app password and fill it in as the value of your $GF_SMTP_PASSWORD variable.
Next, start your PMM instance with the created .env file using the docker command below:
1 | docker run --env-file=.env -p 443:443 -p 80:80 --name=pmm-alerting percona/pmm-server:2 |
Adding MongoDB services
With our PMM instance up and running, let us add a new MongoDB service for PMM to monitor. Start up a MongoDB container in Docker with the command below:
1 | docker run -d -p 27017:27017 -e MONGO_INITDB_ROOT_USERNAME=admin -e MONGO_INITDB_ROOT_PASSWORD=admin --name mongo-pmm-alerting percona/percona-server-mongodb --bind_ip=0.0.0.0 |
Next, run the Docker command below to get the Host IP. This is needed as we will use it as our host address in the next step.
1 | docker network inspect bridge -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}' |
In your PMM dashboard, go to the Add Instance page on PMM ($HOST/graph/add-instance) and click the MongoDB tile.
On the next page, fill in the details of your MongoDB instance. The Hostname is the Host IP we got from the docker command above, while the username and password are the values of MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD we set while starting the Mongo container.
NB: Using the default root credentials is not recommended in production environments, and you can find details on setting up a user for PMM in the Percona documentation.
You can verify the newly added service on the PMM inventory page (at http://localhost/inventory), which should look like the one below: Don’t worry about the ‘N/A’ status of the newly added service, though, as it typically flips to ‘OK’ after a few health checks are performed.
Customizing emails with notification templates
By default, Alert emails from Grafana can get noisy as it includes all the alert labels in the subject line (among other things). To tone things down, we will create a new subject template for our own alerts. Head over to the Contact Points tab in the Alerting section and click the New Template button. In the New Template page, use pmm_subject as your template name and add the code below in the “Content” field:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | {{ define "pmm_subject" }} [{{ .Status | toUpper }} {{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }} {{ if gt (.Alerts.Resolved | len) 0 }} RESOLVED:{{ .Alerts.Resolved | len }} {{ end }} {{ end }}] {{ range .CommonLabels.SortedPairs -}} {{ if eq .Name "alertname" }} {{ .Value }} {{ end }} {{ if eq .Name "cluster" }} - {{ .Value }} {{ end }} {{ if eq .Name "service_name" }} - {{ .Value }} {{ end }} {{ end }} {{ end }} |
In the template above, we define the name (using the same name from the template name field). We then extract the alert status (Firing or Resolved) as well as the number of alerts in each status. In the second section, we loop through a sorted list of the alert labels and extract the alert name, cluster name, and service name in order. That way, our subject line is in the form:
1 | FIRING: 1 MongoDB is down - prod-cluster - mongo-eu-1 |
Where prod-cluster is our cluster name, and mongo-eu-1 is our service name.
To learn more about Grafana’s notification templates and Go templating generally, you can check out the Grafana documentation and the template package documentation, respectively. Our next step is to register the created template with Grafana. Click the edit icon beside the default contact point and update the email address(es) on the settings page.
Next, expand the optional settings and add the following text to the subject field:
1 | {{ template "pmm_subject" . }} |
This will register the template we created earlier and direct Grafana to use it as alerting email subject. At this point, you can go ahead and test your configuration using the “Test” button on the contacts point settings page.
To get a feel of a real alert, though, we will set up an Alert rule based on the Mongo down Alerting template that comes with PMM.
Alert Rules from Percona Alert Templates
PMM includes a set of default templates with common events and expressions for alerting. These templates can then be used as a basis for your alert rules (or even your own templates), and you can find them in the Alert Rule Templates tab. Our next step is to create an alert rule based on one of these templates, specifically, the “Mongo down” alerting template. Click the “+” icon beside the template name (highlighted below), and it will take you to the “New Alert rule page”.
Most of the fields are already pre-filled (since the rule is based on a template), and we only need to specify a Folder/Alerting Group. In our case, we will go with the “MongoDB” folder. Select that and click the “Save” button.
Our new alerting rule is now ready and should appear as “Normal” in the “Alert Rule” tab.
To trigger the alert specified in our rule from the last section, stop the MongoDB docker container and wait for 60 seconds (the default evaluation time set for the alert). The alert should now be firing on both your PMM dashboard and if your email was set up correctly, you’ll also get an email about it.
Conclusion
Percona Alerting templates further help simplify the process of setting up alerting for your environment. In this blog post, we’ve reviewed how to do that, for example, in a MongoDB setup. You can install PMM and further explore Alerting, or ask questions/give your feedback on the PMM forums.
Percona Monitoring and Management is a best-of-breed open source database monitoring solution. It helps you reduce complexity, optimize performance, and improve the security of your business-critical database environments, no matter where they are located or deployed.
Download Percona Monitoring and Management Today
