In many scenarios, the standard alert notification template in Percona Monitoring and Management (PMM), while comprehensive, may not align perfectly with specific operational needs. This often leads to an excess of details in the notification’s “Subject” and “Body”, cluttering your inbox with information that may not be immediately relevant.
The focus today is on tailoring these notifications to fit your unique requirements. We’ll guide you through the process of editing the “Subject” and “Body” in the PMM UI, ensuring that the alerts you receive are filtered and relevant to your specific business context.
Please note: This post assumes a foundational understanding of basic alerting and configuration in PMM. For those new to these concepts, we recommend consulting the documentation on “SMTP” and “PMM Integrated/Grafana alert” for a primer.
Customizing the “Subject” section of alert notification
1) The default “Subject” will look something like below.
2) Now, let’s proceed to edit the “subject” content.
I) First, we need to create a new message template called “email.subject” in Alerting -> Contact points with the following content.
1 | Template_name: email.subject |
1 2 3 | {{ define "email.subject" }} {{ range .Alerts }} PMM Alert | {{ .Labels.alertname }} | {{ .Labels.node_name }} | {{ .Labels.DB }} {{ end }} {{ end }} |
Here, we are simply using the range to iterate over the alert labels. We loop through the alert labels and extract the alert name and node name.
The provided template is written in Go’s templating language. For a more detailed understanding of the syntax and usage of templates, please refer to the official manual.
II) Then we need to edit the default contact point name inside “Alerting->Contact points”
And define the below “Subject” under “Optional Email Settings”.
1 | {{ template "email.subject". }} |
III) After successfully testing, we can save the changes.
That’s it. Now, if the alert triggers, we will observe a customized subject in the email.
Example:
Customizing the “Body” section of alert notification
1) Let’s first see how the notifications appear with the native alerting. This is a basic notification alert that triggers when the database/MySQL is down. As we can see, it includes additional information, such as various labels and a summary.
2) Now, suppose we want to get rid of some content and want only a few relevant details. This can be achieved by following the below outlined steps.
I) Go to Alerting -> Contact points and add new “Message templates”.
II) Next, create a notification template named “email” with two templates in the content: “email.message_alert” and “email.message”.
The “email.message_alert” template is used to display the labels and values for each firing and resolved alert, while the “email.message” template contains the email’s structure.
1 | Template name: email.message |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | ### These are the key-value pairs that we want to display in our alerts.### {{- define "email.message_alert" -}} AlertName = {{ index .Labels "alertname" }}{{ "n" }} Database = {{ index .Labels "DB" }}{{ "n" }} Node_name = {{ index .Labels "node_name" }}{{ "n" }} Service_name = {{ index .Labels "service_name" }}{{ "n" }} Service Type = MySQL {{ "n" }} Severity = {{ index .Labels "severity" }}{{ "n" }} TemplateName = {{ index .Labels "template_name" }}{{ "n" }} {{- end -}} ### Next, we have defined the main section that governs the alerting and firing rules. ### {{ define "email.message" }} There are {{ len .Alerts.Firing }} firing alert(s), and {{ len .Alerts.Resolved }} resolved alert(s){{ "n" }} ###Finally, the alerts and firing rules are invoked and triggered based on the generated alerts or fixes. ### {{ if .Alerts.Firing -}} Firing alerts:{{ "n" }} {{- range .Alerts.Firing }} - {{ template "email.message_alert" . }} {{- end }} {{- end }} {{ if .Alerts.Resolved -}} Resolved alerts:{{ "n" }} {{- range .Alerts.Resolved }} - {{ template "email.message_alert" . }} {{- end }} {{- end }} {{ end }} |
The above template is written in Go’s templating language. To know more in detail about the syntax and template usage you can refer to the manual.
III) Lastly, simply save the template
3) Next, we will edit the default “Contact points” and define the below content under “Update contact point -> Optional Email settings->Message” for email. Similarly, you can add other channels as well, like Telegram, Slack, etc.
Execute the template from the “message” field in your contact point integration.
1 | {{ template "email.message" . }} |
Percona Alerting comes with a pre-configured default notification policy. This policy utilizes the grafana-default-email contact point and is automatically applied to all alerts that do not have a custom notification policy assigned to them.
Reference:- https://docs.percona.com/percona-monitoring-and-management/use/alerting.html#notification-policies
After verifying a successful test message, we can save the updated contact point.
4) Finally, once the alert is triggered, you will be able to see the customized notification reflecting only the defined key/values.
Moreover, we can also use “LABEL LOOPS” instead of defining the separate “Key/Value” pairs as we did in the above steps. In this way, we can have all the default parameters in iteration without explicitly defining each of them.
Here, we use a range to iterate over the alerts such that dot refers to the current alert in the list of alerts, and then use a range on the sorted labels so dot is updated to refer to the current label. Inside the range, use “.Name” and “.Value” to print the name and value of each label.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | ### applying label loop option ### {{- define "email.message_alert" -}} Label Loop: {{ range .Labels.SortedPairs }} {{ .Name }} => {{ .Value }} {{ end }} {{- end -}} {{ define "email.message" }} There are {{ len .Alerts.Firing }} firing alert(s), and {{ len .Alerts.Resolved }} resolved alert(s){{ "n" }} {{ if .Alerts.Firing -}} Firing alerts:{{ "n" }} {{- range .Alerts.Firing }} - {{ template "email.message_alert" . }} {{- end }} {{- end }} {{ if .Alerts.Resolved -}} Resolved alerts:{{ "n" }} {{- range .Alerts.Resolved }} - {{ template "email.message_alert" . }} {{- end }} {{- end }} {{ end }} |
To add some more options, say (summary and description) in the customized alerts below, template changes can be performed.
I) First, you can add/update the “Summary and annotations” section inside the “alert rule” based on your preference.
II) Then, edit the below Message template (“email.message”) in Alerting->contact points with the updated changes.
1 | Template name: email.message |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | {{- define "email.message_alert" -}} AlertName = {{ index .Labels "alertname" }}{{ "n" }} Database = {{ index .Labels "DB" }}{{ "n" }} Node_name = {{ index .Labels "node_name" }}{{ "n" }} Service_name = {{ index .Labels "service_name" }}{{ "n" }} Service Type = {{ index .Labels "service_type" }}{{ "n" }} Severity = {{ index .Labels "severity" }}{{ "n" }} TemplateName = {{ index .Labels "template_name" }}{{ "n" }} {{- end -}} {{ define "email.message" }} There are {{ len .Alerts.Firing }} firing alert(s), and {{ len .Alerts.Resolved }} resolved alert(s){{ "n" }} {{ if .Alerts.Firing -}} Firing alerts:{{ "n" }} {{- range .Alerts.Firing }} - {{ template "email.message_alert" . }} <strong> - {{ template "alerts.summarize" . }}</strong> {{- end }} {{- end }} {{ if .Alerts.Resolved -}} Resolved alerts:{{ "n" }} {{- range .Alerts.Resolved }} - {{ template "email.message_alert" . }} <strong> - {{ template "alerts.summarize" . }}</strong> {{- end }} {{- end }} {{ end }} <strong>{{ define "alerts.summarize" -}} {{ range .Annotations.SortedPairs}} {{ .Name }} = {{ .Value }} {{ end }}</strong> {{ end }} |
Reference:- https://grafana.com/blog/2023/04/05/grafana-alerting-a-beginners-guide-to-templating-alert-notifications/
Sometimes, the alert notifications might appear in a single line instead of separate lines for all the Keys. Although this is not a regular behavior it can be fixed by using the below changes.
I) Access to the PMM Server
1 | sudo docker exec -it pmm-server bash |
II) Thereafter, you can edit the file:- “/usr/share/grafana/public/emails/ng_alert_notification.html” and replace the text in between lines (288 to 290) as below.
Replace:
1 2 3 | {{ if gt (len .Message) 0 }} <div style="white-space: pre-line;" align="left">{{ .Message }} {{ else }} |
With:
1 2 3 | {{ if gt (len .Message) 0 }} <span style="white-space: pre-line;">{{ .Message }}</span> {{ else }} |
Note: Please ensure to take the backup before making any changes to the PMM Server files. Moreover, these changes could be lost when doing a PMM upgrade, especially when Grafana is upgraded as part of PMM, so a backup of the edited version would also be needed for later restoration purposes.
III) Finally, you can restart the Grafana service.
1 | supervisorctl restart grafana |
Summary
Filtering in alert notifications proves useful in concealing extraneous information from the relevant users. Only the specified elements are displayed in the notification email, thereby preventing unnecessary clutter in the alert content.
Percona Monitoring and Management is a best-of-breed open source database monitoring solution. It helps you reduce complexity, optimize performance, and improve the security of your business-critical database environments, no matter where they are located or deployed.
Download Percona Monitoring and Management Today
