When a Monitors alert fires, the dashboard or query that explains it is now one click away on the incident. Between October 9 and 15, Monitors also added built-in time variables and a display time zone to dashboards, a search across all dashboards, and Go templates in related and recovery queries.
Investigation entries on alert rules
The alert rule form has a new sixth block, Investigation entries. Each rule can carry up to 20 of them:
- Add dashboard entry opens a dashboard, with values for its variables.
- Add query entry opens a query with the data source, query type, and query statement you set. Leave the data source empty to use the alert event's data source.
- Variable values can reference the alert's labels (
$labels) and its data source ($datasource). - Each entry has a time range in minutes before and after the alert. New entries default to 60 minutes before and 30 minutes after.
Preview link tries an entry before you save. If the entry has no {{ template, the preview uses the values in the form, unsaved edits included, and the current time, so the rule doesn't need an active alert. An entry that uses {{ }} still previews against one of the rule's active alerts.
Where the entries show up
Incidents and alerts from a Monitors alert rule show an Investigation section in the right column of the detail page. It lists the first three entries and expands to the rest. The same entries appear in the Investigate popover on related alert rows and in the Monitors active alert detail.
Entries come from a snapshot taken when the alert fires. Editing the rule later doesn't change the entries of alerts that already exist, and alerts from before the entries were configured have none.
On-premises deployments: a rule with investigation entries can only be saved when every monit-edge in the cluster of the matched data sources runs v0.75.3 or later.
Dashboards: time variables, display time zone, and search
- Panel queries accept four built-in time variables:
$__interval,$__rate_interval,$__range, and$__auto. The panel's own window and step fill them in as seconds. Which ones you can use depends on the data source and query mode, and custom variable names can no longer start with__. - The bottom of the time picker shows the current time zone. Change time zone switches between Default (your profile time zone), browser time, UTC, and cities grouped by continent.
- Search all dashboards at the top right of the dashboard list searches every folder you can read, 20 results per page.
Go templates in related and recovery queries
Related and recovery query expressions accept {{ $labels.<label name> }} in addition to ${label_name}. The template renders once per alert instance with that instance's labels. One expression can't mix both syntaxes. On-premises deployments need every registered monit-edge in the cluster on v0.75.4 or later to save a rule that uses Go templates.
Get started
Go to Flashduty console → Monitors → Alert rules, open a rule, and add entries under Investigation entries.
Product docs: Investigation entries, Dashboards, Prometheus alert rules.