In an earlier article about log monitoring, How Nightingale Log Monitoring Gets Raw Log Content, we explained why the open-source version of Nightingale cannot include raw log content in log-monitoring alert events. We also mentioned that Flashduty can do this, although the configuration is slightly more involved. This article demonstrates how to configure Flashduty log monitoring so the alert includes the original log content.
Prepare Data
For ElasticSearch log monitoring, Flashduty supports ElasticSearch SQL syntax. My demo data looks like this:
The query targets indexes with fc-insight as the prefix, searches the last 5 minutes, and adds a status = 502 condition.
Alert Rule
Log alerting commonly has two patterns. One counts the number of log lines containing a keyword or matching a query condition such as Level=ERROR. The other treats logs as structured data, extracts values from specific columns, and performs aggregation, such as average or p99 response time for an API.
In this example, I want to monitor the number of logs with status = 502 in the last 5 minutes. If the count is greater than 0, trigger an alert. The alert rule can be configured as:
select count(1) as total from "fc-insight*" where "@timestamp" > now() - INTERVAL 5 MINUTES and status = 502
After this alert fires, we know how many log lines match the condition, but we still do not know the actual log content. To include it, use an associated query. In this example, add another query to fetch the first matching raw log line and render it into the alert event description:
select * from "fc-insight*" where "@timestamp" > now() - INTERVAL 5 MINUTES and status = 502 limit 1The limit 1 in the associated query ensures that only one row is returned. Otherwise, if many logs match the condition, all of them may be inserted into the alert event, making the alert far too long.
The note description references the associated query result:
{{- range $x := $relates.R1}}
request_id: {{$x.Fields.request_id}} , remote addr: {{$x.Fields.remote_addr}}
{{- end}}This displays the request_id and remote_addr fields. You can include more fields based on your actual log schema.
$relates corresponds to the relate_values shown above. After the alert is generated, click the Triggered button on the alert rule to view the associated query result.
The final alert event includes the request_id and remote_addr values:
Closing
Flashduty's associated query feature can include raw log content in alert events, which is very useful for log monitoring. If your current log monitoring tool cannot include raw log content, consider using Flashduty or implementing a similar mechanism. I hope this example gives you a practical reference.
