More and more companies are bringing SRE Agents into production troubleshooting. The usual setup is an "Analyze with AI" button on the alert detail page: an engineer gets an alert, clicks the button, and the AI starts querying metrics, logs, and traces, then reports back what it found.
That works fine during the day. At 3 a.m., though, it's a step too slow. The on-call engineer gets woken up by a phone call, opens a laptop, clicks the button, and then waits for the AI. What you really want is for the analysis to kick off the moment the incident starts, so that by the time the engineer is out of bed, the AI already has initial findings and a direction to investigate.
Automatic triggering runs into two problems pretty quickly. First, many companies have never cleaned up their alerts and generate a huge volume of them every day. If the AI analyzes every single one, the token bill gets large fast. Second, and worse, there are alert storms. A batch of alerts fires at once, each one kicks off an analysis, and every agent goes off to query the same metrics, logs, and traces. That burst of concurrent queries can take down data sources that are already under pressure. The incident isn't fixed yet, and your troubleshooting has already made things worse.
So before you turn on automatic analysis, there are two things worth doing first.
Start with alert governance
An alert should be actionable: when someone receives it, there's a clear action they can take. If an alert has fired countless times and everyone's reaction is "ignore it for now," the rule needs a second look. Maybe the threshold needs tuning, maybe it should notify at a lower severity, or maybe it shouldn't be an alert at all.
AI can help here. Use "actionable" as the guiding principle, and have the AI read the alert rules, their firing history, and the code of the service they cover. Ask it to work out what each alert actually signals, whether it needs a human, and what that human could do about it. This tends to work well, because the AI can use the code to understand what an alert means for the business instead of guessing from the metric name. That said, decisions about how to change a rule should still be signed off by engineers who know the service.
Then reduce alert noise
Governance cuts down alerts that shouldn't exist in the first place. But even when every rule makes sense, a single failure can still set off a lot of alerts. If a database has a problem, every service that depends on it may start throwing errors. Analyzing those alerts one by one burns tokens on duplicate work and sends multiple agents to query the same data sources.
This is exactly what on-call products like Flashduty are built for. In Flashduty, alert data typically goes through two layers of aggregation:
Events reported by monitoring systems
repeated firings and recoveries of the same alert
│ event aggregation
▼
Alert
related alerts, e.g. a database and its dependent services
│ alert aggregation
▼
Incident
│ triggers AI analysis
▼
one initial investigationThe first layer is event aggregation: repeated events for the same alert within the aggregation window are merged into one alert. The second layer is alert aggregation: related alerts are grouped into one incident, which the on-call engineer handles as a single unit. Aggregation has to be configured around services, resources, and similar attributes. You shouldn't lump unrelated problems together just to get the numbers down.
Automatic AI analysis can then be triggered at the incident level, so one analysis reads all the related alerts in that incident instead of starting a separate run for each alert.
Even after alerts are aggregated into incidents, a storm can still produce several similar incidents. Flashduty's automatic incident analysis adds one more layer for this: similar-incident merging. When an automation rule receives a new incident that is similar enough to one that is currently being analyzed or was analyzed recently, the new incident joins the existing analysis session instead of starting its own. This is on by default and can be turned off. The new incident's information is kept, and you avoid repeated analysis and repeated queries.
Alert governance decides which problems are worth waking a person up for. Noise reduction organizes related signals into incidents someone can actually work on. Get both right, and your SRE Agent can get a head start at 3 a.m., so the engineer wakes up to a useful first analysis instead of more waiting and a pile of duplicate reports.
If you'd like to see how Flashduty handles on-call, alert noise reduction, and AI SRE, check it out and give it a try: Flashduty
