Skip to main content
Blog

Working at 3 a.m.: Five ways to use AI SRE Automations

Five ways to use Flashduty AI SRE Automations: weekly insights, alert governance, incident analysis, deploy checks, and change risk checks. Each comes with the exact task prompt and shows how results reach people.

Flashduty Product

Working at 3 a.m.: Five ways to use AI SRE Automations

An automation lets AI SRE start work on its own, including at 03:00. If you try one thing, try this: turn on incident analysis for one channel, Critical only. When a page fires at night, the agent posts its first findings on the incident before your on-call opens it.

Below are five ways we use Automations, each with the prompt we use, and how the results reach people.

How an automation works

An automation is a rule: a task prompt plus at least one trigger. There are three triggers.

  • Run on a schedule: hourly, daily, weekly, or your own cron expression.
  • Call via API: you get a URL and a token, and POST to it from anywhere.
  • On-call incident: pick the channels and severities to watch. A matching incident starts a run.

One rule can have all three. The prompt stays the same; only the trigger differs.

Each run opens a background session. It doesn't appear in your session list, but the agent has the same tools as anywhere else. Afterwards you can open it and see every command, tool call, and output.

The first three use cases ship as templates. The last two are examples you paste and edit. Here's where each result ends up:

ScenarioWhere the result goes
1 · Weekly insightsStays in the run's session; ask it to publish and it becomes a permanent artifact
2 · Alert governanceSame
3 · Incident analysisWritten back onto the incident as a comment, riding the incident's notifications to whoever is on call
4 · Post-deploy checkThe call returns a session_url; post it to the deploy thread or pipeline
5 · Change risk readThe call returns a session_url; paste it onto the change ticket

1. A weekly report on what happened

Every Monday, someone spends a couple of hours writing up last week: incidents, alerts, who was on call, what changed. It's the same query every week.

Use a weekly schedule. Pick a Monday morning time so the report is ready when people arrive. The template is called Weekly Insights:

Generate a weekly insights report. Analyze incidents, alerts, response activity,
notification load, and related changes from the past week. Focus on what happened
this week, which signals deserve attention, and which improvement actions are most
valuable. Do not modify any Flashduty business state.

Keep that last sentence. The agent can change things in Flashduty, so a reporting rule should tell it not to.

The report stays in the run's session. Open the rule, open the run, and read it. To keep it, ask in the prompt for the report to be published as an artifact.

2. A standing audit of your noisiest alerts

Every team knows some alerts are useless. Few teams have a ranked list of which ones, and how many notifications each fix would save.

Run it on a schedule. Monthly is enough; these numbers don't move daily. The template is called Alert Governance:

/alert-governance
Produce the alert-governance report for the past 30 days for this scope. Focus on
the alerts most worth fixing and the expected notification reduction. Report only —
do not modify any Flashduty configuration.

The first line calls a skill. Automations can use skills the same way you do in chat. The skill holds the method, so the prompt only says what to point it at.

The result goes where the weekly report goes: the session, or an artifact if you ask.

It reports and doesn't fix. Turning the report into config changes stays a human call, on purpose.

3. A first look at an incident, before anyone opens it

A Critical fires at night. The on-call sees it a while later and spends the next fifteen minutes looking things up: alerts, timeline, recent deploys. That first round doesn't need a human.

Use the On-call incident trigger. Choose the channels and tick the severities. A new matching incident starts the run. The template is called On-call Incident Analysis:

Analyze the incident that triggered this automation. Use the Flashduty CLI to fetch
incident details, related alerts, and the timeline. Correlate recent changes, alert
context, and existing response notes to judge the likely cause. If evidence is
insufficient, state the uncertainty and recommended next checks.
If you form a useful investigation conclusion, add one summary comment to the
incident. Prefer one complete investigation summary instead of frequent incremental
comments.

The conclusion goes back onto the incident as a comment. A new comment follows the incident's assignment policy, so the on-call gets notified the usual way. They see the findings where they were already looking, and don't need to know an automation ran.

We added "state the uncertainty" on purpose. A confident guess is worse than "the evidence isn't there yet, check these two things next."

This trigger also gives the agent a head start. A scheduled run starts with your prompt. An API run starts with whatever the caller sent. An incident run starts with the incident itself: its ID, channel, and severity are in context before the first step. The agent is also told not to list incidents to find its target. That skips several rounds of tool calls, and each round is a chance to pick the wrong incident. A 09:00 schedule can only tell you about last night's incident at 09:00.

Some details. Only new incidents start a run; acknowledging, resolving, or reopening doesn't. You can filter by channel and severity only, not labels yet. Each incident starts one run and only one, so a duplicated internal message never becomes three parallel investigations.

4. A check after every deploy

After a deploy, you watch dashboards for ten minutes. Most of the time nothing changed.

Use the API trigger. Add it, save the rule, and you get a URL and a token. The console shows the token only once, so copy it before you leave the page. After that you can only reset it, which issues a new token and disables the old one immediately.

Call it from the last step of your pipeline:

curl -X POST 'https://<trigger URL>' \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{"text":"Deployed payment-service v2.14.0 to prod. Changes: retry policy on the settlement callback, new index on t_order.created_at."}'

AI SRE appends the text field to the prompt for this run only. It's the only way to pass outside information into a prompt. A scheduled run has no caller, and an incident run gets its context from the incident.

A prompt to start from:

A deploy just finished; the details are at the end of this message. Compare alert
volume, incident count, and error patterns for the 30 minutes after the deploy
against the same window before it, for the services named. Report only what changed
and whether it plausibly relates to this deploy. If nothing changed, say so in one
line. Do not modify any Flashduty state.

The call returns right away. It doesn't wait for the agent:

{
  "data": {
    "type": "routine_fire",
    "session_id": "<session-id>",
    "session_url": "https://<console>/ai-sre/chat?session_id=<session-id>"
  }
}

Post the session_url to the deploy thread or attach it to the pipeline run. Anyone who wants detail can click through.

Each rule accepts one API call per minute. Two API triggers on the same rule share that budget. It's built for events, not polling.

5. A risk check before a change ships

Change approval is often the only time anyone asks "has this burned us before?" Nobody has time to look.

Use the API trigger. Have your change-management system call it when a change is approved, with the change itself in text.

A prompt to start from:

A change request is described at the end of this message. Look for incidents in the
last 90 days involving the same services or the same kind of change. For each one,
summarize what broke and what fixed it. Then list the specific things worth checking
before and after this change lands. If you find no relevant history, say so plainly
rather than generalizing. Do not modify any Flashduty state.

Like the deploy check, the call returns a session_url. Paste it on the change ticket. The approver reads the history without leaving their tool.

Getting results to people

Automations have no notification channel of their own. Results reach people in two ways.

Incident runs use the incident's notifications. The conclusion is an incident comment, and the incident's assignment policy delivers it to whoever is paged. Nothing to set up.

For other runs, let the agent send it. Scheduled and API runs notify nobody by default; the result stays in the session. To post to a group chat, put an IM bot webhook URL in the prompt. Feishu, DingTalk, and WeCom custom bots are each just a URL. The agent POSTs a summary when it's done:

When the task is done, compress the conclusion into a summary of at most 5 lines and
POST it as JSON with curl to this bot URL:
https://open.feishu.cn/open-apis/bot/v2/hook/<your-token>.
If sending fails, say so in the result; do not retry more than twice.

We do this ourselves. Here's the message the agent posted to our DingTalk group after its daily documentation audit:

The message the agent sent to a DingTalk group after the daily documentation audit run

It says what it audited, how many findings, how many it applied, and where the PR is, with links for detail. One caution: anyone who can see the rule can see the webhook URL, and anyone with the URL can post to that group. Limit who can edit such rules, and turn on the bot's own security settings where available.

Behaviors we chose for you

Your 09:00 job may start at 09:03. That's on purpose. If every scheduled job fired at 09:00:00 sharp, none of them would start quickly. Treat it as "every morning," not a stopwatch.

A run that's still going blocks the next one. If the previous run hasn't finished when the next is due, the next one is skipped, not queued. Otherwise a slow job piles up behind itself.

A missed run expires. If a run is more than a day late, it's dropped. Nobody wants forty queued reports after an outage.

Restarts don't lose or duplicate runs. Runs in flight survive a service restart and are recorded once.

Another team's incident still runs under your rule. If the incident belongs to a different team than the rule, the run uses your rule's permissions. The agent is told up front that the incident belongs to another team and that team's knowledge wasn't loaded. It won't read across that boundary, and it won't skip the run silently.

Getting started

Open AI SRE → Automations in the console. The templates are on the create screen. From a real user error to a fix PR in 8 minutes walks through an incident-triggered run end to end.

How the agent drives Flashduty is in From MCP to a CLI. The full field reference, including cron rules and trigger payloads, is in the docs. AI SRE is in open beta, and free during the beta.

The built-in prompts reflect our habits: Monday reports, monthly alert cleanup, a comment before a page. Rewrite them for your own rotation.

Related articles