We stopped writing docs by hand

We don't have a dedicated product manager, and our docs kept falling behind the code. Now an AI agent handles it. Every week it compares the code with the docs, finds the gaps, writes the missing content in Chinese and English, and opens a pull request (PR). Engineers only review.

On its first run, it looked at one month of code changes and found 21 gaps. A full audit found more than ten times that many. Our conclusion: instead of reminding engineers to write docs, make drift visible.

Here's how it works, and how you can build one.

An undocumented toggle

Flashduty's on-call scheduling has a feature called Fair Rotation. When you turn it on, the system adjusts the rotation order so the same person doesn't always get the weekend shift. Here's the frontend code:

<div className='flex items-center'>
  {t('Fair Rotation')}
  <Tooltip>
    The system automatically adjusts rotation order to ensure
    each member gets on-call shifts across different time periods.
  </Tooltip>
</div>
<Form.Item name='fair_rotation' valuePropName='checked'>
  <Switch size='small' />
</Form.Item>

We built the feature in the frontend and the backend. It shipped, and customers use it. The docs didn't mention it.

Users found the feature only if they noticed a small toggle while editing a rotation rule. That's documentation drift. It doesn't cause an outage, but gaps add up until users stop trusting the docs.

Why docs fall behind code

We're building an AI support agent that answers user questions by reading our docs. If the docs have a gap, the agent can't answer. If the docs are wrong, the agent gives a wrong answer.

Doc quality used to matter mostly to people, such as an engineer who gets an alert at 3:00 AM and searches the docs. Now doc quality also determines the quality of the AI agent's answers.

The problem: nobody owns the docs.

Flashduty has no dedicated product managers. The engineers are the product managers. The same people analyze requirements, write the product requirements document (PRD), build, test, deploy, and handle customer feedback. "Keep the docs in sync" isn't in anyone's job description.

The engineer who spent two weeks on Fair Rotation doesn't want to spend another day writing it up. They want to build the next feature. So the docs stay unwritten, and the answers stay in Feishu, WeCom, and DingTalk chat logs.

We have 12 product modules, more than 20 microservices, and hundreds of doc pages in Chinese and English. Any merged PR can make a page inaccurate. Manual review is possible in theory, but in practice it never happens.

Reminders to update the docs don't work, so we automated the check.

How the agent works

We built a Claude Code skill, which is a reusable AI agent. An engineer can run it with one command or put it on a schedule.

The key component is a YAML mapping file. It lists the source-code paths behind each doc page, so the agent knows which code to read. Without the mapping, the agent either gets lost in unrelated internal code or misses important changes.

flowchart TD
    T1["Code push / scheduled job"] --> SCAN["Scan source repos"]
    T2["Manual trigger"] --> SCAN

    SCAN --> DIFF{"Mode?"}
    DIFF -- "diff" --> GD["Read git diff\nFind new routes, toggles, defaults"]
    DIFF -- "audit" --> WF["Walk frontend code\nInventory every feature"]

    GD --> XREF["Cross-reference against docs"]
    WF --> XREF

    XREF --> FIND["findings.yaml"]

    FIND --> AUTO{"Auto mode?"}
    AUTO -- No --> HUMAN["Engineer reviews\nRemoves false positives"]
    HUMAN --> FIX
    AUTO -- Yes --> FIX["Read source\nGenerate docs (ZH + EN)"]

    FIX --> PR["Open PR"]
    PR --> REVIEW["Engineer reviews content"]
    REVIEW --> MERGE["Merge"]

The skill has two modes.

Diff mode asks: what changed recently? The agent reads the git diff in every linked repo and looks for signals that affect docs:

• New API routes or endpoints
• New UI toggles or form fields
• Changed defaults or validation rules
• Removed or renamed features

The agent checks each signal against the existing docs. If the docs already cover a signal, the agent skips it. Otherwise, the agent records a finding. This mode handles routine upkeep and catches drift early.

Audit mode asks: does every feature have docs? This mode goes deeper. It walks the frontend code and lists every feature a user can see: pages, form fields, toggles, and dropdown options. Then it checks each one for coverage.

Audit mode starts from the frontend because the frontend shows what the product can do. The backend has internal endpoints, reserved fields, customer-specific logic, and enum values the UI never shows. What the frontend shows is what users can see and use.

Audit mode also checks accuracy. It pulls facts you can check mechanically from the docs, like field names, default values, navigation paths, and validation rules. Then it compares each one with the code. For example, if the docs list rotation periods as "day, week, custom," but the code supports hour, day, week, and month, the agent records a finding.

Both modes write a structured findings file. Then comes the fix phase. For each finding, the agent reads the source code and writes complete docs in Chinese and English, with paragraphs, tables, and configuration steps. The PR contains complete content, with no placeholders or TODOs. An engineer reviews the content and merges it.

What the agent found

After running both modes, the agent had opened 20 PRs with more than 9,000 lines of new docs, across more than 370 files and hundreds of findings.

The first run used diff mode: /doc-review --mode diff --since "1 month". It looked only at the last month of changes and produced flashduty-docs PR #27. That one run found 21 gaps, touched 33 files, and added over a thousand lines of docs across both languages. Some examples:

These are all commonly used features, and users couldn't find them in the docs.

We didn't expect 21 gaps. Then we ran audit mode across all the frontend code, and it found more than 10 times as many. So we turned a one-time cleanup into a weekly job.

Build your own agent

You can start from a template. Claude Code has a skill-creator that generates a skill from a description. Here's a prompt template that works:

/skill-creator Create a doc-review skill that cross-references
source code against documentation to find documentation drift.

Context:
- Docs repo: company-docs (Docusaurus, English only)
- Source repos: billing-service (Go), web-app (React), api-gateway (Go)
- Docs structure: docs/ with subdirectories per product area
- Key concern: new features ship without doc updates

The skill should have two modes plus a fix phase:
1. Diff mode: scan recent git changes for doc-relevant signals
2. Audit mode: walk frontend code to inventory features, check coverage

After either mode, a fix phase generates complete documentation and opens a PR.

skill-creator generates the mapping config, the prompts, and the wiring. Building the skeleton takes about an hour. Tuning takes longer. It took us about four rounds to reduce false positives to an acceptable level.

What makes it work is the design rules you put in the prompt. Here are the seven rules we learned in practice.

1. Map modules to repos. Use one YAML file to link each doc page to its source-code paths. Without this file, the agent gets lost in unrelated code or misses important changes. This file is the skeleton of the whole system.

2. Start the inventory from the frontend. The frontend defines which features users can see. The backend has internal endpoints, reserved fields, and reserved types. Start from frontend components: pages, forms, toggles, and dropdowns. Use backend code only to fill in details for features the frontend already shows. Our first two iterations started from the backend, and the results filled up with internal tools that don't belong in public docs. Your product might be API-first, like a command-line interface (CLI) or a software development kit (SDK) with no UI. In that case, start from API route definitions or CLI command registrations.

3. Judge like a product manager. Say it in the prompt: "Decide, like a product manager, what belongs in the public docs." Record only features a user can use in production. Skip test tooling, debug panels, test harnesses, internal admin functions, and customer-specific logic.

4. Check only facts you can verify mechanically. Check field names, default values, API routes, and validation rules. Leave prose that describes behavior to human reviewers. It needs judgment and produces many false positives.

5. Let the frontend decide constraints. If the backend allows 100 but the form allows 50, users see 50, so the docs should say 50. Record a finding only when the docs contradict the frontend.

6. Generate complete content. In the fix phase, the agent must read the source and write real docs: paragraphs, tables, and configuration details. The reviewer should only polish the writing. This rule decides whether you get a usable PR or a list of TODOs.

7. Use two phases with a human check in between. The flow is: analyze, write the findings file, have a person remove false positives, then fix. The findings file is the checkpoint between phases. Once false positives are low enough, --auto skips the findings review. A person still reviews the final PR before it merges.

Expect the first run to be messy. Start with --dry-run, read the findings, and adjust the prompt. Use skill-creator to iterate too: tell it what went wrong and let it refine the prompts and mapping.

Once it's tuned, automate it. A weekly cron job runs diff mode for us. When it finds gaps, it opens a PR automatically. The system detects, writes, and submits. Engineers review.

What we learned

Asking engineers to write better doesn't fix drift. Showing the gaps and giving engineers a complete PR does. Reviewing that PR takes five minutes. Writing a doc from scratch takes at least half a day, so it never makes the priority list.

We thought our docs were in good shape. The results showed a large gap. Your docs probably have gaps of a similar size that nobody has checked yet.

AI is good at the tedious part: reading hundreds of files, cross-checking field names, and comparing defaults with code. People still need to decide whether a feature needs documentation and whether a sentence is clear. So the PR still needs an engineer's review. But the AI agent now handles the repetitive work.

What's next

The Fair Rotation toggle has docs now, and we've resolved the hundreds of other findings. This is the first step.

We're working on two things. First, we're adding diff mode to continuous integration (CI). It will check for drift when code merges, the way tests run, so we catch a gap right away instead of in the weekly scan. Second, we're extending checks to the API docs. Right now the system starts from the frontend, but our Open API needs the same coverage.

Read our docs on the Flashduty documentation site. Engineers now review doc PRs instead of writing docs from scratch. What worked for us was simple: one hour to build the skeleton, four rounds to tune it, then five minutes a week to review a PR.