Our AI SRE operates Flashduty on behalf of users: querying alerts, acknowledging incidents, checking on-call schedules, tracing recent changes. It first got that ability in January, through our own MCP server, wired into every session as a built-in with 18 tools. In late May we replaced that path with a command-line tool, and a week later we deleted the MCP path. Since then the CLI has been the agent's only way into the platform.
18 tools weren't enough
At first this worked fine. The 18 tools were picked around the incident lifecycle: query, acknowledge, close, escalation rules, change records, status-page notices. Everyday requests landed cleanly, and with so few tools the agent rarely picked the wrong one.
Investigations were a different story. An investigation starts without anyone knowing what the next step is. It might be alerts, the timeline, the on-call record, or some change ticket. The platform's public API has 290 operations and any of them can turn out to matter; 18 tools don't cover that.
Growing the MCP was the first idea
The obvious move was to make the MCP bigger. There were two ways to do it.
Full mirroring: wrap all 290 operations as tools. We dropped this one quickly. Hundreds of tool schemas sit in the model's context permanently, and the more tools there are, the worse the model gets at choosing between them. We later wrote this into the MCP server's README: mirroring an entire API 1:1 both bloats context and degrades tool selection.
Progressive loading: don't lay the tools out at all, give the agent a search tool and an invoke tool, and let it fetch what it needs. AWS's MCP works this way. We knew the mechanism well, because our loading path for third-party connectors worked along similar lines at the time: the agent names a server and gets that server's tool catalog back.
Two things worried us. First, accuracy: with schemas out of sight, the agent picks a tool and fills its parameters off a single retrieval, and if the search misses, or hits but gets misread, nothing catches it. Second, context only grows: every search result and every loaded schema stays in the conversation. An investigation makes dozens of calls, the session keeps getting longer, tool descriptions keep piling up, and the model's performance sags as its context fills.
We had no experimental data for either concern. It was intuition. But both point the same way: search-based MCP spreads discovery cost across every call, and the longer the session, the more you pay.
Switching to a CLI
The agent's execution environment already has bash. A CLI inverts both concerns: help text is read on demand and is just text afterward, not a tool definition living in context; output can be piped and filtered instead of read back in full; and coverage doesn't have to be built by hand, because commands can be generated from the API description. Today 289 of the 290 operations are generated, the one exception is a hand-written streaming response, and a test asserts the ratio.
The switch landed in late May. The CLI became the agent's primary path, and a week later we deleted the MCP fallback config too. There was no rigorous head-to-head behind this decision, just intuition plus the analysis above. The bet was about cost shape: MCP's cost is spread over every call and you pay for as long as you use it; a CLI's cost is a one-time engineering investment. That investment turned out to be substantial.
Helping the agent find the right command
The first problem showed up right after the switch. The first version of the skill doc listed eight common commands and told the agent to run --help for the rest. This seemed safe, since help output always matches the installed version and can't go stale. In practice the agent had to guess a command's name before it could look up its help. The doc taught statuspage create-incident; the real command is status-page change-create. Guess wrong, look it up, guess again. The second version added a derivation rule, group from the first segment of the API path and verb from the rest joined by hyphens, confirmed via help, with a table for the high-frequency mismatches. Fewer errors, not zero.
The current design has two layers. The main file is a bilingual intent index that routes a user's request to a domain. Each domain has one card with all of its commands, flags, enums, and workflows, 346 commands in total now. The cards aren't written by hand; they're generated by reflecting over the compiled command tree, and CI checks every example on every card against that same tree. One wrong example turns the build red.
Two things watch accuracy. One is a synthetic question set: a stronger model generates questions in bulk, the AI SRE answers, a model grades, over a thousand questions in all. It covers the high-frequency ground, but the question-writer, answerer, and grader are all models, and generated questions aren't distributed the way real ones are. So there's a second: periodic audits of live sessions, fifty-some per round, looking for the failures the question set doesn't think of. One audit caught the agent reading "this rule hasn't fired in 90 days" as "this rule isn't configured" and flagging a P0 check as missing. That kind of error has nothing to do with command selection; the data was right and the interpretation was wrong. Recurring audit findings get written back into the skill as rules, like "an empty result means no, don't retry with different keywords" and "a command you didn't run can't be reported as returning empty." We're still paying into this part.
JSON escaping
One more problem that looks small and wasn't: passing JSON on the command line. JSON has quotes, bash has quotes, and with one nested in the other, models get it wrong easily. Our execution environment also parses every command as full shell syntax for a permission check before running it, and any quote imbalance is rejected outright, so this surfaced more often for us than it normally would.
The main fix is keeping the agent away from raw JSON. When commands are generated, every flat field in the request body becomes its own flag, like --chat-id and --member-ids, so common calls contain no JSON at all and the escaping problem never starts. For the cases that genuinely need a JSON blob, nested structures, or parameters carrying SQL (quotes, commas, and newlines all at once), the answer is stdin: --data - reads the body from standard input, paired with a quoted-delimiter heredoc, which shell expands nothing inside, so the content passes through verbatim with zero escaping.
fduty monit-agent invoke --data - <<'FDUTY'
{"tools":[{"tool":"query","params":{"sql":"SELECT a, b FROM t WHERE s = 'x'"}}]}
FDUTYThis rule came from failures too. An early name=xxx,params={...} mini-format split parameters on commas, and a single SELECT a, b in SQL shattered it; the whole thing was deleted in favor of stdin. The skill doc used to say "prefer heredoc when the parameters contain quotes"; then in one production session, perfectly innocent-looking inline JSON got rejected by the command parser twice, and the agent recovered by switching to heredoc on its own. The guidance now reads: always heredoc, never inline.
Running the right binary
The agent executes a bare command name. If the host has another version installed, or an unrelated program with the same name, the command still runs, and the results are intermittently wrong in ways that are painful to debug.
The core fix is PATH. On startup the execution environment writes its bundled CLI into its own directory and prepends that directory to the search path of every bash session, so a bare command name always resolves to our bundled copy no matter what the host has installed. Naming adds one incidental layer of insurance on top: the bundled binary uses the short name fduty while the official install script installs a long name by default, so the two don't collide to begin with.
There is one place PATH doesn't work: a code path that starts processes directly without a shell resolves the command name before environment variables take effect, so no amount of path priority arrives in time, and that path rewrites the command to an absolute path instead.
We also hit a dumber failure: "which directory to install into" and "which directory to put first on PATH" were computed in two separate places. The tool landed in A, PATH pointed at B, the environment started normally, and every agent call failed with command-not-found. After the fix, both values come from one source.
Credentials
With MCP, the connection and its auth came from config. With a CLI, identity, permissions, and credential delivery all had to be designed. The easiest case to miss is the multi-user session: A opens a war room, B and C speak in it too, and the question becomes whose identity the agent uses when it runs a command.
We went with "whoever sent this message." Each turn, a credential is minted for the actual sender and deleted the moment the turn ends, with validation bypassing cache so deletion takes effect immediately. The gateway resolves the sender's identity with the same function it uses when that person logs in from a browser, so the agent gets exactly their permissions, and if the sender can't be identified, nothing is issued. A per-session design was rejected during review, because in a war room it would hand A's identity to B's commands.
The short lifetime only handles replay; while the credential is live, it can still get printed. Agents are curious, and when one is debugging an environment problem, running env or printenv is a natural move that would drop the credential straight into the session transcript. So every bash command passes a check before it runs: anything that bulk-dumps the environment is blocked, including env, printenv, bare set, export -p, declare -p, compgen -e, and reads of /proc/*/environ, along with any command that literally references the credential's variable name. The rejection message tells the agent to read the specific variable it needs instead. This check doesn't stop deliberate string concatenation; it stops casual printing.
The next layer down is keeping the credential out of the environment entirely. On self-hosted Linux runners we built a socket bridge: before the subprocess starts, a Unix socket pair is created and the credential travels to the CLI over that channel, appearing in neither command-line arguments nor environment variables, invisible to env. Cloud sandboxes can't do this and fall back to a turn-scoped environment variable. So "the credential never enters the environment" isn't true of our system; it's true only where the bridge is supported.
What we don't defend is worth listing too. Other processes under the same sandbox user using the credential within its window: not defended. If the CLI can authenticate with it, a neighboring process can in principle get it, and encryption doesn't change that. What we do defend against is exfiltration and later replay, and the defense is that the credential doesn't outlive a turn. We also haven't narrowed permissions below the sender's own; if the speaker is the account owner, the agent has owner permissions for that turn.
Versions
The CLI has no release of its own. It's embedded in the execution environment's binary and ships with it: the environment writes it to disk at startup, the version is pinned in a constant, and upgrading the CLI means releasing the environment, so the two always upgrade together. No "new environment with an old CLI" or "old environment with a new CLI" combination can exist in production. Incompatibility is gone structurally, which is why there is no runtime version check.
We did build the more flexible design: a control-plane manifest with hot updates at runtime. The code was written and merged, and about a dozen hours later we deleted all of it. It saved one environment release and added a version state that would need maintaining forever. Not worth it.
Keeping the CLI aligned with the API
Hundreds of commands maintained by hand will drift. Our approach is one source generating downward: the code is the single truth, the API's route registrations and implementations are the source, and every hop below is produced by tooling, with nothing copied by hand.
The chain isn't fully automatic. The scheduled job from API description to SDK opens a merge request but doesn't merge it; SDK to CLI takes a manual dependency bump and a re-run of the generator. An API change reaching the command level passes through three human sign-offs. Only one gate in the chain is automatic and blocking: the check that cards match the real command tree. The allocation follows from the two kinds of drift doing very different damage. A new operation the CLI hasn't caught up with is a missing feature, and the agent finds nothing and says so. A card teaching a command that doesn't exist is wrong information, and the agent retries around it or invents something plausible. The automated gate blocks the second kind.
Composition lives in the top skill layer, not in the CLI. A full incident summary needs six aspects, which map to six separate commands. A script in the skill pack runs all six and puts the results in one block, so every section of the summary is backed by real output and the agent has no opening to skip a command and paper over the gap. The CLI itself stays close to 1:1, with only small ergonomic patches, like resolving the member IDs a list API returns into names.
This layer got reworked too. The script initially forced all six commands into a token-saving compact format, and a production audit measured about 80,000 characters per incident: the compact format flattens out every empty field, and the agent needed three or four reads to page through it. We reverted to each command's default rendering and added a field projection to the detail command only. Output meant for a model should be readable in one pass; every extra page or filter is another chance to be wrong.
MCP is still around
Two weeks after the switch, we moved third-party connector loading to search-based: the agent boots with no tool descriptions loaded, just a search tool and an invoke tool. The same shape we didn't give our own platform.
There's no contradiction. Third-party systems' APIs and release pipelines aren't ours. Generating a CLI, pinning versions, and bundling binaries are all unavailable, and short of a shell, search-based loading is the best option. The catalog now holds 35 third-party connectors, about half of them monitoring and observability systems. Anthropic later published numbers pointing the same way: in multi-server setups, tool descriptions consumed about 55k tokens, search-based discovery cut that by more than 85%, and tool-selection accuracy rose from 49% to 74%. Note that this compares search against full loading, not search against a CLI; for that second comparison we haven't seen public data.
The public MCP server is still there, now at 23 tools, serving clients like Cursor and Claude Desktop that have no shell. The first screen of its README says: if your agent has a shell, use the CLI directly.
Looking back
The variable that actually decided this was never MCP versus CLI. It's whether the platform is yours. When it is, the API description, the release pipeline, and the execution environment are all in hand, and you can convert costs that recur on every call, like discovery, credentials, and versions, into a one-time engineering investment. When it isn't, none of those levers exist, MCP is the right choice, and we use it that way ourselves.
The decision ran on intuition. The list of costs came out longer than expected, and the tool-selection part is still accruing. This works for our situation; yours is different, and your conclusion may well be too.
