AI SRE can now look inside your Kubernetes cluster. Your cluster dials out to us, the agent runs real kubectl over that connection, and your own RBAC decides every request. No kubeconfig leaves the cluster, and you open no inbound ports.
Before this, a pod could restart in a loop and the agent couldn't see it. It could read the alert, the code, and the dashboards, but not the pod's state or its recent events. It had to guess, or you ran kubectl and pasted the output.
This post covers why we rejected the three usual approaches, how ours works, and what it still can't do.
What we wanted
- The agent gets everything kubectl can express.
- Credentials never leave the cluster.
- You open no inbound ports.
- Permissions stay tight, without a capability list we maintain by hand.
The last one is the hard one. Every cluster has its own long tail of CRDs, so a hand-written list is incomplete on day one. And the more general the interface, the harder it is to keep permissions tight.
Three approaches we rejected
Approach one: hand over a kubeconfig and connect directly.
The credential leaves the cluster. Most production API servers also aren't reachable from the internet. You'd have to expose yours and allowlist our IPs, or run the agent inside your network. Security teams usually say no to both.
It doesn't even work on managed clusters. An EKS, GKE, or AKS kubeconfig holds no credential. It holds instructions to run a local binary, like aws-iam-authenticator or gke-gcloud-auth-plugin, that trades something for a token. Copy the file to another machine and it stops working.
Its one strength is coverage: anything kubectl can do, it can do. We kept that part.
Approach two: run a Kubernetes MCP server.
Coverage is the first problem. MCP gives the agent tools someone defined in advance. If nobody added a query like kubectl get pods -o jsonpath='{.items[?(@.status.phase!="Running")].metadata.name}', or your cluster's CRD, the agent can't run it. We moved our own platform from MCP to a CLI for this reason; see From MCP to a CLI.
The security problem is less obvious. CVE-2026-46519 in mcp-server-kubernetes, disclosed this May, is an example. Its read-only switch applied when the server listed tools, but not when it ran them. Anyone who knew a tool name could call delete directly. The advisory notes that the bypass can't exceed the permissions Kubernetes already gave the server. So the cluster's RBAC was what stopped it, not the switch.
Approach three: build our own Kubernetes tools.
Even a small read-only set has approach two's coverage problem. We'd be chasing the Kubernetes API and your CRDs forever. It also costs context. Every tool schema sits in the model's context window, and the model has to learn tools it has never seen. The model already knows kubectl from training, and it writes jsonpath better than most people. A tool list turns a fluent speaker into someone reading from a phrasebook.
Vendor and community options today mostly fall into these three. So we set three rules. Keep all of kubectl. Don't connect directly to your cluster. Don't make permission decisions on our side.
How it works
The agent runs real kubectl, with no wrapper. Only the server address in its kubeconfig changes. It points at our endpoint instead of your API server. The agent never holds a real cluster credential; kubectl just has to think it does.
Your cluster connects first. You install a small program in the cluster. On startup it dials out to us and keeps the connection open. Every request travels back down that connection.
So the network question becomes one question: can your cluster reach the internet? You don't need firewall tickets, a VPN, a bastion host, or an IP allowlist.

Hop by hop:
kubectl (in the session's execution environment)
→ our public endpoint /safari/k8s/proxy/<cluster>/...
→ checks: token, cluster ownership, request shape
→ back down the connection the cluster dialed out
→ the forwarder in your cluster, replaying the request verbatim
→ your apiserver, deciding with its own RBAC
Limits per request: 60-second timeout, 1 MiB request body, 6 MiB response body. The connection sends a heartbeat every 54 seconds.
The three approaches side by side:
| Hand over a kubeconfig | Run an MCP server | Kubernetes App | |
|---|---|---|---|
| Inbound ports you open | apiserver must be reachable | depends | 0 |
| Credentials that leave the cluster | the whole kubeconfig | depends | none |
| Queries it can express | all of kubectl | whatever is in the tool list | all of kubectl |
| Long tail of CRDs | supported | wait for the tool author | passed through verbatim |
| Works on managed clusters | no | depends | yes |
| What runs in your cluster | nothing | depends | one 2.7 MB binary with no shell |
The connection is request and response, so streaming commands don't work. See the limits at the end.
A forwarder with nothing in it
The program in your cluster only forwards requests. We cut it down to one static binary.
docker run --rm --entrypoint sh \
registry.flashcat.cloud/public/flashduty-k8s-agent:v0.1.0 -c 'echo hi'
exec: "sh": executable file not found in $PATH
There's no bash, sh, ash, dash, busybox, kubectl, curl, wget, cat, ls, env, python, or perl. The image is 2.7 MB in 2 layers.
Why so strict? If our side were fully compromised, an attacker would still find no shell to run inside your cluster. The image is public, so you can run the command above and check.
Many clusters, one kubeconfig
The agent uses native kubectl, so multiple clusters are just kubectl contexts.
At the start of each turn, we rewrite the kubeconfig in the agent's environment. It lists the clusters this session can see now. Each cluster gets one context with its own server address and its own token. No credential is shared, so each cluster stays separate at the credential level. To switch clusters, the agent uses a command it already knows:
kubectl --context <cluster-alias> get pods -n payment
Two details we care about.
We set a default context only when there's one cluster. With two or more, we leave it unset on purpose. The instructions tell the model not to guess. It should run kubectl config get-contexts first and ask when unsure. That removes the failure where prod happens to be the default and a command lands there.
We rewrite the file every turn. When nothing is permitted, we write an empty file. So when you revoke access, the context is gone on the next turn. A disabled cluster never gets written at all.
Kubernetes decides permissions
We built a version with four write tools, each asking a human to confirm before it ran. Then we deleted it.
Any check that runs before the API server is only advice. Kubernetes already has a full permission system: who, which resource, which verb, which namespace. Adding our own layer on top wouldn't make anything safer. It would add a second system to maintain and audit, and that second system would be weaker.
So permissions live in one place: an RBAC YAML that the console generates and you apply. There are three levels.
- Read-only binds the built-in
viewClusterRole: get, list, and watch on the usual resources. - Read plus limited write adds only two things: scaling replicas and evicting pods.
- Full access binds the built-in
cluster-admin. It's unrestricted.
Read-only started as a resource list we wrote by hand. We replaced it with the built-in view for two reasons. Our list had silently left out configmaps. And no hand-written list keeps up with CRDs. view is maintained upstream and never grants secrets. It also has an aggregationRule: when an operator marks its CRD read roles as view-safe, they're included automatically. The generated manifest comes down to this binding:
roleRef:
kind: ClusterRole
name: view # built-in read surface, no secrets; view-safe CRDs aggregate in
# …the full three-level YAML is generated by the console; see the product docs linked below
Limited write keeps one rule: it can't change a pod's effective spec. That means no changes to image, command, env, identity, or mounts. It can only change how many copies of a declared spec run, or evict a pod so its controller rebuilds it unchanged.
Here's why we drew the line there. If spec edits were allowed, someone could change a command to print a mounted credential. Then read-only's pods/log would hand it back, and the whole permission model would mean nothing.
The cost: kubectl rollout restart and rollout undo are out too. Both are PATCHes on the workload, and no authorization layer can tell an honest rollback from a swapped template. Restart has a workaround: evict pods one at a time. Rollback has none, so the agent proposes it to a human.
You read the YAML and apply it yourself. You can check the boundary instead of trusting our description.
RBAC alone isn't enough, though. RBAC sees the verb and the resource, not what an endpoint actually does. Horizon3's security team published an example. An account with only read access to nodes/proxy looks read-only on paper. Yet it can reach arbitrary command execution through the kubelet, because the connection opens with a GET, which RBAC treats as a read. Upstream marked it won't-fix.
So we add one check on our side. It limits the shape of a request, not its permissions. We split the path on / and reject any segment equal to exec, attach, or portforward. We also reject watch=true and follow=true.

What each layer covers:
| Catches | Misses | What covers the gap | |
|---|---|---|---|
| Our request-shape narrowing | paths containing exec, attach or portforward, including the kubelet route .../proxy/exec/... | the kubelet's other execution entry points (.../proxy/run/..., for instance) are not on the list | we never grant nodes/proxy at all, so the apiserver rejects it |
| Your cluster's RBAC | neither read-only nor read plus limited write contains nodes/proxy | "all namespaces plus full access" is unrestricted and does contain it | only the red warning in the console — on this row, human judgment is the backstop |
Neither layer is complete on its own. You're protected where they overlap.
You run the install
The console could install everything for you if you gave us a cluster credential. That's approach one, which we rejected.
Instead, the console generates a command and you run it. It's one extra step, and in return you can read everything before you apply it. It runs on production clusters, so we made the script defensive:
- The whole body is wrapped in
main() { ... }. A truncated or tampered download fails the syntax check instead of running half a script. kubectl auth can-ichecks the eight permissions it needs first. If any is missing, it lists them and exits withNo cluster resources were changed.- It runs
kubectl apply --dry-run=serverso your API server validates the manifest before the actual install. - Uninstall deletes only by an exact ownership label, never "label exists." It removes only what this App installed and leaves the shared namespace alone.
- Install and uninstall links expire after 10 minutes.
The Kubernetes App sits in the same console catalog as the GitHub and GitLab Apps, but it connects differently. Those two use an authorization flow to get credentials; see GitHub and GitLab Apps for AI SRE. This one has your cluster dial out.
Limits
- No streaming.
logs -f,exec,port-forward, andwatchdon't work. To follow changes, poll. - Read-only covers the built-in
viewsurface. The usual resources are in;secretsnever are. A CRD is readable if its operator marks its read roles view-safe. If not, reading it needs full access. - Limited write can't touch a pod's spec: no image, command, or env changes, only replica counts and evictions. So
rollout restartandrollout undoaren't available. Restarts go through one-by-one evictions, and rollbacks go to a human. - "All namespaces plus full access" has no restrictions. Take the red warning in the console seriously.
- Revoking cuts the connection. What's installed in the cluster stays until you uninstall it.
For a feature summary, see the Kubernetes App changelog entry. For setup steps, see the product documentation. To try it, create a Kubernetes App under AI SRE → Apps in the console. Start with read-only on one namespace and see what it finds.


