Skip to main content
Blog

Safe cluster access for AI SRE: No kubeconfig, no open ports

To bring a cluster into AI SRE we wanted four things at once: full kubectl, no inbound ports, no credentials leaving the cluster, and tight permissions without a hand-written tool list. What we shipped runs real kubectl, has your cluster dial out to us, and leaves every permission decision to Kubernetes.

Flashduty Engineering

Safe cluster access for AI SRE: No kubeconfig, no open ports

AI SRE can now look inside your Kubernetes cluster. Your cluster dials out to us, the agent runs real kubectl over that connection, and your own RBAC decides every request. No kubeconfig leaves the cluster, and you open no inbound ports.

Before this, a pod could restart in a loop and the agent couldn't see it. It could read the alert, the code, and the dashboards, but not the pod's state or its recent events. It had to guess, or you ran kubectl and pasted the output.

This post covers why we rejected the three usual approaches, how ours works, and what it still can't do.

What we wanted

  • The agent gets everything kubectl can express.
  • Credentials never leave the cluster.
  • You open no inbound ports.
  • Permissions stay tight, without a capability list we maintain by hand.

The last one is the hard one. Every cluster has its own long tail of CRDs, so a hand-written list is incomplete on day one. And the more general the interface, the harder it is to keep permissions tight.

Three approaches we rejected

Approach one: hand over a kubeconfig and connect directly.

The credential leaves the cluster. Most production API servers also aren't reachable from the internet. You'd have to expose yours and allowlist our IPs, or run the agent inside your network. Security teams usually say no to both.

It doesn't even work on managed clusters. An EKS, GKE, or AKS kubeconfig holds no credential. It holds instructions to run a local binary, like aws-iam-authenticator or gke-gcloud-auth-plugin, that trades something for a token. Copy the file to another machine and it stops working.

Its one strength is coverage: anything kubectl can do, it can do. We kept that part.

Approach two: run a Kubernetes MCP server.

Coverage is the first problem. MCP gives the agent tools someone defined in advance. If nobody added a query like kubectl get pods -o jsonpath='{.items[?(@.status.phase!="Running")].metadata.name}', or your cluster's CRD, the agent can't run it. We moved our own platform from MCP to a CLI for this reason; see From MCP to a CLI.

The security problem is less obvious. CVE-2026-46519 in mcp-server-kubernetes, disclosed this May, is an example. Its read-only switch applied when the server listed tools, but not when it ran them. Anyone who knew a tool name could call delete directly. The advisory notes that the bypass can't exceed the permissions Kubernetes already gave the server. So the cluster's RBAC was what stopped it, not the switch.

Approach three: build our own Kubernetes tools.

Even a small read-only set has approach two's coverage problem. We'd be chasing the Kubernetes API and your CRDs forever. It also costs context. Every tool schema sits in the model's context window, and the model has to learn tools it has never seen. The model already knows kubectl from training, and it writes jsonpath better than most people. A tool list turns a fluent speaker into someone reading from a phrasebook.

Vendor and community options today mostly fall into these three. So we set three rules. Keep all of kubectl. Don't connect directly to your cluster. Don't make permission decisions on our side.

How it works

The agent runs real kubectl, with no wrapper. Only the server address in its kubeconfig changes. It points at our endpoint instead of your API server. The agent never holds a real cluster credential; kubectl just has to think it does.

Your cluster connects first. You install a small program in the cluster. On startup it dials out to us and keeps the connection open. Every request travels back down that connection.

So the network question becomes one question: can your cluster reach the internet? You don't need firewall tickets, a VPN, a bastion host, or an IP allowlist.

A kubectl request, hop by hop: the request flows left to right, but the connection was dialed out by the cluster on the right, which is why you never open an inbound port

Hop by hop:

kubectl (in the session's execution environment)
  → our public endpoint /safari/k8s/proxy/<cluster>/...
  → checks: token, cluster ownership, request shape
  → back down the connection the cluster dialed out
  → the forwarder in your cluster, replaying the request verbatim
  → your apiserver, deciding with its own RBAC

Limits per request: 60-second timeout, 1 MiB request body, 6 MiB response body. The connection sends a heartbeat every 54 seconds.

The three approaches side by side:

Hand over a kubeconfigRun an MCP serverKubernetes App
Inbound ports you openapiserver must be reachabledepends0
Credentials that leave the clusterthe whole kubeconfigdependsnone
Queries it can expressall of kubectlwhatever is in the tool listall of kubectl
Long tail of CRDssupportedwait for the tool authorpassed through verbatim
Works on managed clustersnodependsyes
What runs in your clusternothingdependsone 2.7 MB binary with no shell

The connection is request and response, so streaming commands don't work. See the limits at the end.

A forwarder with nothing in it

The program in your cluster only forwards requests. We cut it down to one static binary.

docker run --rm --entrypoint sh \
  registry.flashcat.cloud/public/flashduty-k8s-agent:v0.1.0 -c 'echo hi'

exec: "sh": executable file not found in $PATH

There's no bash, sh, ash, dash, busybox, kubectl, curl, wget, cat, ls, env, python, or perl. The image is 2.7 MB in 2 layers.

Why so strict? If our side were fully compromised, an attacker would still find no shell to run inside your cluster. The image is public, so you can run the command above and check.

Many clusters, one kubeconfig

The agent uses native kubectl, so multiple clusters are just kubectl contexts.

At the start of each turn, we rewrite the kubeconfig in the agent's environment. It lists the clusters this session can see now. Each cluster gets one context with its own server address and its own token. No credential is shared, so each cluster stays separate at the credential level. To switch clusters, the agent uses a command it already knows:

kubectl --context <cluster-alias> get pods -n payment

Two details we care about.

We set a default context only when there's one cluster. With two or more, we leave it unset on purpose. The instructions tell the model not to guess. It should run kubectl config get-contexts first and ask when unsure. That removes the failure where prod happens to be the default and a command lands there.

We rewrite the file every turn. When nothing is permitted, we write an empty file. So when you revoke access, the context is gone on the next turn. A disabled cluster never gets written at all.

Kubernetes decides permissions

We built a version with four write tools, each asking a human to confirm before it ran. Then we deleted it.

Any check that runs before the API server is only advice. Kubernetes already has a full permission system: who, which resource, which verb, which namespace. Adding our own layer on top wouldn't make anything safer. It would add a second system to maintain and audit, and that second system would be weaker.

So permissions live in one place: an RBAC YAML that the console generates and you apply. There are three levels.

  • Read-only binds the built-in view ClusterRole: get, list, and watch on the usual resources.
  • Read plus limited write adds only two things: scaling replicas and evicting pods.
  • Full access binds the built-in cluster-admin. It's unrestricted.

Read-only started as a resource list we wrote by hand. We replaced it with the built-in view for two reasons. Our list had silently left out configmaps. And no hand-written list keeps up with CRDs. view is maintained upstream and never grants secrets. It also has an aggregationRule: when an operator marks its CRD read roles as view-safe, they're included automatically. The generated manifest comes down to this binding:

roleRef:
  kind: ClusterRole
  name: view  # built-in read surface, no secrets; view-safe CRDs aggregate in
  # …the full three-level YAML is generated by the console; see the product docs linked below

Limited write keeps one rule: it can't change a pod's effective spec. That means no changes to image, command, env, identity, or mounts. It can only change how many copies of a declared spec run, or evict a pod so its controller rebuilds it unchanged.

Here's why we drew the line there. If spec edits were allowed, someone could change a command to print a mounted credential. Then read-only's pods/log would hand it back, and the whole permission model would mean nothing.

The cost: kubectl rollout restart and rollout undo are out too. Both are PATCHes on the workload, and no authorization layer can tell an honest rollback from a swapped template. Restart has a workaround: evict pods one at a time. Rollback has none, so the agent proposes it to a human.

You read the YAML and apply it yourself. You can check the boundary instead of trusting our description.

RBAC alone isn't enough, though. RBAC sees the verb and the resource, not what an endpoint actually does. Horizon3's security team published an example. An account with only read access to nodes/proxy looks read-only on paper. Yet it can reach arbitrary command execution through the kubelet, because the connection opens with a GET, which RBAC treats as a read. Upstream marked it won't-fix.

So we add one check on our side. It limits the shape of a request, not its permissions. We split the path on / and reject any segment equal to exec, attach, or portforward. We also reject watch=true and follow=true.

The holes in the two layers sit in different places: we narrow the request shape, your cluster decides permissions with RBAC, and neither layer is complete on its own

What each layer covers:

CatchesMissesWhat covers the gap
Our request-shape narrowingpaths containing exec, attach or portforward, including the kubelet route .../proxy/exec/...the kubelet's other execution entry points (.../proxy/run/..., for instance) are not on the listwe never grant nodes/proxy at all, so the apiserver rejects it
Your cluster's RBACneither read-only nor read plus limited write contains nodes/proxy"all namespaces plus full access" is unrestricted and does contain itonly the red warning in the console — on this row, human judgment is the backstop

Neither layer is complete on its own. You're protected where they overlap.

You run the install

The console could install everything for you if you gave us a cluster credential. That's approach one, which we rejected.

Instead, the console generates a command and you run it. It's one extra step, and in return you can read everything before you apply it. It runs on production clusters, so we made the script defensive:

  • The whole body is wrapped in main() { ... }. A truncated or tampered download fails the syntax check instead of running half a script.
  • kubectl auth can-i checks the eight permissions it needs first. If any is missing, it lists them and exits with No cluster resources were changed.
  • It runs kubectl apply --dry-run=server so your API server validates the manifest before the actual install.
  • Uninstall deletes only by an exact ownership label, never "label exists." It removes only what this App installed and leaves the shared namespace alone.
  • Install and uninstall links expire after 10 minutes.

The Kubernetes App sits in the same console catalog as the GitHub and GitLab Apps, but it connects differently. Those two use an authorization flow to get credentials; see GitHub and GitLab Apps for AI SRE. This one has your cluster dial out.

Limits

  • No streaming. logs -f, exec, port-forward, and watch don't work. To follow changes, poll.
  • Read-only covers the built-in view surface. The usual resources are in; secrets never are. A CRD is readable if its operator marks its read roles view-safe. If not, reading it needs full access.
  • Limited write can't touch a pod's spec: no image, command, or env changes, only replica counts and evictions. So rollout restart and rollout undo aren't available. Restarts go through one-by-one evictions, and rollbacks go to a human.
  • "All namespaces plus full access" has no restrictions. Take the red warning in the console seriously.
  • Revoking cuts the connection. What's installed in the cluster stays until you uninstall it.

For a feature summary, see the Kubernetes App changelog entry. For setup steps, see the product documentation. To try it, create a Kubernetes App under AI SRE → Apps in the console. Start with read-only on one namespace and see what it finds.

Related articles