If you operate Kubernetes clusters, you know the routine. An alert fires, you open Grafana, then a terminal, then k9s, then the runbook wiki, then Slack to ask who owns the failing service. By the time you’ve stitched the context together, ten minutes are gone and you haven’t actually fixed anything yet.

A ChatOps AI agent can compress that loop. Instead of switching between dashboards, terminals, logs, and documentation, you ask a question in a chat interface: “Why is checkout-api crashlooping in staging?” The agent pulls pod status, recent events, and logs, summarizes what it found, and suggests a next step — and if that next step is a rollout restart, it asks for your approval before touching anything.

This post is a design guide for building that agent. Not the code — the architecture, the safety model, and the operational decisions that make it viable in a real environment. Because the hard part isn’t wiring an LLM to kubectl. The hard part is doing it without handing a language model the keys to production.

What the Agent Should Do

The core value is conversational access to cluster state and a small set of controlled operations:

  • Inspect resources — pods, deployments, services, nodes, namespaces.
  • Retrieve logs from specific pods and containers.
  • Summarize events instead of dumping raw kubectl get events output.
  • Detect unhealthy workloads — crashloops, image pull failures, pending pods, failed jobs.
  • Explain component status in plain language, including rollout progress.
  • Guide troubleshooting by correlating symptoms with likely causes and runbook knowledge.
  • Execute approved operations — rollout restarts, scaling, safe pod deletion — with confirmation.
  • Show its work — every response includes the exact command it ran and the raw result, not just the AI’s interpretation.

That last point matters more than it looks. An agent that shows command traces builds trust and stays debuggable. An agent that only gives you a paraphrased summary is a liability during an incident.

High-Level Architecture

The system decomposes into a handful of components, each with a single responsibility:

  • Web chat UI — the conversational frontend, behind your identity provider.
  • Backend API — session handling, authentication, request routing.
  • AI orchestration layer — the LLM plus the logic that maps user intent to structured actions.
  • Policy and approval layer — validates every proposed action against an allowlist and decides whether human confirmation is required.
  • Kubernetes executor — a small, isolated service that runs pre-defined, parameterized commands. It is the only component with cluster credentials.
  • RBAC service account — the scoped identity the executor uses against the Kubernetes API.
  • Audit log storage — an append-only record of every request, decision, and command.
  • Observability layer — metrics, logs, and traces for the agent itself.
  • Optional runbook/RAG knowledge base — internal docs that inform troubleshooting guidance.

The flow looks like this:

User
 → Web Chat UI
 → Backend API
 → AI Orchestrator
 → Policy / Approval Layer
 → Kubernetes Executor
 → Kubernetes API Server
 → Results back to Chat

The critical design decision is that the LLM never talks to the cluster directly. It proposes structured actions; the policy layer and executor decide whether and how they run. The model reasons, the platform executes.

Request Lifecycle

Walking through one request makes the boundaries concrete:

  1. The user asks a question: “restart the payments deployment in staging.”
  2. The backend authenticates the user and forwards the message to the orchestrator.
  3. The AI classifies intent and produces a structured action — not a shell string, but something like {action: rollout_restart, namespace: staging, deployment: payments}.
  4. The policy layer checks: is this action on the allowlist? Is this user allowed to perform it in this namespace and environment?
  5. Because it’s a write operation, the agent responds with what it intends to do and waits for explicit approval in the chat.
  6. On approval, the executor runs the corresponding controlled command — ideally with a dry-run first for mutating operations.
  7. The output goes back to the AI, which summarizes it: what happened, current rollout status, and a recommended follow-up (e.g., “watch rollout status for the next two minutes”).
  8. The full exchange — user identity, intent, proposed action, approval, command, output — is written to the audit log.

Read-only requests follow the same path but skip the approval step. Either way, the response always includes the command trace and raw result alongside the summary.

Read-Only Operations

Most of the agent’s daily value comes from inspection, and this is where you can be relatively generous:

  • Get and describe pods, deployments, services, and nodes.
  • Retrieve logs for a pod or container, with sensible line limits.
  • List and summarize events in a namespace.
  • Check rollout status of a deployment.
  • Show resource usage via top pods / top nodes when a metrics server is available.
  • Give a namespace overview: what’s running, what’s failing, what’s pending.

“Relatively generous” does not mean unscoped. Read access still leaks information — environment variables in pod specs, log contents, internal service names. The service account should be scoped to the namespaces the agent is meant to cover, and log output should pass through redaction before it reaches the model or the user.

Write Operations

Write operations are a much shorter list, and every entry should be deliberate:

  • Rollout restart of a deployment.
  • Scale a deployment within pre-defined bounds.
  • Delete a failed pod — only when it’s owned by a controller that will recreate it.
  • Restart a job or cronjob when appropriate.
  • Apply a manifest — only from a trusted, version-controlled source, never from free-form chat input.

And the hard rules:

  • No unrestricted shell. Ever.
  • No arbitrary kubectl — only parameterized, pre-defined actions.
  • No cluster-admin credentials anywhere in the system.
  • No destructive action without human approval.
  • No write action in production without explicit confirmation, regardless of how routine it seems.

If a proposed action can’t be expressed as a parameterized entry on the allowlist, the agent’s answer is “I can’t do that — here’s the command a human could run.” That’s a feature, not a limitation.

Safety and Governance

This is the section that determines whether the project survives its first security review:

  • Least-privilege RBAC — the service account gets exactly the verbs and resources the allowlist needs, nothing more.
  • Namespace isolation — separate service accounts (or separate agent deployments) per environment.
  • Command allowlist — the executor only knows how to run named, parameterized actions.
  • Denylist as a backstop — even within allowed actions, block dangerous parameters (e.g., kube-system, node draining).
  • Human approval for all write actions, with the approver’s identity recorded.
  • Dry-run before changes wherever the API supports it.
  • Audit logs — append-only, shipped off-cluster, covering every request and decision.
  • Session history tied to real user identity, not a shared bot account.
  • Environment separation — dev, QA, and prod are different trust zones with different allowlists.
  • Rate limits — an AI in a retry loop can hammer your API server surprisingly fast.
  • Secret redaction and output sanitization — before command output touches the LLM, scrub tokens, connection strings, and anything shaped like a credential.

Deployment Model

The agent runs in Kubernetes, deployed like any other workload:

  1. The web UI as a frontend deployment, exposed through an ingress with authentication in front — SSO, not a shared password.
  2. The backend API and orchestrator as a deployment in a dedicated namespace.
  3. The executor as its own deployment, bound to the scoped service account. Keeping it separate means the component with cluster credentials has the smallest possible codebase and attack surface.
  4. The allowlist and policy configuration in a ConfigMap, so allowed actions are reviewable in git and changes go through pull requests.
  5. LLM and API credentials in Secrets (or an external secret manager), never in config or images.
  6. Network policies restricting who can talk to the executor — only the policy layer, nothing else.
  7. Logs, metrics, and traces wired into your existing observability stack from day one.

Treat the agent’s own deployment with the same rigor you’d demand of any service that holds cluster credentials — because that’s exactly what it is.

Observability

You need to observe the agent as carefully as the clusters it operates:

  • Request volume and who’s asking what.
  • Command execution latency and failure rates.
  • Denied actions — a spike means either a misconfigured allowlist or someone probing the boundaries.
  • Approval events: requested, granted, rejected, expired.
  • Kubernetes API errors from the executor.
  • LLM latency, token usage, and cost per conversation.
  • A feedback mechanism for hallucinated or unsafe suggestions — users should be able to flag bad guidance, and someone should review those flags.
  • Audit trail completeness — periodically verify that every executed command has a corresponding audit record.

Runbook and RAG Integration

A vanilla LLM knows Kubernetes in general. It doesn’t know your Kubernetes. Retrieval over internal knowledge closes that gap:

  • Runbooks and incident postmortems.
  • Service ownership and escalation docs.
  • Architecture documentation.
  • Alert definitions and what they actually mean in your environment.
  • Deployment standards and known failure modes.

With that context, “checkout-api is crashlooping” becomes “checkout-api is crashlooping, which the March postmortem attributes to connection pool exhaustion after config changes — check the recent ConfigMap history first.”

One boundary must stay firm: RAG informs reasoning; it never overrides policy. A runbook that says “restart the database when X happens” does not authorize the agent to restart the database. Retrieved knowledge shapes suggestions; the allowlist and approval flow decide what actually executes.

Production Readiness Checklist

Before the agent touches anything real:

  • RBAC reviewed and approved by whoever owns cluster security.
  • Command allowlist defined, version-controlled, and reviewed.
  • Write approval flow enabled and tested — including the rejection path.
  • Audit logs enabled, shipped off-cluster, and retained.
  • Secrets protected and redaction verified against real log samples.
  • Production namespace restrictions configured and tested from the agent itself.
  • Observability dashboards and alerts for the agent in place.
  • Rollback plan documented — including how to disable the agent entirely.
  • Human escalation path defined: when the agent can’t help, it should say who can.

Common Mistakes

Patterns I’d flag in any design review of a system like this:

  • Giving the agent cluster-admin “just to get started.” You will not walk it back later.
  • Allowing arbitrary kubectl because the allowlist feels restrictive. The restriction is the point.
  • Skipping approvals for “safe” write actions. Safe is context-dependent; approvals are not.
  • Exposing secrets in logs by piping raw pod output through the LLM without redaction.
  • Mixing dev and prod permissions in one service account.
  • Relying only on AI reasoning for safety instead of enforcing it in the policy layer. The model is a component that fails; plan for it.
  • Not storing audit logs, then having nothing to reconstruct after an incident.
  • No rollback strategy for the agent’s own actions or its own deployment.
  • No ownership model — if nobody owns the agent, nobody maintains its allowlist, and it quietly rots into a risk.

Final Perspective

A Kubernetes ChatOps AI agent should be treated as an operational control plane, not just a chatbot. The natural-language interface is the visible part, but it’s the least important one. The real value is the combination underneath: AI reasoning to interpret intent and summarize state, a policy layer that constrains what can happen, an executor with least-privilege credentials, audit trails that make every action reconstructable, and a human who approves anything that changes the cluster.

Built that way, the agent doesn’t replace your existing tools or your judgment — it removes the friction between them. And on a bad day, when something is down and context is scattered across five systems, that friction is exactly what you want gone.