CDE Operational Runbooks

Identify the Runaway Workspace

# Find workspaces with abnormal resource usage
kubectl top pods -n workspaces --sort-by=cpu | head -10
kubectl top pods -n workspaces --sort-by=memory | head -10

# Check for high process counts inside workspace
kubectl exec -it WORKSPACE_POD -n workspaces -- ps aux --sort=-%cpu | head -20

# Review agent session logs
kubectl logs WORKSPACE_POD -n workspaces --tail=200 | grep -i "agent\|llm\|copilot"

Kill Runaway Processes and Contain

# Kill the agent process tree inside the workspace
kubectl exec -it WORKSPACE_POD -n workspaces -- pkill -9 -f "claude\|copilot\|aider\|agent"

# If subprocess spawning is out of control, apply cgroup limits
kubectl exec -it WORKSPACE_POD -n workspaces -- bash -c 'echo 50 > /sys/fs/cgroup/pids/pids.max'

# If workspace is unresponsive, force-stop it
coder workspaces stop USERNAME/WORKSPACE_NAME
kubectl delete pod WORKSPACE_POD -n workspaces --grace-period=10 --force

Check LLM API Token Spend

# Check proxy/gateway logs for token consumption
kubectl logs -n ai-gateway -l app=litellm --tail=100 | grep -i "tokens\|cost\|budget"

# If using a cost proxy, check budget status
curl -s http://litellm-proxy:4000/budget/info | jq

# Revoke API key if spend is out of control
curl -X POST http://litellm-proxy:4000/key/delete \
  -H "Authorization: Bearer ADMIN_KEY" \
  -d '{"keys": ["sk-compromised-key-id"]}'

Post-Incident Hardening

After containment, apply preventive controls:

• Set CPU/memory resource limits on agent workspace templates
• Configure per-user LLM API token budgets with hard caps
• Enable max-process-count (pids.max) cgroup limits in templates
• Add agent session timeout policies (e.g., 30-minute idle kill)
• Set up alerting on sustained high CPU/memory in agent workspaces

Isolate and Freeze the Workspace Immediately

# Apply network policy to block all egress immediately
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: emergency-isolate-agent
  namespace: workspaces
spec:
  podSelector:
    matchLabels:
      workspace: WORKSPACE_ID
  policyTypes:
    - Egress
  egress: []
EOF

# Freeze the workspace (do NOT delete - preserve for forensics)
coder workspaces stop USERNAME/WORKSPACE_NAME

# Revoke the user's agent API keys
curl -X POST http://litellm-proxy:4000/key/delete \
  -H "Authorization: Bearer ADMIN_KEY" \
  -d '{"keys": ["sk-affected-key"]}'

Collect Forensic Evidence

# Snapshot the workspace volume before cleanup
kubectl get pvc -n workspaces -l workspace=WORKSPACE_ID -o json > /tmp/pvc-snapshot.json

# Export agent session logs
kubectl logs WORKSPACE_POD -n workspaces --all-containers > /tmp/agent-forensics.log

# Check network flow logs for unauthorized connections
kubectl logs -n kube-system -l app=calico-node --tail=500 | grep WORKSPACE_POD_IP

# Review audit logs for privilege escalation
kubectl logs -n workspaces WORKSPACE_POD | grep -iE "sudo\|chmod\|chown\|curl\|wget\|ssh"

# Check if secrets were accessed
kubectl logs WORKSPACE_POD -n workspaces | grep -iE "password\|token\|secret\|api.key\|credentials"

Rotate Exposed Credentials

# Rotate any secrets that were mounted in the workspace
kubectl get secret -n workspaces -l workspace=WORKSPACE_ID -o json | jq '.items[].metadata.name'

# Rotate Git credentials
# GitHub: Revoke PAT, generate new one
# GitLab: Revoke token in Admin > Credentials

# Rotate cloud provider credentials
aws iam update-access-key --access-key-id AKIA... --status Inactive --user-name cde-agent
aws iam create-access-key --user-name cde-agent

# Notify security team
/incident new "AI Agent Sandbox Escape" severity:sev1 --notify security-oncall

Harden Agent Sandbox Configuration

After investigation, implement stronger isolation:

• Deploy agent workspaces in Firecracker microVMs instead of containers
• Apply read-only filesystem with explicit writable paths only
• Enforce NetworkPolicy: allow-list only required egress (Git host, LLM API, registry)
• Use short-lived, scoped credentials injected via Vault or IRSA instead of long-lived keys
• Enable seccomp and AppArmor profiles to restrict syscalls
• Enable comprehensive audit logging for all agent workspace activity