Collect describe, events and logs of a failing Kubernetes pod

When a pod crashes, gets stuck or restarts, the evidence is spread over several commands, and some of it (events, previous logs) disappears after a while. This script grabs all of it in one go and writes each piece to its own file. It only reads from the cluster; it never changes or deletes anything.

The script

#!/usr/bin/env bash
set -euo pipefail

usage() {
  echo "Usage: $(basename "$0") POD [-n NAMESPACE] [-o OUTPUT_DIR] [-t TAIL_LINES]" >&2
  exit 2
}

[[ $# -ge 1 && $1 != -* ]] || usage
POD=$1
shift
NAMESPACE=$(kubectl config view --minify -o jsonpath='{..namespace}' 2>/dev/null || true)
NAMESPACE=${NAMESPACE:-default}
OUT=""
TAIL=500

while getopts ":n:o:t:" opt; do
  case $opt in
    n) NAMESPACE=$OPTARG ;;
    o) OUT=$OPTARG ;;
    t) TAIL=$OPTARG ;;
    *) usage ;;
  esac
done

command -v kubectl >/dev/null || { echo "kubectl not found" >&2; exit 1; }
kubectl get pod "$POD" -n "$NAMESPACE" >/dev/null || exit 1

OUT=${OUT:-"debug-${NAMESPACE}-${POD}-$(date +%Y%m%d-%H%M%S)"}
mkdir -p "$OUT"
echo "Collecting pod $NAMESPACE/$POD into $OUT"

run() {
  local file=$1
  shift
  if "$@" >"$OUT/$file" 2>&1; then
    echo "  ok    $file"
  else
    echo "  empty $file (command failed, see file)"
  fi
}

run status.txt kubectl get pod "$POD" -n "$NAMESPACE" -o wide
run describe.txt kubectl describe pod "$POD" -n "$NAMESPACE"
run events.txt kubectl events -n "$NAMESPACE" --for "pod/$POD"
run pod.yaml kubectl get pod "$POD" -n "$NAMESPACE" -o yaml

CONTAINERS=$(kubectl get pod "$POD" -n "$NAMESPACE" \
  -o jsonpath='{.spec.initContainers[*].name} {.spec.containers[*].name}')

for c in $CONTAINERS; do
  run "logs-$c.txt" kubectl logs "$POD" -n "$NAMESPACE" -c "$c" --tail="$TAIL" --timestamps
  run "logs-$c-previous.txt" kubectl logs "$POD" -n "$NAMESPACE" -c "$c" --tail="$TAIL" --timestamps --previous
done

NODE=$(kubectl get pod "$POD" -n "$NAMESPACE" -o jsonpath='{.spec.nodeName}')
if [[ -n "$NODE" ]]; then
  run node-describe.txt kubectl describe node "$NODE"
else
  echo "  note  pod has no node yet (Pending); check events.txt for scheduler messages"
fi

echo
echo "Container states:"
kubectl get pod "$POD" -n "$NAMESPACE" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.state}{"\tlastState="}{.lastState}{"\trestarts="}{.restartCount}{"\n"}{end}'
echo
echo "Done. Review the files in $OUT before sharing them: pod.yaml can contain plain environment values."

Usage

chmod +x k8s-pod-debug.sh
./k8s-pod-debug.sh web-7d9c5b8f6-x2kqp -n shop
./k8s-pod-debug.sh web-7d9c5b8f6-x2kqp -n shop -o /tmp/case-123 -t 2000

What you get

• describe.txt: container states, last termination reason and exit code, probe settings, and recent events. Usually the fastest way to the cause.
• events.txt: events for this pod only. The API server keeps events only for a limited time, so collect them early.
• logs-<container>-previous.txt: output of the last crashed instance. For CrashLoopBackOff this is the file that matters, because the current container may not have printed anything yet.
• node-describe.txt: node conditions (MemoryPressure, DiskPressure) and allocated resources, useful for evictions and OOM kills.

Notes

• It needs kubectl events, which recent kubectl versions include. On an old client, replace that line with kubectl get events -n NS --field-selector involvedObject.name=POD.
• A missing -previous log is normal for a container that has not restarted.
• You need RBAC permission to get pods, pods/log, events and nodes. Without node access, that one file will contain a Forbidden error and the rest still works.
• Review the output before attaching it to a ticket: pod.yaml shows environment values written directly in the spec, and logs may contain personal data.

Official documentation

• Debug Pods
• kubectl events
• kubectl Quick Reference