Collect an OpenShift project report with oc

When an app misbehaves, the first ten minutes go into running the same oc get and oc describe commands. This script runs them all for one project and writes the output into a folder you can read, diff or attach to a ticket. It only reads: it never edits, scales or deletes anything.

The script

#!/usr/bin/env bash
set -euo pipefail

usage() {
  cat <<USAGE
Usage: $(basename "$0") [-n PROJECT] [-o OUTPUT_DIR] [-l LOG_LINES]

Collects a read-only report of one OpenShift project with oc.

  -n PROJECT     project to inspect (default: the current oc project)
  -o OUTPUT_DIR  where to write the report (default: ./report-PROJECT-TIMESTAMP)
  -l LOG_LINES   log lines to keep per problem pod (default: 100)
USAGE
}

PROJECT=""
OUT=""
LOG_LINES=100

while getopts ":n:o:l:h" opt; do
  case "$opt" in
    n) PROJECT=$OPTARG ;;
    o) OUT=$OPTARG ;;
    l) LOG_LINES=$OPTARG ;;
    h) usage; exit 0 ;;
    :) echo "option -$OPTARG needs a value" >&2; usage >&2; exit 2 ;;
    \?) echo "unknown option -$OPTARG" >&2; usage >&2; exit 2 ;;
  esac
done

command -v oc >/dev/null 2>&1 || { echo "oc not found in PATH" >&2; exit 1; }
oc whoami >/dev/null 2>&1 || { echo "not logged in: run oc login first" >&2; exit 1; }

if [[ -z "$PROJECT" ]]; then
  PROJECT=$(oc config view --minify -o jsonpath='{..namespace}')
  [[ -n "$PROJECT" ]] || { echo "no current project: pass -n PROJECT" >&2; exit 1; }
fi

STAMP=$(date '+%Y%m%d-%H%M%S')
OUT=${OUT:-"./report-$PROJECT-$STAMP"}
mkdir -p "$OUT/logs"

run() {
  local file=$1; shift
  {
    echo "\$ $*"
    "$@" 2>&1 || echo "(command failed or not permitted)"
    echo
  } >> "$OUT/$file"
}

{
  echo "Project report: $PROJECT"
  echo "Generated:      $(date '+%Y-%m-%d %H:%M:%S %z')"
  echo "User:           $(oc whoami)"
  echo "Server:         $(oc config view --minify -o jsonpath='{.clusters[0].cluster.server}')"
  echo
} > "$OUT/summary.txt"

run summary.txt oc status -n "$PROJECT"
run workloads.txt oc get deployments,statefulsets,daemonsets -n "$PROJECT" -o wide
run workloads.txt oc get pods -n "$PROJECT" -o wide
run networking.txt oc get services -n "$PROJECT" -o wide
run networking.txt oc get routes -n "$PROJECT"
run builds.txt oc get buildconfigs,builds -n "$PROJECT"
run builds.txt oc get imagestreams -n "$PROJECT"
run storage.txt oc get pvc -n "$PROJECT"
run config.txt oc get configmaps,secrets -n "$PROJECT"
run events.txt oc get events -n "$PROJECT" --sort-by=.lastTimestamp
run events.txt oc get events -n "$PROJECT" --field-selector type=Warning --sort-by=.lastTimestamp

PROBLEM_PODS=$(oc get pods -n "$PROJECT" \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{range .status.containerStatuses[*]}{.ready}{","}{end}{"\n"}{end}' \
  | awk '$2 != "Succeeded" && ($2 != "Running" || $3 ~ /false/) { print $1 }')

if [[ -z "$PROBLEM_PODS" ]]; then
  echo "No problem pods found." >> "$OUT/summary.txt"
else
  echo "Problem pods:" >> "$OUT/summary.txt"
  for pod in $PROBLEM_PODS; do
    echo "  $pod" >> "$OUT/summary.txt"
    run "logs/$pod.describe.txt" oc describe pod "$pod" -n "$PROJECT"
    run "logs/$pod.log" oc logs "$pod" -n "$PROJECT" --all-containers --tail="$LOG_LINES"
    run "logs/$pod.previous.log" oc logs "$pod" -n "$PROJECT" --all-containers --previous --tail="$LOG_LINES"
  done
fi

echo "Report written to $OUT"

Usage

chmod +x project-report.sh
oc login https://api.cluster.example.com:6443 -u developer

./project-report.sh                     # current project
./project-report.sh -n shop-dev -l 300  # another project, more log lines
./project-report.sh -n shop-dev -o /tmp/shop-report

What you get

• summary.txt: who ran it, against which API server, oc status and the list of problem pods.
• workloads.txt, networking.txt, builds.txt, storage.txt, config.txt: deployments, pods, services, routes, builds, image streams, PVCs, and ConfigMap and Secret names.
• events.txt: all events sorted by time, then only warnings.
• logs/: for every pod that is not Running and ready (and not Succeeded), its oc describe output, current logs and logs of the previous, crashed container.

How it decides a pod is a problem

A JSONPath query prints each pod’s name, phase and the readiness of each container. awk keeps pods that are not Succeeded and are either not Running or have a container with ready=false. That catches Pending, CrashLoopBackOff (phase Running but not ready) and failed pods.

Notes

• You only need view rights in the project. Commands you are not allowed to run are recorded as failed instead of stopping the script.
• Secret contents are never read, only their names. Logs can still contain sensitive data, so check the folder before sharing it.
• Events are kept for a limited time by the cluster, so run the script soon after a problem happens.

Official documentation

• OpenShift CLI developer command reference
• Getting started with the OpenShift CLI