OpenShift cluster health snapshot: oc adm, nodes, MachineConfigPools and updates

OpenShift is administered mostly through the API: the platform’s components are cluster Operators, node configuration is declared in MachineConfig objects, and updates are run by the Cluster Version Operator. Day-2 administration is therefore largely a matter of reading the right status objects. This page summarizes the key ones and gives a script that checks them in one go. You need the cluster-admin role.

The objects to watch

• ClusterVersion (oc get clusterversion, oc adm upgrade): current version, update channel, available updates and update progress.
• ClusterOperators (oc get clusteroperators): every platform component with its Available, Progressing and Degraded conditions.
• Nodes (oc get nodes, oc adm top nodes, oc describe node): readiness, roles, versions and resource use.
• MachineConfigPools (oc get mcp): the Machine Config Operator applies operating system configuration per pool, typically master and worker. UPDATED=True means all nodes run the current rendered config; DEGRADED or a stuck UPDATING pool blocks updates.
• Certificate signing requests (oc get csr): new or replaced nodes need their CSRs approved, normally automatically.
• OLM (oc get subscriptions,csv -A): add-on Operators and their install state.

Update channels

Updates are offered through channels tied to a minor version. For OpenShift Container Platform the documentation describes candidate-4.y (early, unsupported builds for testing), fast-4.y (releases as soon as they are generally available), stable-4.y (the same releases after a promotion delay while data is collected) and eus-4.y (for moving between extended update support versions). The OKD documentation lists its own stable-4 channel. Check which channels and targets your cluster sees before changing anything:

oc adm upgrade
oc adm upgrade channel stable-4.18
oc adm upgrade --to-latest=true

Warning: oc adm upgrade --to-latest or --to starts a real cluster update that rolls through control plane and worker nodes with reboots. Read the release notes, make sure every cluster Operator is healthy and no MachineConfigPool is degraded, and take an etcd backup first. Do not use --force: the documentation warns it bypasses release verification and precondition checks.

Node maintenance

oc adm cordon worker-1.example.com
oc adm drain worker-1.example.com --ignore-daemonsets --delete-emptydir-data
oc adm uncordon worker-1.example.com
oc debug node/worker-1.example.com

Warning: draining evicts all pods from the node and discards their emptyDir data. List them first with oc get pods -A -o wide --field-selector spec.nodeName=NODE and make sure PodDisruptionBudgets allow it. Use oc debug node/NODE for a shell on a node instead of SSH; the host file system is under /host.

must-gather for support cases

oc adm must-gather
oc adm must-gather --dest-dir=./mg
oc adm must-gather -- /usr/bin/gather_audit_logs
oc adm inspect clusteroperator/ingress

oc adm must-gather runs a pod in a temporary project, collects cluster diagnostics and copies them to a local must-gather.local.* folder. Compress it with tar before uploading. Extra --image arguments gather data for specific products. The output contains cluster details, so treat it as sensitive.

The script

This script saves the outputs listed above into a folder and prints a warning for every cluster Operator that is unavailable or degraded, every node that is not Ready, every degraded or updating MachineConfigPool, and pending CSRs. It exits with status 1 if it found anything, so you can run it from a scheduled job. It only reads.

#!/usr/bin/env bash
set -uo pipefail

OUT=${1:-"./cluster-health-$(date '+%Y%m%d-%H%M%S')"}
ISSUES=0

command -v oc >/dev/null 2>&1 || { echo "oc not found in PATH" >&2; exit 1; }
oc whoami >/dev/null 2>&1 || { echo "not logged in: run oc login first" >&2; exit 1; }
mkdir -p "$OUT"

section() {
  local name=$1; shift
  {
    echo "\$ $*"
    "$@" 2>&1 || echo "(command failed or not permitted)"
    echo
  } >> "$OUT/$name.txt"
}

warn() {
  echo "WARN: $*" | tee -a "$OUT/summary.txt"
  ISSUES=$((ISSUES + 1))
}

echo "Cluster health snapshot $(date '+%Y-%m-%d %H:%M:%S %z') as $(oc whoami)" > "$OUT/summary.txt"

section version oc get clusterversion
section version oc adm upgrade
section operators oc get clusteroperators
section nodes oc get nodes -o wide
section nodes oc adm top nodes
section machineconfig oc get machineconfigpools
section certificates oc get csr
section olm oc get subscriptions,clusterserviceversions -A
section pods oc get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded -o wide

while read -r name available degraded; do
  [[ -z "$name" ]] && continue
  [[ "$available" != "True" ]] && warn "clusteroperator $name Available=$available"
  [[ "$degraded" == "True" ]] && warn "clusteroperator $name Degraded=True"
done < <(oc get clusteroperators -o jsonpath='{range .items[*]}{.metadata.name}{" "}{range .status.conditions[?(@.type=="Available")]}{.status}{end}{" "}{range .status.conditions[?(@.type=="Degraded")]}{.status}{end}{"\n"}{end}' 2>/dev/null)

while read -r name ready; do
  [[ -z "$name" ]] && continue
  [[ "$ready" != "True" ]] && warn "node $name Ready=$ready"
done < <(oc get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}' 2>/dev/null)

while read -r name degraded updating; do
  [[ -z "$name" ]] && continue
  [[ "$degraded" == "True" ]] && warn "machineconfigpool $name Degraded=True"
  [[ "$updating" == "True" ]] && warn "machineconfigpool $name is still updating"
done < <(oc get machineconfigpools -o jsonpath='{range .items[*]}{.metadata.name}{" "}{range .status.conditions[?(@.type=="Degraded")]}{.status}{end}{" "}{range .status.conditions[?(@.type=="Updating")]}{.status}{end}{"\n"}{end}' 2>/dev/null)

PENDING_CSR=$(oc get csr --no-headers 2>/dev/null | grep -c Pending || true)
[[ "${PENDING_CSR:-0}" -gt 0 ]] && warn "$PENDING_CSR certificate signing request(s) pending"

if [[ $ISSUES -eq 0 ]]; then
  echo "No issues found by the quick checks." | tee -a "$OUT/summary.txt"
fi
echo "Snapshot written to $OUT"
exit $(( ISSUES > 0 ? 1 : 0 ))

Usage

chmod +x cluster-health.sh
oc login https://api.cluster.example.com:6443 -u admin
./cluster-health.sh                   # writes ./cluster-health-TIMESTAMP
./cluster-health.sh /tmp/health-before-update

A useful habit is to run it right before and after an update or node maintenance and compare the folders. A pending CSR or a pool still updating right after maintenance is often normal for a few minutes; a degraded Operator that stays degraded is not.

Official documentation

• OpenShift CLI administrator command reference
• Gathering data about your cluster (must-gather)
• OpenShift 4.18: Updating a cluster using the CLI