verify-latest-backup: alert when the newest dump is too old or too small

Most backup failures are quiet: the job stops running, or it keeps running and produces a 20-byte file with an error message inside. A monitoring check that looks at the result, not the job, catches both. This script answers two questions about a directory of dumps: is the newest file recent enough, and is it big enough to plausibly contain data?

The script

Save as /usr/local/bin/verify-latest-backup.sh and make it executable. It needs GNU find (standard on Linux) for the -printf option.

#!/usr/bin/env bash
set -euo pipefail

usage() {
  echo "usage: $0 -d DIR [-p PATTERN] [-a MAX_AGE_HOURS] [-s MIN_SIZE_BYTES]"
  echo "exit codes: 0 ok, 2 no matching file, 3 newest file too old, 4 newest file too small, 5 archive integrity test failed"
}

DIR=""
PATTERN="*"
MAX_AGE_HOURS=26
MIN_SIZE=1024

while getopts ":d:p:a:s:h" opt; do
  case "$opt" in
    d) DIR="$OPTARG" ;;
    p) PATTERN="$OPTARG" ;;
    a) MAX_AGE_HOURS="$OPTARG" ;;
    s) MIN_SIZE="$OPTARG" ;;
    h) usage; exit 0 ;;
    *) usage; exit 1 ;;
  esac
done

if [ -z "$DIR" ] || [ ! -d "$DIR" ]; then
  usage
  exit 1
fi

newest=$(find "$DIR" -maxdepth 1 -type f -name "$PATTERN" -printf '%T@ %s %p\n' | sort -n | tail -n 1)
if [ -z "$newest" ]; then
  echo "CRITICAL: no file matching '$PATTERN' in $DIR"
  exit 2
fi

mtime=${newest%% *}
rest=${newest#* }
size=${rest%% *}
file=${rest#* }
mtime=${mtime%.*}
now=$(date +%s)
age_hours=$(( (now - mtime) / 3600 ))

echo "newest: $file"
echo "age:    ${age_hours}h (limit ${MAX_AGE_HOURS}h)"
echo "size:   ${size} bytes (minimum ${MIN_SIZE})"

if [ "$age_hours" -gt "$MAX_AGE_HOURS" ]; then
  echo "CRITICAL: newest backup is older than ${MAX_AGE_HOURS} hours"
  exit 3
fi

if [ "$size" -lt "$MIN_SIZE" ]; then
  echo "CRITICAL: newest backup is smaller than ${MIN_SIZE} bytes"
  exit 4
fi

case "$file" in
  *.gz)
    if ! gzip -t "$file"; then
      echo "CRITICAL: gzip integrity test failed for $file"
      exit 5
    fi
    ;;
  *.zst)
    if command -v zstd >/dev/null && ! zstd -t -q "$file"; then
      echo "CRITICAL: zstd integrity test failed for $file"
      exit 5
    fi
    ;;
esac

echo "OK: latest backup looks healthy"
exit 0

Usage

verify-latest-backup.sh -d /var/backups/db/postgresql -p 'app_*.dump' -a 26 -s 1000000
verify-latest-backup.sh -d /var/backups/db/mysql -p '*.sql.gz' -a 26 -s 50000
echo "exit code: $?"

• -d is the directory, -p a shell glob for the file names (quote it so your shell does not expand it), -a the maximum age in hours and -s the minimum size in bytes.
• A 26-hour limit for a nightly job leaves room for a slow run or a delayed start without masking a missed night.
• Set the minimum size to roughly half of a normal dump. If your dumps are usually 80 MB and today’s is 2 KB, the database was probably unreachable and the file contains an error message.
• Exit codes are distinct so a wrapper can tell the cases apart: 2 nothing found, 3 too old, 4 too small, 5 the archive fails its integrity test.

Wire it to an alert

Run it from a systemd timer an hour after the backup job. A non-zero exit puts the unit into the failed state, which shows up in systemctl --failed, in the Server health check script and in any monitoring agent that watches systemd units. To send an email or chat message, add an OnFailure= line to the service pointing at a notify unit:

[Unit]
Description=Verify newest PostgreSQL dump
OnFailure=notify-failure@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/verify-latest-backup.sh -d /var/backups/db/postgresql -p '*.dump' -a 26 -s 1000000

For Prometheus, run the script from a wrapper that writes a backup_check_status metric into the node_exporter textfile collector directory, and alert when the value is not 0 for more than an hour.

What this does not prove

A file that is fresh, large and decompresses cleanly can still be a dump of the wrong database, or a dump that will not import because of a version mismatch. This check is the daily smoke test; the monthly restore drill is the real proof. Use both.

Official documentation

• GNU findutils manual
• systemd.unit: OnFailure=