This checker vs. watching your backup's exit code yourself

What watching the exit code tells you

Checking the return code is the cheapest, most common way anyone verifies a scheduled job, and it is not wrong to do it: a nonzero exit is real signal that something broke, and most backup tools are honest about reporting one when they crash, lose their connection, or hit a permission error. Cron's own default of mailing you the job's output only on nonzero exit is built on exactly this assumption, and so is a green line in systemctl status for a job that already ran.

What none of those checks look at is the artifact the command was supposed to produce. The exit code is the process's own opinion of how it went, decided the moment the process's last line runs -- it has no way to represent "I returned successfully, and what I wrote is unusable," because those are two separate facts and only one of them is encoded in a number between 0 and 255.

A real run, from this checker's own shipped example

This is not a hypothetical. It is one of the run records shipped with this checker's own test suite, reproduced exactly:

{
  "tool": "pg_dump",
  "exit_code": 0,
  "started_at": "2026-08-25T03:00:00Z",
  "finished_at": "2026-08-25T03:00:04Z",
  "artifact": {
    "path": "/backups/nightly.sql.gz",
    "exists": true,
    "size_bytes": 312,
    "modified_at": "2026-08-25T03:00:03Z"
  },
  "expected": { "max_age_hours": 26, "min_size_bytes": 1000000 }
}

pg_dump connected, ran, and exited 0 in four seconds. Cron has nothing to mail, because nothing failed by its own measure. systemctl status on the timer's service shows the last run as inactive (exited with 0) -- the same thing it shows for a full, correct dump. A human running echo $? right after the job sees 0 and moves on.

Running backup-evidence check against the same record instead:

$ PYTHONPATH=src python3 -m backup_evidence.cli check run.json
{
  "latest": {
    "status": "suspicious",
    "reasons": [
      "artifact is 312 bytes, below the expected minimum of 1.0 MB"
    ],
    "tool": "pg_dump",
    "finished_at": "2026-08-25T03:00:04Z"
  }
}
$ echo $?
1

The command that ran connected fine; whatever wrote the 312-byte file afterward -- a full destination, a broken pipe, a dump that silently matched zero tables -- never touched the process's own return value. The exit code and the artifact disagree, and only one of the two checks above is looking at the artifact.

Side by side

Using both together

Nothing here argues for dropping cron mail or your systemd unit status -- a job that cannot even start is a real failure this checker will never see, because it only looks at records for jobs that ran. The change is adding one more check after the exit code, so a run that returns 0 still has to clear a second, independent test before you treat it as done:

#!/bin/sh
set -e
pg_dump mydb | gzip > /backups/nightly.sql.gz
PYTHONPATH=src python3 -m backup_evidence.cli record \
    --tool pg_dump --artifact /backups/nightly.sql.gz \
    --min-size-bytes 1000000 --max-age-hours 26 \
    --output /backups/last-run.json -- true
PYTHONPATH=src python3 -m backup_evidence.cli check /backups/last-run.json

The last line exits 1 unless the newest recorded run is verified, so it composes with whatever already watches this script's own exit code -- cron mail on nonzero, a systemd OnFailure= unit, or a monitoring ping gated on &&. The command you were already watching now means something closer to "the backup ran and left a plausible artifact," instead of just "the backup ran."

Open the verification checker to run this against a record of your own, or see this checker vs. Healthchecks.io for the layer that watches whether the job runs on schedule at all.

Common questions

Isn't checking $? or the systemd unit status good enough?

It tells you the command finished and returned success, which is real information and worth having. It cannot tell you whether the thing the command was supposed to produce actually exists at the size and freshness you expect, because the exit code and the artifact are two separate facts and a broken pipeline can get the first one right while getting the second one badly wrong. A backup script that hits a full disk partway through, or a database dump that connects but returns no rows, typically still reaches its last line and still exits 0.

What does watching the exit code actually miss, concretely?

Anything that happens to the output after the command decides it succeeded. On this checker's own shipped example, a nightly pg_dump run exits 0, finishes in four seconds, and writes an artifact that is 312 bytes -- far below the 1.0 MB minimum that run's own record expected. Cron mail reports nothing, because there is no error to mail. systemctl status shows the unit as inactive (exited with 0). A human glancing at either would see a clean run. Running backup-evidence check against the same record reports it suspicious, with the reason spelled out: "artifact is 312 bytes, below the expected minimum of 1.0 MB", and the check command exits 1.

How would I have actually caught the 312-byte example by hand?

By remembering to run ls -la or du -sh on the artifact after every single backup, forever, across every host and every schedule -- and by remembering what size counts as normal for that particular job, which drifts over time as the underlying data grows. That is a manual step nothing in the exit code or the unit status prompts you to take, which is exactly why it tends to get skipped the day it would have mattered.

Does this replace watching my job's exit code or unit status?

No, and it is not trying to. A nonzero exit code is still real signal -- this checker's own classification treats it as an automatic failed, no further checks needed. What it adds is the layer above that: for the runs that returned 0, did the artifact they left behind actually match what you expected? Keep watching exit codes for the job that crashes outright; use this for the job that quietly succeeds at doing nothing useful.

Do I have to change how I already run the backup to use this?

No. It reads a run record you can build from information you already have -- the tool name, the exit code, when it finished, and a stat of the artifact -- and it never runs the backup command itself. The record subcommand can capture that record around your existing command with no change to the command itself, or you can write the record by hand from a script you already maintain.