SecuryBlack
← Blog/Tutorial

Heartbeats: How to Know Your Backups Actually Run

·7 min read
Heartbeats: How to Know Your Backups Actually Run

The silent backup problem

You set up a cronjob that makes a backup every night. Perfect. But how do you know it actually runs? That it didn't fail last week and nobody noticed?

Most people find out when they try to restore and there's nothing to restore. It's the worst possible moment to find out: when you need it most and when you can no longer do anything about it.

This scenario is more common than it seems. Not because people don't make backups — most set them up. The problem is that a backup is a script that runs silently, with no interface, no visible confirmation. If it fails, there's no one to see it fail. The cronjob shows as active in crontab -l, the disk has space, there's no alarm signal. It simply didn't run, or it ran but failed before copying anything.

Why cronjobs that "work" fail

There's a set of classic failures that affect cronjobs that have been running smoothly for months:

Cron's environment is not your user environment. When you run a script manually, you have your PATH, your environment variables, your aliases. Cron starts with a minimal environment. If your script calls rclone or restic and those binaries are in /home/user/bin but not in /usr/local/bin, the cronjob will find them when you test manually and fail silently when cron runs it.

Relative paths. A script that works from /home/user may fail when cron runs it from / because relative paths no longer point where they should.

Full disk or destination unavailable. If your NAS is off, if the NFS mount dropped, or if the S3 bucket has access issues, the script may finish without error but without having copied anything.

Silent timeout. A backup that normally takes 20 minutes may take 3 hours if there's a network problem, and if you have no control mechanism, you'll never know.

What is a heartbeat

A heartbeat is a signal your script sends to SecuryBlack to confirm it ran successfully to the end. If the signal doesn't arrive within the configured time, you receive an alert.

The logic is inverted compared to traditional uptime monitoring. In uptime monitoring, you check if something is alive by making periodic requests. With heartbeats, it's your script that notifies when it finishes. If it doesn't notify, something went wrong.

This inversion is important: a heartbeat detects both that the script didn't run and that it ran but failed before reaching the end. If you put the curl at the end of the script, it will only arrive if everything before it worked correctly.

How to set it up

Add a line at the end of your backup script:

curl -fsS https://securyblack.com/api/heartbeat/YOUR_ID > /dev/null

The -f flag makes curl fail silently if the server returns an HTTP error. The -s flag suppresses normal output. The -S flag shows errors if there are any. Redirecting to /dev/null prevents output from contaminating cron logs.

That's it. If the script runs and reaches the end, the signal arrives. If it doesn't run or fails before — for any reason — it doesn't arrive, and SecuryBlack alerts you.

Configure the grace period

In SecuryBlack you can configure how much time can pass without receiving the heartbeat before considering it failed. This is called the grace period.

If your backup runs at 2:00 AM and normally takes between 15 and 45 minutes depending on how much data has changed, a grace period of 90 minutes is reasonable: if by 3:30 AM the ping hasn't arrived, something is wrong.

The grace period avoids false alarms from normal execution time variations without letting real problems go unnoticed for too long.

Script with error handling

A more robust pattern is to use set -e at the beginning of the script so any error stops it immediately, and thus the heartbeat is only sent if everything went well:

#!/bin/bash
set -euo pipefail

# Your backup logic here
rclone sync /data remote:backup/

# We only get here if everything above worked
curl -fsS https://securyblack.com/api/heartbeat/YOUR_ID > /dev/null

With set -euo pipefail, if any command in the script fails, the script stops and the curl doesn't execute, which triggers the alert.

Typical use cases

Heartbeats aren't just for backups. Any scheduled task you need to verify is running is a valid use case:

  • Nightly backups with rsync, rclone, restic, or borgbackup
  • Synchronization scripts between servers or to cloud storage
  • Maintenance tasks — log rotation, temporary file cleanup, local database updates
  • Notification scripts — if you have a script that sends you a daily summary and one day it doesn't arrive, you want to know why
  • Any critical cronjob whose silent failure would have consequences

The rule is simple: if a script can fail and no one would know until it's too late, it deserves a heartbeat.

Want to set up heartbeats for your scripts and backups? SecuryBlack Heartbeats is available for free during the beta — add one curl line and you're done.