Alerting on systemd failures: Email, Slack, Teams, SNS and more

A practical, copy‑pasteable guide for catching job failures from your systemd services and timers—and actually hearing about them.

Why this guide

Cron jobs fail silently. systemd is better: it tracks unit state, captures logs, and can trigger follow‑up units on failure. This guide shows how to wire those features into the channels you already use—email, Slack/Teams, wall, MOTD, and generic webhooks (PagerDuty, healthchecks, custom REST). Everything here is production‑oriented: templated units, minimal dependencies, and secrets handled via EnvironmentFile=.

Quick wins

1. Add an OnFailure handler to the service that your timer triggers.
2. Drop in one of the templated notifier services below (email, Slack, webhook, etc.).
3. Reload systemd and test with a failing job.

Test a failure and confirm notifications fire:

How systemd decides something “failed”

A unit is considered failed when its Result is not success (e.g., exit-code, signal, timeout). When a service fails, systemd queues all units listed in OnFailure= on that service. Those notifier units receive the original unit name as their instance (%i/%n) so they can include context in messages.

Useful fields you can query:

Reusable metadata helper

All notifier scripts below build a small context block for consistent messages:

Make it executable:

Option A — Email on failure (local MTA or relay)

Script

Unit template

Config

Reload: sudo systemctl daemon-reload

Tip: If you don’t run a local MTA, configure ssmtp, msmtp, or relay through Postfix/Exim to your provider.

Option B — Slack via Incoming Webhook

Script (Slack)

Unit template

Config

Option B2 — Microsoft Teams via Incoming Webhook

Teams expects a slightly different JSON shape (an Adaptive Card or a simple message). Below is a minimal message card using the Incoming Webhook connector.

Script (Teams)

Unit template

Config

Option C — Generic REST/webhook (PagerDuty, Healthchecks.io, custom)

PagerDuty (Events v2) — drop‑in example

You can either use the generic webhook above with a TEMPLATE, or create a dedicated script:

Healthchecks.io: create a check with a fail endpoint; set WEBHOOK_URL to that URL and post a short JSON body.

Option D — Broadcast to logged-in users with wall

Option E — Append to /etc/motd.d/ so admins see it next login

Option F — AWS SNS (works with CloudWatch/EventBridge)

Use this when your fleet is in AWS and you want failures to fan out to email, SMS, Lambda, or incident tools via Amazon SNS. On EC2/ECS, prefer an IAM role for credentials; elsewhere configure aws CLI with an access key that can sns:Publish.

Script (SNS)

Unit template

Config

Tip: Point the SNS topic at multiple subscriptions (email, SMS, webhook, Lambda). You can also bridge into EventBridge for routing/enrichment.

Other useful targets

• Discord via webhook (Slack JSON usually works with minor tweaks)
• Google Chat incoming webhook
• Opsgenie REST API
• Splunk On-Call (VictorOps) REST API
• Email-to-ticket (Jira, Zendesk) via a dedicated mailbox
• AWS SES (if you want to send email directly from AWS)

Wiring it up: OnFailure best practices

• Attach OnFailure= to the .service, not the .timer. Failures occur on the service unit your timer triggers (e.g., backup.service).
• Use multiple OnFailure= lines to fan out to multiple channels.
• Prefer templated notifiers (notify-foo@.service) and pass the failing unit via %n.
• Keep secrets in EnvironmentFile= readable by root only (e.g., /etc/systemd/system/notify-*.conf, 0640).
• Add RuntimeMaxSec= to convert hangs into failures you can alert on.
• For chat tools, escape content and include recent journal lines.

Example (attach to your service):

Reload after changes:

Rate limiting & deduping

It’s easy to spam yourself if a flapping service fails repeatedly.

• Wrap notifiers with systemd’s built‑in start‑rate limiting:

• Or point OnFailure to an aggregator unit that writes events to a spool directory. A separate .timer runs every few minutes to summarize and send one message.

Troubleshooting

• Did the notifier run?
  
  systemctl status notify-email@backup.service journalctl -u notify-email@backup.service -b

  1 2 3

  systemctl status notify-email@backup.service journalctl -u notify-email@backup.service -b

• What was the service’s failure mode?
  
  systemctl show backup.service -p Result,ExecMainStatus

  1 2	systemctl show backup.service -p Result,ExecMainStatus

• Simulate failures safely:
  
  systemd-run --unit=failure-test.service /bin/false

  1 2	systemd-run --unit=failure-test.service /bin/false

• Timer didn’t run? Ensure your .timer has Unit=<service> and is enabled and active. Check with:
  
  systemctl list-timers --all

  1 2	systemctl list-timers --all

Security notes

• Store webhooks and email recipients in root‑readable EnvironmentFile= files.
• Avoid embedding secrets in unit files; unit files are often world‑readable.
• Validate outbound destinations (firewall allow‑list) if servers are sensitive.

Optional: catching “hung” services with WatchdogSec=

If you own the service code, consider integrating sd_notify(READY=1, WATCHDOG=1) and setting WatchdogSec=. When the process stops heartbeating, systemd marks it failed and your OnFailure chain runs. For simple shell jobs, prefer RuntimeMaxSec= as shown earlier.

Appendix: minimal dependencies

• bash, curl, wall, mail/sendmail (optional)
• jq for the generic webhook script (optional; remove if you supply TEMPLATE)

Example: all channels for a logrotate job

Enjoy quiet servers that yell only when it matters.