Get alerts in Slack or Google Chat
The monitoring Grafana watches the runs it records and raises an alert when:
| Alert | When |
|---|---|
| Run failed | A run ended with errors in the last 10 minutes |
| Run much slower than usual | A run that just finished took more than 3 times its usual time |
| Run still running long after its usual time | A run has been going for more than 3 times its usual time, or for 6 hours if it has no history yet |
A run is a top-level workflow or pipeline, the same runs the dashboard counts. "Usual" is the median of that run's last 10 successful runs, used once it has run successfully 5 times. Runs shorter than a minute never count as slow.
The rules are on from the start. Without a chat to send to, they show under Alerting → Alert rules in Grafana only.
Send them to Slack
-
In Slack, create an incoming webhook for the channel the alerts should go to (Slack: sending messages using incoming webhooks).
-
Add it to
.env:PUTKI_ALERT_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T000/B000/XXXX -
Apply it:
docker compose -f docker-compose-monitoring.yml up -d.
Send them to Google Chat
-
In the Google Chat space, open Apps & integrations → Webhooks and add one (Google: send messages to Google Chat with incoming webhooks).
-
Add it to
.env:PUTKI_ALERT_GOOGLE_CHAT_WEBHOOK_URL=https://chat.googleapis.com/v1/spaces/AAAA/messages?key=...&token=... -
Apply it as above.
Set both to be told in both. Microsoft Teams is not offered yet.
What a message contains
Each alert names the project and the run, for example sales / load-orders failed, and links to the dashboard. One message is sent when the alert starts and one when it clears. An alert that stays active is sent again every 4 hours.
Tune the rules
| Setting | Default | Meaning |
|---|---|---|
PUTKI_ALERT_SLOW_FACTOR | 3 | "Much slower" and "long after": this many times the usual time |
PUTKI_ALERT_MIN_HISTORY | 5 | Successful runs needed before a run's usual time counts |
PUTKI_ALERT_MIN_SECONDS | 60 | Runs shorter than this are never slow |
PUTKI_ALERT_STUCK_HOURS | 6 | A run with no history counts as stuck after this long |
PUTKI_ALERT_WINDOW_MINUTES | 10 | How recent a failure or slow run must be |
PUTKI_ALERT_REPEAT | 4h | How often an alert that stays active is sent again |
PUTKI_ALERTS | true | false removes the rules altogether |
The rules are written when Grafana starts, so a change applies after
docker compose -f docker-compose-monitoring.yml up -d. Rules you add yourself in Grafana are
kept. The notification policy is not: with a webhook set, Grafana's default policy sends every
alert to Putki alerts, and changes made to it in Grafana are reset on the next start.
Check that it works
Alerting → Contact points lists Putki alerts with your chats. Its Test button sends a test message. Alerting → Alert rules shows the three rules in the Putki folder and their state.
If no message arrives, see Troubleshooting.
In the full Putki stack
Set the same variables in the stack's .env and run up -d: the stack's Grafana has the same
rules. With your own Grafana, the rules are not installed for you.