Execution monitoring reference

Applies toPutki 2026.09 – 2026.122026.09: the updated 2026.09 monitoring bundle, the one with a bin folder.

Requirements

Apache Hop	2.x, in Putki or Apache Hop. Tested with 2.18 and 2.19
PostgreSQL	12 or later. The bundle runs 16
Docker Compose	2.20 or later, for the bundled stack
Grafana	10 or later, for your own Grafana. The bundle runs 13.1
jq	For install-logging.sh --environment-file

Bundle contents

Path	What it is
docker-compose-monitoring.yml	Grafana, the optional PostgreSQL, and the one-off schema job
docker-compose-monitoring.join.yml	Overlay that attaches to an existing Putki stack's network
docker-compose-monitoring.logs.yml	Overlay that forwards container logs to Loki, OpenSearch or Splunk. 2026.12 and later
alloy/	Its Grafana Alloy configuration: docker.alloy, and one file per platform
.env.example	Template for .env, the only file you edit
init-env.sh	Writes .env from .env.example and fills in the required settings: asks, or --generate makes up the passwords. Never changes a value already set
install-logging.sh	Installs the Hop side into a project
hop/	The logging pipelines and metadata the installer copies
sql/init-logging.sql	The schema. Idempotent: creates and migrates, never deletes
grafana/datasources/datasources.yml	The logging datasource, configured from environment variables
grafana/dashboards/	The two dashboards and their provider definition
grafana/alerting/, grafana/plugins/	Empty. The alert rules are written at start, into a copy
bin/putki-entrypoint.sh	Starts Grafana: writes the alert rules and contact points, sets up single sign-on
bin/putki-alerting.sh, bin/putki-sso.sh	Used by the entrypoint; the same files the Putki stack's Grafana image runs
VERSION	The bundle version, for support requests

.env settings

Setting	Default	Meaning
GRAFANA_ADMIN_PASSWORD	required	Password of Grafana's admin user, applied on first start only
LOGGING_DB_PASSWORD	required	Password of the logging database user, which Hop writes with
GRAFANA_DB_PASSWORD	required	Password of Grafana's read-only database user
GRAFANA_DB_USER	grafana_reader	Grafana's read-only database user, created by logging-db-init
COMPOSE_PROFILES	logging-db	Remove to use your own PostgreSQL instead of the bundled one
LOGGING_DB_HOST	logging-db	Host of the logging database, as Grafana sees it
LOGGING_DB_PORT	5432	Port of the logging database, as Grafana sees it
LOGGING_DB_NAME	logging	Database name. Hop always writes to logging
LOGGING_DB_USER	hop_logging	Database user Hop writes with; also creates the tables and Grafana's user
LOGGING_DB_SSLMODE	disable	require or verify-full for a remote database
LOGGING_DB_PUBLISHED_PORT	5432	Host port of the bundled database, for Hop outside Docker
LOGGING_DB_BIND_ADDR	127.0.0.1	Interface the bundled database listens on
GRAFANA_PORT	3000	Host port of Grafana
GRAFANA_BIND_ADDR	127.0.0.1	Interface Grafana listens on
GRAFANA_ROOT_URL	http://localhost:3000/	Public URL, when behind a proxy
GRAFANA_SERVE_FROM_SUB_PATH	false	true when the public URL has a path
GRAFANA_COOKIE_SECURE	false	true when served over HTTPS
GRAFANA_ANONYMOUS	false	true for read-only access without login
GRAFANA_EXPLORE_ENABLED	false	true to allow ad-hoc SQL in Explore
GRAFANA_THEME	light	Default theme for new users
MONITORING_NETWORK	putki-monitoring	Name of the Docker network the stack creates
PUTKI_NETWORK	putki-net	With the join overlay: the Putki stack's network
COMPOSE_PROJECT_NAME	putki-monitoring	Names the containers and data volumes

Alerts

See Get alerts in Slack or Google Chat.

Setting	Default	Meaning
PUTKI_ALERT_SLACK_WEBHOOK_URL		Slack incoming webhook to send alerts to
PUTKI_ALERT_GOOGLE_CHAT_WEBHOOK_URL		Google Chat space webhook to send alerts to
PUTKI_ALERT_SLOW_FACTOR	3	A run is much slower, or stuck, at this many times its usual time
PUTKI_ALERT_MIN_HISTORY	5	Successful runs of a name needed before its usual time counts
PUTKI_ALERT_MIN_SECONDS	60	Runs shorter than this are never slow
PUTKI_ALERT_STUCK_HOURS	6	A run without history is stuck after this many hours
PUTKI_ALERT_VOLUME_FACTOR	10	A run wrote an unusual number of rows at this many times more, or fewer, than usual. 2026.12 and later
PUTKI_ALERT_MIN_ROWS	1000	Below this many rows, for the run and its usual, the volume is never unusual. 2026.12 and later
PUTKI_ALERT_WINDOW_MINUTES	10	How recent the end of a run must be to alert
PUTKI_ALERT_REPEAT	4h	How often an alert that stays active is sent again
PUTKI_ALERTS	true	false installs no alert rules

Log forwarding

2026.12 and later, with docker-compose-monitoring.logs.yml. See Forward logs to Loki, OpenSearch or Splunk.

Setting	Default	Meaning
LOG_FORWARD_SINK	required	loki, opensearch or splunk: which file in alloy/ Alloy loads
LOG_FORWARD_URL	required	Loki push URL; for OpenSearch, host:port of the Data Prepper OTLP source; for Splunk, the HEC URL
LOG_FORWARD_USER		Basic-auth user, Loki and OpenSearch. Empty for none
LOG_FORWARD_PASSWORD		Its password or token; for Splunk, the HEC token
LOG_FORWARD_TENANT		Loki: X-Scope-OrgID, for a multi-tenant Loki
LOG_FORWARD_INDEX		Splunk: the index. Empty for the token's default
LOG_FORWARD_CONTAINERS	all	Regular expression on container names; only matching containers are forwarded
LOG_FORWARD_HOST	Alloy's container id	The host label on every line

Single sign-on

See Sign in with Google, Microsoft or your SSO.

Setting	Meaning
GRAFANA_GOOGLE_CLIENT_ID, _SECRET	Google OAuth client. Callback <GRAFANA_ROOT_URL>login/google
GRAFANA_ENTRA_TENANT_ID, GRAFANA_ENTRA_CLIENT_ID, _SECRET	Microsoft Entra ID app registration. Callback <GRAFANA_ROOT_URL>login/azuread
GRAFANA_OIDC_ISSUER, GRAFANA_OIDC_CLIENT_ID, _SECRET	Any OpenID Connect provider. Callback <GRAFANA_ROOT_URL>login/generic_oauth
GRAFANA_OIDC_NAME	Text on the sign-in button. Default Single sign-on
GRAFANA_OIDC_AUTH_URL, _TOKEN_URL, _API_URL	Endpoints, for providers whose paths differ from Keycloak's
GRAFANA_OIDC_INTERNAL_URL	The issuer as Grafana's server reaches it, when that differs from the browser's address
GRAFANA_SSO_ALLOWED_DOMAINS	Comma-separated e-mail domains allowed to sign in. Required with Google
GRAFANA_SSO_ALLOW_ANY_ACCOUNT	true to use Google without allowed domains
GRAFANA_SSO_ADMINS, GRAFANA_SSO_EDITORS	Comma-separated e-mail addresses that get the Admin or Editor role
GRAFANA_SSO_DEFAULT_ROLE	Role of everyone else: Viewer (default) or None
GRAFANA_SSO_ROLE_ATTRIBUTE_PATH	A JMESPath over the user's claims, instead of the lists
GRAFANA_SSO_ONLY	true hides the username and password form

install-logging.sh options

Option	Meaning
--project-home DIR	Required. The Hop project folder, containing project-config.json
--project-name NAME	Written to every run and listed in the dashboard's Project filter
--environment-file FILE	Hop environment file to add the LOGGING_* variables to. Repeatable
--logging-host HOST	LOGGING_HOSTNAME. Default warehouse-db
--logging-port PORT	LOGGING_PORT. Default 5432
--logging-user USER	LOGGING_USERNAME. Default developer
--logging-password VALUE	LOGGING_PASSWORD, plain or in Hop's Encrypted … form
--no-prompt	Never ask; use the defaults for anything not given
--no-clobber	Keep existing copies of the installed files
--dry-run	Report what would change, change nothing

The defaults are the Putki compose stack's values. Run in a terminal, the installer asks for any LOGGING_* value you did not pass; otherwise it uses the default and says so. It never changes a variable that is already in the environment file, and re-running it reports files as unchanged.

It installs five files into the project:

File	Role
logging/pipeline-log.hpl	Runs at the start and end of every pipeline; records it and its transforms
logging/workflow-log.hpl	The same for workflows and their actions
metadata/rdbms/logging.json	The logging database connection, built from the variables below
metadata/pipeline-log/pipeline-log.json	Tells Hop to run pipeline-log.hpl
metadata/workflow-log/workflow-log.json	Tells Hop to run workflow-log.hpl

Hop variables

Variable	Meaning
LOGGING_HOSTNAME	Host of the logging database, as Hop sees it
LOGGING_PORT	Port of the logging database
LOGGING_USERNAME	Database user; needs SELECT, INSERT and UPDATE on both tables
LOGGING_PASSWORD	Its password. The Encrypted … form is obfuscation, not encryption
PROJECT_NAME	Informational. The name written to each run is set in the pipelines at install time

All four LOGGING_* variables must be defined in the environment a project runs under. In the Putki stack (2026.12) the Hop containers get them from the stack's .env, so a project there needs none of them in its environment files.

Tables

log_object: one row per workflow, pipeline, action and transform run.

Column	Type	Meaning
channel_id	varchar(36)	Hop's unique ID for the run. Primary key
parent_channel_id	varchar(36)	The run that started this one; empty for a top-level run
name	varchar(255)	Workflow, pipeline, action or transform name
status	varchar(32)	Hop's status text, such as Finished or Stopped (with errors)
start_date, end_date	timestamptz	When the run started and ended
type	text	WORKFLOW, PIPELINE, ACTION or TRANSFORM
rows_processed	integer	For a transform: the rows it wrote. Empty for pipelines, workflows and actions
time_processed	integer	Duration in seconds
svg	varchar	Rendered image of the workflow or pipeline
project	varchar(255)	The --project-name it was installed with

log_detail: the log text of a run, for rows that produced any. channel_id references log_object and is deleted with it.

Dashboards

A run is a pipeline or workflow execution that no other workflow started: parent_channel_id is empty and type is PIPELINE or WORKFLOW. Its outcome comes from Hop's status:

Outcome	Status
Succeeded	Finished
Failed	Finished (with errors), Stopped (with errors)
Stopped	Stopped
Running	anything else, such as Running or Waiting: not ended, or its end was never logged

Putki monitoring (putki-monitoring). Filters: Project, Runs (top-level runs, or every execution including the pipelines and workflows a run started), Name, Outcome, Log search and the time range.

Panel	Shows
Runs, Succeeded, Failed, Running	Runs started in the time range, by outcome
Success %	Succeeded out of the runs that are no longer running
Rows written per day	Per pipeline execution, the most rows any one of its transforms wrote, summed per day. Counts each row once, not once per transform. Follows Project only
Run time per day	Total duration of the runs that finished, by the day they started
Failed runs, All runs	Runs with their outcome, status, duration and rows written, newest first, at most 1000. Click a name to open its log
Log search	Executions whose log contains the Log search text, ignoring case, with the text around it. Empty while the search is empty

Execution log detail (putki-log-detail). One run: its outcome and duration, the executions it started, and its log.

Both are provisioned: Grafana restores the shipped version when it restarts. Save a copy under a new name to customise one.

Alert rules

In the Putki folder, group Putki runs, evaluated every minute. Each alert carries the labels project and name and links to the dashboard.

Rule	Fires for a run that
Run failed	Ended Failed in the last PUTKI_ALERT_WINDOW_MINUTES
Run much slower than usual	Ended in that window, ran at least PUTKI_ALERT_MIN_SECONDS, and took more than PUTKI_ALERT_SLOW_FACTOR times the median of its previous 10 successful runs (with at least PUTKI_ALERT_MIN_HISTORY of them)
Run still running long after its usual time	Started in the last 7 days, has not ended, and has been running more than PUTKI_ALERT_SLOW_FACTOR times its usual time, or PUTKI_ALERT_STUCK_HOURS without history
Run wrote no rows	Ended successfully in that window having written 0 rows, while the median of its previous 10 successful runs (at least PUTKI_ALERT_MIN_HISTORY) is above 0. 2026.12 and later
Run wrote an unusual number of rows	Ended successfully in that window having written more than PUTKI_ALERT_VOLUME_FACTOR times that median, or less than that median divided by it; not when both are below PUTKI_ALERT_MIN_ROWS. 2026.12 and later
Login refused during a run	Ended in that window, with or without errors, and the log text of the run or of anything it ran contains a refused-login message (list in bin/putki-alerting.sh, AUTH_MESSAGES). Indicative: matched on text. 2026.12 and later

Rows written by a pipeline are the largest rows_processed of its transforms (Hop's lines written), as on Rows written per day. A workflow's are the sum over the pipelines below it, followed through parent_channel_id across actions and child workflows. A run whose pipelines recorded no transforms has no row count, and the two row rules leave it out.