Execution monitoring reference
Requirements
| Apache Hop | 2.x, in Putki or Apache Hop. Tested with 2.18 and 2.19 |
| PostgreSQL | 12 or later. The bundle runs 16 |
| Docker Compose | 2.20 or later, for the bundled stack |
| Grafana | 10 or later, for your own Grafana. The bundle runs 13.1 |
| jq | For install-logging.sh --environment-file |
Bundle contents
| Path | What it is |
|---|---|
docker-compose-monitoring.yml | Grafana, the optional PostgreSQL, and the one-off schema job |
docker-compose-monitoring.join.yml | Overlay that attaches to an existing Putki stack's network |
.env.example | Template for .env, the only file you edit |
install-logging.sh | Installs the Hop side into a project |
hop/ | The logging pipelines and metadata the installer copies |
sql/init-logging.sql | The schema. Idempotent: creates and migrates, never deletes |
grafana/datasources/datasources.yml | The logging datasource, configured from environment variables |
grafana/dashboards/ | The two dashboards and their provider definition |
VERSION | The bundle version, for support requests |
.env settings
| Setting | Default | Meaning |
|---|---|---|
GRAFANA_ADMIN_PASSWORD | required | Password of Grafana's admin user, applied on first start only |
LOGGING_DB_PASSWORD | required | Password of the logging database user |
COMPOSE_PROFILES | logging-db | Remove to use your own PostgreSQL instead of the bundled one |
LOGGING_DB_HOST | logging-db | Host of the logging database, as Grafana sees it |
LOGGING_DB_PORT | 5432 | Port of the logging database, as Grafana sees it |
LOGGING_DB_NAME | logging | Database name. Hop always writes to logging |
LOGGING_DB_USER | hop_logging | Database user |
LOGGING_DB_SSLMODE | disable | require or verify-full for a remote database |
LOGGING_DB_PUBLISHED_PORT | 5432 | Host port of the bundled database, for Hop outside Docker |
LOGGING_DB_BIND_ADDR | 127.0.0.1 | Interface the bundled database listens on |
GRAFANA_PORT | 3000 | Host port of Grafana |
GRAFANA_BIND_ADDR | 127.0.0.1 | Interface Grafana listens on |
GRAFANA_ROOT_URL | http://localhost:3000/ | Public URL, when behind a proxy |
GRAFANA_SERVE_FROM_SUB_PATH | false | true when the public URL has a path |
GRAFANA_COOKIE_SECURE | false | true when served over HTTPS |
GRAFANA_ANONYMOUS | false | true for read-only access without login |
GRAFANA_EXPLORE_ENABLED | false | true to allow ad-hoc SQL in Explore |
GRAFANA_THEME | light | Default theme for new users |
MONITORING_NETWORK | putki-monitoring | Name of the Docker network the stack creates |
PUTKI_NETWORK | putki-net | With the join overlay: the Putki stack's network |
COMPOSE_PROJECT_NAME | putki-monitoring | Names the containers and data volumes |
install-logging.sh options
| Option | Meaning |
|---|---|
--project-home DIR | Required. The Hop project folder, containing project-config.json |
--project-name NAME | Written to every run and listed in the dashboard's Project filter |
--environment-file FILE | Hop environment file to add the LOGGING_* variables to. Repeatable |
--logging-host HOST | LOGGING_HOSTNAME. Default warehouse-db |
--logging-port PORT | LOGGING_PORT. Default 5432 |
--logging-user USER | LOGGING_USERNAME. Default developer |
--logging-password VALUE | LOGGING_PASSWORD, plain or in Hop's Encrypted … form |
--no-prompt | Never ask; use the defaults for anything not given |
--no-clobber | Keep existing copies of the installed files |
--dry-run | Report what would change, change nothing |
The defaults are the Putki compose stack's values. Run in a terminal, the installer asks for any
LOGGING_* value you did not pass; otherwise it uses the default and says so. It never changes a
variable that is already in the environment file, and re-running it reports files as
unchanged.
It installs five files into the project:
| File | Role |
|---|---|
logging/pipeline-log.hpl | Runs at the start and end of every pipeline; records it and its transforms |
logging/workflow-log.hpl | The same for workflows and their actions |
metadata/rdbms/logging.json | The logging database connection, built from the variables below |
metadata/pipeline-log/pipeline-log.json | Tells Hop to run pipeline-log.hpl |
metadata/workflow-log/workflow-log.json | Tells Hop to run workflow-log.hpl |
Hop variables
| Variable | Meaning |
|---|---|
LOGGING_HOSTNAME | Host of the logging database, as Hop sees it |
LOGGING_PORT | Port of the logging database |
LOGGING_USERNAME | Database user; needs SELECT, INSERT and UPDATE on both tables |
LOGGING_PASSWORD | Its password. The Encrypted … form is obfuscation, not encryption |
PROJECT_NAME | Informational. The name written to each run is set in the pipelines at install time |
All four LOGGING_* variables must be defined in the environment a project runs under.
Tables
log_object: one row per workflow, pipeline, action and transform run.
| Column | Type | Meaning |
|---|---|---|
channel_id | varchar(36) | Hop's unique ID for the run. Primary key |
parent_channel_id | varchar(36) | The run that started this one; empty for a top-level run |
name | varchar(255) | Workflow, pipeline, action or transform name |
status | varchar(32) | Hop's status text, such as Finished or Stopped (with errors) |
start_date, end_date | timestamptz | When the run started and ended |
type | text | WORKFLOW, PIPELINE, ACTION or TRANSFORM |
rows_processed | integer | Rows written by the transform, or by the pipeline or workflow as a whole |
time_processed | integer | Duration in seconds |
svg | varchar | Rendered image of the workflow or pipeline |
project | varchar(255) | The --project-name it was installed with |
log_detail: the log text of a run, for rows that produced any. channel_id references
log_object and is deleted with it.
Dashboards
Putki monitoring (putki-monitoring). Filters: Project and the time range.
| Panel | Shows |
|---|---|
| Total runs, Success, Failed, Success % | Top-level workflows started in the time range: those not started by another workflow. Failed counts every one whose status is not Finished, including runs still in progress |
| Volume | Sum of rows_processed per day over every recorded row: workflows, pipelines, actions and transforms. A row that passes through several transforms is counted once for each |
| Time processed | Seconds of run time per day |
| Failed, Success | Top-level workflow and pipeline runs with status Stopped (with errors) or Finished. Click a name to open its log |
Execution log detail (putki-log-detail). The log of one run, opened from either table.
Both are provisioned: Grafana restores the shipped version when it restarts. Save a copy under a new name to customise one.