Skip to main content

Set up execution monitoring

In this tutorial you start Grafana and a logging database with Docker, connect one Apache Hop project to them, run a workflow, and open its log from the dashboard. Everything runs on one machine. Once it works, the how-to guides show how to fit it into your own environment.

Before you start​

You need:

  • Docker with Compose v2.20 or later (docker compose version).
  • An Apache Hop project you can run, in Putki or Apache Hop 2.x, with a Hop environment file for it. Any project with at least one workflow will do.
  • jq, used by the installer to edit the environment file.
  • Ports 3000 (Grafana) and 5432 (PostgreSQL) free on this machine. If either is taken, see Troubleshooting.

1. Download the bundle​

Download putki-monitoring-<version>.zip from the Releases page of your Putki distribution repository on forge.putki.io, next to the Putki client. Unzip it and open a terminal in the folder:

unzip putki-monitoring-<version>.zip
cd putki-monitoring-<version>

2. Set the passwords​

./init-env.sh

It creates .env and asks for three passwords. Type one, or press Enter to have one made up:

  • GRAFANA_ADMIN_PASSWORD: Grafana's admin.
  • LOGGING_DB_PASSWORD: the database user Hop writes with.
  • GRAFANA_DB_PASSWORD: a separate, read-only user that Grafana reads with. The stack creates it on start.

.env is where they are kept, readable only by you. To look one up: grep GRAFANA_ADMIN_PASSWORD .env.

The 2026.09 bundle has no init-env.sh: run cp .env.example .env and fill in the three values at the top of .env by hand.

Installing from a script

./init-env.sh --generate asks nothing and makes up every password. Give the ones you choose yourself as NAME=value arguments, or as environment variables to keep them out of the process list: GRAFANA_ADMIN_PASSWORD="$FROM_YOUR_VAULT" ./init-env.sh --generate. It never changes a value that is already in .env, so running it again is safe.

Leave everything else as it is. By default the stack runs its own PostgreSQL for the logs.

3. Start the stack​

docker compose -f docker-compose-monitoring.yml up -d
docker compose -f docker-compose-monitoring.yml ps -a

After half a minute you should see grafana running, logging-db healthy, and logging-db-init exited: that one-off container created the logging tables and stopped.

Open http://localhost:3000 and log in as admin with the Grafana password from step 2. You land on the Putki monitoring dashboard. It is empty, because nothing has run yet.

4. Connect your Hop project​

Run the installer against your project. Replace the paths, pick a project name, and use LOGGING_DB_PASSWORD from .env:

./install-logging.sh \
  --project-home /path/to/your/project \
  --project-name sales \
  --environment-file /path/to/your/project/environments/dev.json \
  --logging-host localhost \
  --logging-user hop_logging \
  --logging-password 'the LOGGING_DB_PASSWORD from .env'

The installer copies two logging pipelines and their metadata into the project, and adds the LOGGING_* connection variables to the environment file. It prints each file it installs and each variable it adds. --project-name is what the dashboard's Project filter will show.

tip

To keep the password off your screen and out of your shell history, run the installer without --logging-password from a terminal: it asks for anything you left out. The password is still stored in the environment file, so treat that file as a secret.

5. Run a workflow​

Run any workflow in the project, using the environment you just updated. In Hop GUI, select the project and environment and run the workflow as usual. From the command line:

hop-run.sh -j sales -e dev -f '${PROJECT_HOME}/main.hwf' -r local

6. See it on the dashboard​

Back in Grafana, refresh the Putki monitoring dashboard. Runs shows 1, and the run appears in the All runs table, and in Failed runs if it failed. Select your project in the Project filter to see only its runs.

Click the run's name. The Execution log detail dashboard opens with the run, the pipelines and workflows it started, and the log Hop wrote for it.

That is the whole loop: every run of every workflow and pipeline in this project is now recorded and on the dashboard.

Next steps​