Skip to main content

Set up execution monitoring

In this tutorial you start Grafana and a logging database with Docker, connect one Apache Hop project to them, run a workflow, and open its log from the dashboard. Everything runs on one machine. Once it works, the how-to guides show how to fit it into your own environment.

Before you start​

You need:

  • Docker with Compose v2.20 or later (docker compose version).
  • An Apache Hop project you can run, in Putki or Apache Hop 2.x, with a Hop environment file for it. Any project with at least one workflow will do.
  • jq, used by the installer to edit the environment file.
  • Ports 3000 (Grafana) and 5432 (PostgreSQL) free on this machine. If either is taken, see Troubleshooting.

1. Download the bundle​

Download putki-monitoring-<version>.zip from the Releases page of your Putki distribution repository on forge.putki.io, next to the Putki client. Unzip it and open a terminal in the folder:

unzip putki-monitoring-<version>.zip
cd putki-monitoring-<version>

2. Set the two passwords​

cp .env.example .env

Open .env and fill in the two values at the top:

GRAFANA_ADMIN_PASSWORD=choose-an-admin-password
LOGGING_DB_PASSWORD=choose-a-database-password

Leave everything else as it is. By default the stack runs its own PostgreSQL for the logs.

3. Start the stack​

docker compose -f docker-compose-monitoring.yml up -d
docker compose -f docker-compose-monitoring.yml ps -a

After half a minute you should see grafana running, logging-db healthy, and logging-db-init exited: that one-off container created the logging tables and stopped.

Open http://localhost:3000 and log in as admin with the Grafana password from step 2. You land on the Putki monitoring dashboard. It is empty, because nothing has run yet.

4. Connect your Hop project​

Run the installer against your project. Replace the paths, pick a project name, and use the database password from step 2:

./install-logging.sh \
  --project-home /path/to/your/project \
  --project-name sales \
  --environment-file /path/to/your/project/environments/dev.json \
  --logging-host localhost \
  --logging-user hop_logging \
  --logging-password 'choose-a-database-password'

The installer copies two logging pipelines and their metadata into the project, and adds the LOGGING_* connection variables to the environment file. It prints each file it installs and each variable it adds. --project-name is what the dashboard's Project filter will show.

tip

To keep the password off your screen and out of your shell history, run the installer without --logging-password from a terminal: it asks for anything you left out. The password is still stored in the environment file, so treat that file as a secret.

5. Run a workflow​

Run any workflow in the project, using the environment you just updated. In Hop GUI, select the project and environment and run the workflow as usual. From the command line:

hop-run.sh -j sales -e dev -f '${PROJECT_HOME}/main.hwf' -r local

6. See it on the dashboard​

Back in Grafana, refresh the Putki monitoring dashboard. Total runs shows 1, and the run appears in the Success table (or Failed, if it failed). Select your project in the Project filter to see only its runs.

Click the run's name. The Execution log detail dashboard opens with the log Hop wrote for that run.

That is the whole loop: every run of every workflow and pipeline in this project is now recorded and on the dashboard.

Next steps​