Execution monitoring
Putki execution monitoring gives you one place to see every workflow and pipeline run across your Apache Hop projects: how many ran, which failed, how many rows and how much time they took, and the log of any single run. It is built from two parts, and you need both.
| Part | What it does | Where it runs |
|---|---|---|
| Hop side | Two logging pipelines that Hop runs after every workflow and pipeline, writing one row per run to a PostgreSQL database | Inside your Hop project, wherever Hop runs |
| Grafana side | A datasource and two dashboards that read those rows | The Grafana in the monitoring bundle, or a Grafana you already operate |
Both parts ship in one download, the monitoring bundle (putki-monitoring-<version>.zip),
published with every Putki release.
How it works
- Hop starts or finishes a workflow or pipeline.
- The project's Pipeline Log and Workflow Log metadata tell Hop to run the logging
pipelines, which write a row to
log_objectfor the run and for each of its actions and transforms (name, status, start and end time, rows processed, project), and the log text tolog_detail. - Grafana queries
log_objectfor the counters, trends and tables, andlog_detailwhen you open a single run.
Nothing reads Hop's own files or logs, and nothing needs to run next to Hop: the only connection between the two parts is the database.
What you see
- Putki monitoring dashboard: total runs, successes, failures and success rate; rows and seconds processed per day; tables of failed and successful runs. Filter by project and time range.
- Execution log detail dashboard: the captured log of one run, opened by clicking a run in either table.
Design choices worth knowing
- One database, many projects. Each project is installed with its own name, written to every row. That name is what the Project filter lists and the only thing that separates projects sharing a database.
- Times are stored as absolute instants. A Hop server in Brussels and one in UTC log to the same database without their runs drifting apart, and the dashboards show them in your browser's time zone.
- A run is recorded when it starts, and updated when it ends. A run that is still going is already visible, and one whose Hop process died stays visible with the status it last had instead of disappearing.
- Log text is stored for every run that produces it. That makes any run inspectable after the
fact, and it makes
log_detailthe fastest-growing table. See Manage log volume.
What it does not do
Execution monitoring records and shows runs. It does not send alerts, enforce SLAs, or show host resource usage such as CPU or memory. To notify a team when a workflow fails, add the Putki Chat action to the workflow's failure path: it posts to a Slack channel.
Where to go next
- New to it? Set up execution monitoring takes you from the download to your first run on a dashboard in about fifteen minutes.
- Fitting it into your environment? The how-to guides cover your own PostgreSQL, your own Grafana, an existing Putki stack, exposing Grafana safely, upgrading and troubleshooting.
- Looking something up? The reference lists every setting, variable, table and panel.