Skip to main content

Execution information

Apache Hop can record everything about a run in a graph database: the run, each transform and action in it, their row counts and logs, the errors, and which run started which. Putki sets this up for your projects in Neo4j, so you can answer questions such as "what did last night's load run, in what order, and where did it fail?" with a query.

PartWhat it doesWhere it runs
Hop sideA graph database connection, an execution information location that uses it, and the project's run configurations pointing at that locationInside each Hop project
Neo4jStores the graph; the Neo4j Browser shows and queries itThe execution information bundle, the Putki stack, or a Neo4j you run

The bundle (putki-graph-logging-<version>.zip) is published with every Putki release for Professional and Enterprise.

What Hop records​

Two things, switched on separately. The installer switches on both.

  • Execution information, through the execution information location in a run configuration. Every pipeline and workflow run, and every transform in a pipeline, becomes an Execution node: name, type, status, start and end, its log text, and whether it failed. A pipeline's run has ExecutionMetric nodes (HAS_METRIC) with each transform's rows read, written and in error, and its duration. EXECUTES relationships connect a run to the runs it started.
  • The execution graph, through the HOP_GRAPH_LOGGING_CONNECTION variable: Workflow, Action, Pipeline and Transform nodes, connected by ACTION_OF_WORKFLOW, TRANSFORM_OF_PIPELINE and PRECEDES, and a second kind of Execution node per run with its line counts and duration.

Both kinds of Execution node carry the same label. Those of the execution information have a filename; filter on it to see one kind.

What you see​

In the Neo4j Browser, for example the latest runs:

MATCH (e:Execution)
WHERE e.filename IS NOT NULL AND e.executionType IN ["Pipeline", "Workflow"]
RETURN e.name, e.executionType, e.statusDescription, e.failed, e.registrationDate
ORDER BY e.registrationDate DESC LIMIT 25

and what a workflow run started, as a graph:

MATCH p = (:Execution {executionType: "Workflow"})-[:EXECUTES*1..3]->(:Execution)
RETURN p LIMIT 25

More queries are in the reference.

Design choices worth knowing​

  • The project holds no password. The connection names ${NEO4J_HOSTNAME}, ${NEO4J_PASSWORD} and so on. The values are in the project's environment files, or, in the Putki stack, in the containers' environment.
  • The execution graph needs an environment. HOP_GRAPH_LOGGING_CONNECTION is a variable in the project's environment files, so runs started without an environment record execution information but not the graph of workflows and pipelines.
  • Every run is kept. Nothing deletes old runs. See Back up and restore for the data, and plan for the graph to grow with every run.

What it does not do​

  • It needs Apache Hop 2.20 or later. The installer writes Hop 2.20's graph database connection.
  • Neo4j Community has no read-only users. Everyone who can sign in can change or delete the graph. To let people look without that, use the read-only copy.
  • Dashboards of runs are Execution monitoring; the flow of data between datasets is Lineage.

Where to go next​