Lineage
Putki lineage answers "where does this data come from, and what uses it?" across all your Apache Hop projects. Every pipeline and workflow run reports what it read and what it wrote, and Marquez, the reference server for the open OpenLineage standard, puts the reports together into one graph you can browse.
| Part | What it does | Where it runs |
|---|---|---|
| Hop side | Apache Hop's OpenLineage sink, switched on and pointed at Marquez. Putki's lineage delivery plugin makes sure a run's report is sent before Hop exits | In every Hop installation that runs your projects |
| Marquez | Stores the reports and shows the lineage graph, jobs, datasets and runs | The lineage bundle, or the Putki stack |
The lineage bundle (putki-lineage-<version>.zip) is published with every Putki release for
Professional and Enterprise.
How it works
- A pipeline or workflow starts, runs and ends. Hop's lineage hub collects what happens: the run itself, each transform and action, the files and tables read and written, and for transforms that read SQL, the tables in it.
- The OpenLineage sink turns that into OpenLineage events and sends them to Marquez.
- Marquez links the events by dataset: a table one pipeline writes and another reads becomes one node between them, whichever project each pipeline is in.
What you see
In the Marquez UI, per namespace (putki unless you choose another):
- Jobs: every pipeline and workflow, and their transforms and actions, with their runs and how each run ended.
- Datasets: the files and database tables your runs read and wrote, with their columns where Hop knows them.
- The lineage graph of a job or dataset: what feeds it and what it feeds.
Design choices worth knowing
- Lineage is a setting of the Hop installation, not of a project. Switch it on once per Hop, Hop Server or container; every project run there reports.
- Files are named the same way everywhere. A file read at run time and the same file in a design-time export land on one node.
- Marquez has no login of its own. The bundle puts a proxy in front of it: Hop sends with an API key, people sign in with a password. See Expose lineage safely.
What it does not do
- It needs Apache Hop 2.20 or later. Earlier versions have no lineage sink.
- It records what ran, not what could run. A pipeline appears after its first run.
- Execution logs and dashboards are Execution monitoring; this is the graph of data, not the history of runs.
Where to go next
- New to it? Set up lineage takes you from the download to your first run in the lineage graph.
- Fitting it into your environment? The how-to guides cover Hop in Docker, exposing Marquez safely, the Putki stack, upgrading and troubleshooting.
- Looking something up? The reference lists every setting and variable.