Skip to main content

Set up lineage

In this tutorial you start Marquez with Docker, point one Apache Hop installation at it, run a pipeline, and follow the data it wrote in the lineage graph. Everything runs on one machine.

Before you start​

You need:

  • Docker with Compose v2.20 or later (docker compose version).
  • Putki 2026.12 or later, Professional or Enterprise: its Hop has the OpenLineage sink and Putki's lineage delivery plugin. With Apache Hop 2.20 or later instead, install Lineage delivery from the Putki Marketplace first (see Install Putki plugins); without it, runs started with hop-run can lose their lineage when Hop exits.
  • jq, used by the installer to edit Hop's configuration.
  • Ports 5080 (lineage from Hop) and 3001 (the lineage UI) free on this machine.

1. Download the bundle​

Download putki-lineage-<version>.zip from the Releases page of your Putki distribution repository on forge.putki.io. Unzip it and open a terminal in the folder:

unzip putki-lineage-<version>.zip
cd putki-lineage-<version>

2. Set the passwords and the API key​

./init-env.sh

It creates .env and asks for MARQUEZ_DB_PASSWORD, MARQUEZ_API_KEY and MARQUEZ_UI_PASSWORD. Type one, or press Enter to have one made up; for the API key, Enter is the better choice. Hop sends its lineage with the API key; you sign in to the lineage UI with MARQUEZ_UI_PASSWORD.

.env is where they are kept, readable only by you. To look one up: grep MARQUEZ_API_KEY .env.

Installing from a script

./init-env.sh --generate asks nothing and makes up every password. Give the ones you choose yourself as NAME=value arguments, or as environment variables to keep them out of the process list: MARQUEZ_UI_PASSWORD="$FROM_YOUR_VAULT" ./init-env.sh --generate. It never changes a value that is already in .env, so running it again is safe.

3. Start Marquez​

docker compose -f docker-compose-lineage.yml up -d
docker compose -f docker-compose-lineage.yml ps

After about a minute marquez is healthy and lineage-proxy is running. Open http://localhost:3001 and sign in as admin with MARQUEZ_UI_PASSWORD. Marquez is empty: nothing has run yet.

4. Point Hop at it​

Close Hop GUI if it is open: it rewrites its configuration when it exits. Then, with the folder Hop is installed in:

./install-lineage.sh --hop-home /path/to/hop --api-key 'the MARQUEZ_API_KEY from .env'

The installer writes six HOP_LINEAGE_* settings into Hop's config/hop-config.json and lists them. It keeps a copy of the file as it was, hop-config.json.before-lineage.

tip

Hop that keeps its configuration elsewhere (HOP_CONFIG_FOLDER, or ~/.hop/config): pass that file with --hop-config instead of --hop-home. Hop in a container: see Lineage from Hop in Docker.

5. Run a pipeline​

Start Hop GUI and run any pipeline that reads or writes a file or a database table, or run one from the command line:

/path/to/hop/hop-run.sh -j <project> -r local -f '${PROJECT_HOME}/my-pipeline.hpl'

6. Follow the data​

Back in the lineage UI, choose the putki namespace. The pipeline is listed under Jobs, its last run COMPLETED (or FAILED). Open it to see its transforms and, on the graph, the datasets it read and wrote. Click a dataset to see every job that touches it.

That is the whole loop: every run on this Hop installation now reports its lineage.

Next steps​