Lineage reference
Requirements
| Apache Hop | 2.20 or later: Putki 2026.12 Professional or Enterprise, or Apache Hop with the Lineage delivery plugin |
| Docker Compose | 2.20 or later |
| jq | For install-lineage.sh --hop-home and --hop-config |
Bundle contents
| Path | What it is |
|---|---|
docker-compose-lineage.yml | Marquez, its database, the lineage UI, and the proxy in front of them |
.env.example | Template for .env, the only file you edit |
init-env.sh | Writes .env from .env.example and fills in the required settings: asks, or --generate makes up the passwords. Never changes a value already set |
install-lineage.sh | Points a Hop installation at the bundle |
caddy/Caddyfile | The proxy's two doors |
config/marquez.yml | Marquez's configuration, read from the environment |
VERSION | The bundle version, for support requests |
.env settings
| Setting | Default | Meaning |
|---|---|---|
MARQUEZ_DB_PASSWORD | required | Password of Marquez's database |
MARQUEZ_API_KEY | required | The key Hop sends lineage with |
MARQUEZ_UI_PASSWORD | required | Password to sign in to the lineage UI |
MARQUEZ_UI_USER | admin | User to sign in to the lineage UI |
LINEAGE_INGEST_PORT | 5080 | Host port Hop sends lineage to |
LINEAGE_UI_PORT | 3001 | Host port of the lineage UI |
LINEAGE_BIND_ADDR | 127.0.0.1 | Interface both ports listen on |
LINEAGE_NETWORK | putki-lineage | Docker network; a Hop container on it reaches lineage-proxy:5080 |
install-lineage.sh options
| Option | Meaning |
|---|---|
--hop-home DIR | The Hop installation; edits DIR/config/hop-config.json and checks the OpenLineage plugin is there |
--hop-config FILE | A hop-config.json elsewhere |
--print-env | Print the settings as NAME=value lines instead of editing a file |
--api-key KEY | Required: MARQUEZ_API_KEY |
--url URL | Where to send lineage. Default http://localhost:5080/api/v1/lineage |
--namespace NAME | OpenLineage namespace. Default putki |
--disable | Switch lineage off, keeping the other settings |
--dry-run | Show what would change, change nothing |
Hop settings
Set in hop-config.json, as environment variables (which win over the file), or as -D
options (which win over both).
| Setting | Set to | Meaning |
|---|---|---|
HOP_LINEAGE_ENABLED | Y | Switches the lineage hub on |
HOP_LINEAGE_SINK_IDS | openlineage | Which sink receives the events |
HOP_LINEAGE_OPENLINEAGE_URL | the ingest URL | Where the OpenLineage events go |
HOP_LINEAGE_OPENLINEAGE_NAMESPACE | putki | Namespace of the jobs in Marquez |
HOP_LINEAGE_OPENLINEAGE_API_KEY | MARQUEZ_API_KEY | Sent as Authorization: Bearer <key> |
HOP_LINEAGE_OPENLINEAGE_FILE_SPEC_NAMING | Y | Names file datasets by their full path and scheme. Choose before Marquez has data: changing it later gives every file a second node |
HOP_LINEAGE_OPENLINEAGE_SHUTDOWN_DRAIN_MS | 5000 in containers | How long Hop waits at exit to send what is left. Default 30000 |
PUTKI_LINEAGE_SETTLE_MS | 500 | Lineage delivery plugin: how long hop-run waits for a run's last events before it exits |