Skip to main content

Lineage from Hop in Docker

A container usually gets its settings from its environment rather than from a file you edit once. Hop reads the lineage settings from HOP_LINEAGE_* environment variables too, and they take precedence over hop-config.json.

Print them with the installer:

./install-lineage.sh --print-env --api-key "$MARQUEZ_API_KEY" \
  --url http://lineage-proxy:5080/api/v1/lineage
HOP_LINEAGE_ENABLED=Y
HOP_LINEAGE_SINK_IDS=openlineage
HOP_LINEAGE_OPENLINEAGE_URL=http://lineage-proxy:5080/api/v1/lineage
HOP_LINEAGE_OPENLINEAGE_NAMESPACE=putki
HOP_LINEAGE_OPENLINEAGE_API_KEY=...
HOP_LINEAGE_OPENLINEAGE_FILE_SPEC_NAMING=Y

and pass them to the container: docker run -e ... for each line, an environment: block in compose, or an --env-file.

Which address​

  • On the same Docker host as the bundle: attach the container to the bundle's network (putki-lineage, or LINEAGE_NETWORK from .env) and use http://lineage-proxy:5080/api/v1/lineage.
  • Elsewhere: the address of the lineage proxy as that container reaches it, over HTTPS: see Expose lineage safely.

Short-lived containers​

A container that runs one pipeline with hop-run and exits must have the lineage delivery plugin, or it can exit before its lineage is sent. Putki's images have it from 2026.12. In an image of your own, add it from the Putki Marketplace. Set HOP_LINEAGE_OPENLINEAGE_SHUTDOWN_DRAIN_MS=5000 as well: the default lets a container wait up to 30 seconds for an unreachable Marquez before it exits.