Expose lineage safely
Marquez has no authentication of its own: anyone who reaches its API can add lineage, read it, or delete a whole namespace. In the bundle nothing of Marquez is published. Its only way in is the proxy, which has two doors:
| Door | Port | Who | Lets in |
|---|---|---|---|
| Ingest | 5080 (LINEAGE_INGEST_PORT) | Hop | POST /api/v1/lineage with Authorization: Bearer <MARQUEZ_API_KEY>, nothing else |
| UI | 3001 (LINEAGE_UI_PORT) | People | Everything, after signing in as MARQUEZ_UI_USER with MARQUEZ_UI_PASSWORD |
Both listen on 127.0.0.1 by default.
Hop on other machines
- Put a reverse proxy that terminates TLS in front of port
5080, for examplehttps://lineage-ingest.example.com. Without TLS the API key crosses the network readable. - If that proxy runs on another machine, bind the bundle's doors to an interface it can reach
in
.env, with a firewall in between:LINEAGE_BIND_ADDR=0.0.0.0. - Give each Hop installation the HTTPS address:
./install-lineage.sh ... --url https://lineage-ingest.example.com/api/v1/lineage.
The UI for your team
Put the UI door (3001) behind TLS the same way, for example
https://lineage.example.com. Everyone signs in with the one MARQUEZ_UI_USER account. The UI
can also delete namespaces and datasets: share it with the people who look after your data,
not with everyone who can read a dashboard.
Change the key or the password
Change MARQUEZ_API_KEY or MARQUEZ_UI_PASSWORD in .env and run
docker compose -f docker-compose-lineage.yml up -d. A new API key has to reach every Hop
installation as well: re-run install-lineage.sh there.