Skip to main content

Set up impact analysis

In this tutorial you start the graph database, the sync job and Grafana with Docker, point them at your projects, and look up what reads and writes one of your tables.

Before you start​

You need:

  • Docker with Compose v2.20 or later.
  • Access to forge.putki.io with your Putki token: the sync job runs the Putki Tools image. Run docker login forge.putki.io once.
  • Your Hop projects in one folder on this machine, each a folder with a project-config.json. A single project folder works too.
  • Port 3002 free.

1. Download the bundle​

Download putki-impact-<version>.zip from the Releases page of your Putki distribution repository on forge.putki.io, unzip it, and open a terminal in the folder.

2. Configure​

./init-env.sh

It creates .env and asks for four values:

  • PROJECTS_DIR: the folder holding your Hop projects, or one project. The projects are read, never written.
  • GRAFANA_ADMIN_PASSWORD: Grafana's admin.
  • GRAPH_DB_PASSWORD: the graph database, as the sync job writes it.
  • GRAPH_DB_READER_PASSWORD: Grafana's read-only login to it.

For a password, type one or press Enter to have one made up. .env is where they are kept, readable only by you. To look one up: grep GRAFANA_ADMIN_PASSWORD .env.

Installing from a script

./init-env.sh --generate asks nothing and makes up every password. Give PROJECTS_DIR, and any password you choose yourself, as NAME=value arguments, or as environment variables to keep them out of the process list: PROJECTS_DIR=/srv/hop/projects ./init-env.sh --generate. It never changes a value that is already in .env, so running it again is safe.

3. Start it​

docker compose -f docker-compose-impact.yml up -d
docker compose -f docker-compose-impact.yml logs project-graph-sync

Within a minute the sync job prints a line per project:

synced sales: graph:project_local_sales,changed:48,removed:0,files:48,...

4. Look something up​

Open http://localhost:3002 and log in as admin with GRAFANA_ADMIN_PASSWORD. On the Putki impact analysis dashboard:

  1. Choose a project under Project. The top row counts its pipelines, workflows, tables and connections, and its broken references.
  2. Choose a table under Table. What reads and writes lists every transform and action that touches it, and the file it is in; Downstream the tables written by the pipelines that read it; Workflows that run them what starts those pipelines.

Change a pipeline and save it: within five minutes the dashboard shows the change.

Next steps​