Impact analysis
Before you change a table, a connection or a pipeline, you want to know what depends on it. Putki impact analysis reads your Apache Hop projects, pipelines, workflows and metadata, into a graph, and a Grafana dashboard answers the questions from it:
- What reads and writes this table, in which pipeline, and is it found in an SQL statement?
- Which tables do those pipelines write, so what can a change here reach?
- Which workflows run those pipelines?
- Which calls point at a file that is not there, and which connections does no metadata define?
It reads the project files only. Nothing has to run, and it works the same for a project that never ran.
| Part | What it does |
|---|---|
| Sync job | Putki Tools reads each project folder into the graph, and again every five minutes. Only files that changed are read again |
| Graph database | Apache AGE: PostgreSQL with a graph extension. One graph per project |
| Grafana | The impact analysis dashboard, with a read-only login to the graph database |
All three ship in the impact analysis bundle (putki-impact-<version>.zip), for Professional
and Enterprise. The Putki stack of those editions runs them too.
What is in the graph
Pipelines, workflows, their transforms and actions, and what they refer to: database connections, tables (from the schema and table fields, and from SQL), files, other metadata, and variables. Calls between pipelines and workflows are links, with the file names resolved the way Hop resolves them. The reference lists every kind of node and link.
Design choices worth knowing
- A Grafana of its own. The dashboard shows the inside of every project: table names, SQL, file paths. Who may see that is often not who may see run statistics, so it does not share the monitoring Grafana unless you choose to: see Show it in the monitoring Grafana.
- Read-only, also through Grafana. Grafana's login to the graph database can read the graphs and nothing else, whatever SQL a viewer sends.
- A variable stays a variable. A table or file name built from
${VARIABLES}is shown as written: the graph describes the files, not one environment.
What it does not do
- It does not see what ran. For the data that actually moved, see Lineage. For the runs themselves, see Execution monitoring and Execution information.
- It follows one step at a time. "Downstream" lists what the readers of a table write; a change further along the chain needs the next table chosen in turn.
Where to go next
- New to it? Set up impact analysis.
- Fitting it in? The how-to guides: your own PostgreSQL, the monitoring Grafana, the Putki stack, the command line, upgrading and troubleshooting.
- Looking something up? The reference.