Skip to main content

Assess a Pentaho project

Before you plan a migration, audit the project and import it once into a scratch folder. Neither changes anything in PDI. The audit lists what the project holds and the risks; the import's report lists, step by step, what needs a person.

1. Collect the files​

  • The transformations and jobs, as .ktr and .kjb files, in the folder layout they run from. Jobs name the transformations they run by path, so keep the layout.
  • A PDI repository is not read. Export it to files first (in Spoon, Tools → Repository → Export Repository), and check the audit's References through a PDI repository risk afterwards.
  • shared.xml, if the project keeps its connections there. Without it, steps that use a shared connection point at a connection no file defines.
  • kettle.properties of each environment, usually in ~/.kettle/. It holds the variables the files use but do not define.

Leave out what is not part of the project: copies, backups, other teams' work. Every file under the input folder counts.

2. Run the audit​

<hop>/hop-import.sh -t pdi-audit -i <pdi-project> -o /tmp/pdi-audit \
  -s <path>/shared.xml -k <path>/kettle.properties

with full paths, or use File → Audit a PDI project before a migration… in Hop GUI. The audit runs the Kettle import into a temporary folder for the expected conversion, removes it, and writes pre-migration-audit.md and audit.json.

Read it in this order:

  1. Inventory: the size of the project. Jobs no other job runs are your entry points; Lines of that code is the code in User Defined Java Classes and scripts.
  2. Expected conversion: the statuses the import will give.
  3. Risks, highest level first. Each row says what the risk means and what reduces it; the audit risks reference has them all.
  4. Not in the audit: what it cannot see, such as schedules and the JDBC drivers in PDI's lib/ folder.

3. Import into a scratch folder​

<hop>/hop-import.sh -t pdi -i <pdi-project> -o /tmp/pdi-assessment \
  -s <path>/shared.xml -k <path>/kettle.properties

Use an empty folder: by default, an import leaves files that already exist in the output folder alone.

Open /tmp/pdi-assessment/pdi-migration-report.md:

  1. Totals: how many steps and entries became a Hop plugin (AUTO). The report calls every transformation and job a "job".
  2. Most frequent source components: many ScriptValueMod (JavaScript) or UserDefinedJavaClass entries mean code to check; many TRANS and JOB entries mean a deep call tree to run end to end.
  3. Needs review: every flagged step with the reason. Group it by reason: Excel Output in forty transformations is one decision, made once.
  4. Per job: files with nothing flagged can move first, and are the quickest way to try your validation process.

4. Size the work​

For each reason in Needs review and each risk in the audit, decide who does it and how long one takes, then multiply. Add what neither sees:

  • every connection and its password, which you set in Hop by hand; see Finish connections and passwords;
  • ${Internal.Transformation.Filename.Directory} and the other internal variables, to rename after the import; see Troubleshoot;
  • the log tables, and the reports and dashboards that read them;
  • schedules in Pentaho Server, cron or Carte;
  • the JDBC drivers and plugins PDI has in its lib/ and plugins/ folders;
  • validation.

Keep the scratch output: the converted files with nothing flagged are a good first batch to validate.

Why and how to move, and how we help: Migrations to Apache Hop, and from Pentaho PDI in particular.