Pentaho (PDI) importer

Coming soon

The PDI importer is about to be released as open source (Apache License 2.0). It needs Apache Hop 2.20.0, which is not released yet. Until then, we run it for you: contact us.

The PDI importer converts Pentaho Data Integration (Kettle) transformations (.ktr) and jobs (.kjb) into Apache Hop pipelines and workflows, and writes a migration report that classifies every step and job entry. A pre-migration audit, run before any conversion, lists what the project holds and the risks it carries.

It differs from the other importers in one way that matters: the conversion is Apache Hop's own. Every Apache Hop release ships a Kettle import (File → Import from Kettle/PDI, hop-import -t kettle). The PDI importer runs that import unchanged and adds what it does not have: the report, the audit, and the hooks for the reporting add-on. Improvements to the conversion go to Apache Hop, so they reach everyone who imports PDI files, with or without this importer.

Part	What it does	Where it runs
Hop plugin	Adds File → Import from PDI (with migration report)… and File → Audit a PDI project before a migration… to Hop GUI, and `-t pdi` and `-t pdi-audit` to `hop-import`	Inside Apache Hop 2.20.0 or later
Migration report	`pdi-migration-report.md` in the output folder	Written by every import
Pre-migration audit	`pre-migration-audit.md` and `audit.json`; writes no Hop files	Written by every audit

There is no standalone jar. The Kettle import uses the database and transform plugins of the Hop it runs in, so the importer runs inside a Hop installation.

How it works

The importer lists the .ktr and .kjb files under the input folder.
It looks for database connections whose type the Hop it runs in has no database plugin for. Until Apache Hop 2.21.0, one such connection stops the whole Kettle import; when there are any, the importer runs the import on a temporary copy of the input in which only those connections have the GENERIC type. The report lists them.
Apache Hop's Kettle import converts the files, with every option you gave it: transformations become pipelines, jobs become workflows, connections go to the metadata folder, variables to an environment file.
The classifier reads each PDI file next to the Hop file the import wrote from it, matches steps and job entries by name, and gives each one a status from the Hop plugin it became.
The report is written from that classification, followed by the Kettle import's own summary.

What you get

For an input folder with transformations/Load orders.ktr and jobs/Nightly.kjb, in the output folder:

transformations/Load orders.hpl and jobs/Nightly.hwf, in the same folder layout;
the other files of the input folder (data files, scripts), copied as they are;
metadata/rdbms/: one Hop connection per connection name;
metadata/pipeline-run-configuration/local.json and metadata/workflow-run-configuration/local.json, when the target has no local run configuration;
connections.csv: every connection in every file, with the name Hop gave it;
migration/conversion-metadata.json: the classification, for tools and the reporting add-on;
pdi-migration-report.md: see The migration report.

What the report classifies

Status	When
AUTO	The step or entry became an Apache Hop plugin. Some carry a note to review: a Formula, which runs as Hop's OpenFormula; a step the import renamed
PARTIAL	A Hop plugin exists, with a known difference: Data Validator
MANUAL	Code that still calls the Pentaho API (User Defined Java Class, JavaScript, Java Filter, Janino), or a step to replace or install as a plugin: Excel Output, LDIF Input, Mail Validator, Syslog, MQTT, Fixed File Input, Edi to XML, the message box entry
UNSUPPORTED	No Hop plugin has the type's id (Pentaho Reporting Output, Single Threader, HL7 Input, XBase Input, Automatic Documentation Output, enterprise steps), or the step is missing from the converted file

Which types are Hop plugins comes from a catalog of the Hop version the importer was built against, plus the plugins installed in the Hop that runs the import.

What the Kettle import leaves out

The audit finds these before you convert; the import itself does not say:

Log tables. Transformation, step and job log tables are not carried over. Log with Hop's Pipeline Log and Workflow Log, or with execution information.
Slave servers, clustering and partitioning. Steps run locally, in one process, without partition schemas.
Connections that share a name. Hop keeps one connection per name; the import keeps the first settings it meets.
Passwords. In Hop 2.20.0 the import encrypts PDI's already-encrypted passwords a second time, so no connection password survives. See Finish connections and passwords.
Schedules. Which jobs run, and when, is in Pentaho Server, cron or Carte, not in the files.
A PDI repository. The importer reads files. Export the repository to files first.

It does not validate the result. Do that with a parallel run.

What we tested it on

The samples that ship with PDI: assemblies/samples of pentaho-kettle at tag 9.4.0.0-343, the last release under the Apache License 2.0 (later releases are under the Business Source License). That is 208 files: 189 transformations and 19 jobs. They are examples of what PDI can do, one feature each, not a production project: your own migration report gives your figures.

	Result	What was measured
Converted to a Hop plugin	95% of steps and entries (1,020 of 1,070); 9 more with a known difference	The migration report of an import of the samples. Every file converts. 29 (3%) are left to a person, mostly code that calls the Pentaho API; 12 (1%) have no Hop plugin
Flagged for review	58 steps and entries	Every PARTIAL, MANUAL and UNSUPPORTED one, and AUTO steps with a note

Converted to a Hop plugin means Hop has a transform or action for the step, with its settings. It does not mean the converted file runs, or computes what PDI computed. A test of the importer keeps these figures up to date with its code.

Where to go next

New to it? Get started imports the PDI samples, walks you through the report and runs one converted pipeline.
Working on your own project? The how-to guides cover assessing a project, importing into a Hop project, finishing connections and passwords, and what to do when something does not run.
Looking something up? The reference has every option, every report section and every audit risk.
Why migrate, and how we help? Migrations to Apache Hop, and from Pentaho PDI in particular.

How it works​

What you get​

What the report classifies​

What the Kettle import leaves out​

What we tested it on​

Where to go next​