The PDI migration report
Every import writes pdi-migration-report.md into the output folder. It classifies what Apache
Hop's Kettle import wrote; it does not change the converted files. The report uses the words of all
the importers: a job is a PDI transformation or job, a component is a step or a job entry.
Sections
| Section | What it holds |
|---|---|
| Totals | Files, steps and entries, and the count per status; the number flagged for review |
| Per job | For each file: its steps or entries and how many are flagged |
| Most frequent source components | Each PDI type id (ScriptValueMod, TRANS, …), how often it occurs, how often it is flagged |
| Needs review | Every flagged step and entry: file, type, name, status, and the reason |
| Scope of this report | What the statuses do and do not claim |
| Files without a converted file | PDI files the Kettle import wrote no pipeline or workflow for; only when there are any |
| Connections without a Hop database plugin | Connections imported as GENERIC, with their PDI type and file; only when there are any |
| Apache Hop's Kettle import | The Hop version that converted the files, the Hop version of the plugin catalog, and the Kettle import's own summary: files, connections, renamed connections |
From the report of an import of the PDI 9.4 samples, four rows of the Needs review section:
| Job | Component | Name | Status | Why |
|---|---|---|---|---|
| transformations/Excel Output - Write different data types.ktr | ExcelOutput | Excel Output | MANUAL | Its library cannot be shipped with Apache Hop. Replace the step with Microsoft Excel Writer, which writes .xls and .xlsx, or install the Excel Output plugin separately. |
| transformations/User Defined Java Class - Calculate the date of Easter.ktr | UserDefinedJavaClass | User Defined Java Class | MANUAL | The code still uses StepMetaInterface; port it to Hop's API (org.apache.hop), then test it. |
| transformations/Data Validator - validate data against external reference data.ktr | Validator | Data Validator | PARTIAL | Check each validation of the step in Hop's Data Validator and test it with rows that must fail. |
| transformations/XBase - dBase.ktr | XBaseInput | XBase input | UNSUPPORTED | Hop opens it as a missing plugin: rebuild what it does with Hop transforms, or install a plugin with this id. |
How a component's status is decided
The classifier reads each PDI file and the Hop file the Kettle import wrote from it, and matches steps and entries by name, also when the import renamed them. Each one's status comes from the Hop type it became:
| What it found | Reported as |
|---|---|
| The Hop type is an Apache Hop plugin | AUTO |
| The same, with a note to review: a Formula, which runs as Hop's OpenFormula transform and evaluates PDI's formula syntax; a step the import renamed | AUTO, flagged |
| A Hop plugin exists, with a known difference: Data Validator, of which not every validation carries over | PARTIAL |
A User Defined Java Class, JavaScript, Java Filter or Janino step whose code uses the Pentaho API: org.pentaho classes, KettleException and the other renamed classes, StepMetaInterface, _step_, getTrans(), getTransMeta() | MANUAL |
| A step that exists only as a separate plugin, or has a replacement in Hop: Excel Output, LDIF Input, Mail Validator, Syslog Message, MQTT Producer and Consumer, Fixed File Input, Edi to XML, the message box entry | MANUAL |
| No Hop plugin has the type's id | UNSUPPORTED |
| The step or entry is missing from the converted file | UNSUPPORTED |
Which ids are Hop plugins comes from two places: a catalog of the transform and action ids of the Hop version the importer was built against, and the plugins installed in the Hop that runs the import. Install the Excel Output plugin, say, and Excel Output steps count as AUTO.
What the report does not see
The classifier compares step types. It does not check what is inside a step's settings, so these are not in the report:
${Internal.Pipeline.Filename.Folder}, the name Hop 2.20.0's Kettle import gives PDI's${Internal.Transformation.Filename.Directory}, which Hop does not set; see Troubleshoot;- connection passwords and JNDI connections; see Finish connections and passwords;
- what the Kettle import leaves out: log tables, slave servers, clustering and partitioning. The pre-migration audit lists those.
Richer reports
An add-on that implements the importer's IImportReportProvider interface replaces the first five
sections with its own. Nothing in the importer depends on one being installed.
Why and how to move, and how we help: Migrations to Apache Hop, and from Pentaho PDI in particular.