Skip to main content

Get started with the Pentaho (PDI) importer

Coming soon

The PDI importer is about to be released as open source (Apache License 2.0). It needs Apache Hop 2.20.0, which is not released yet. Until then, we run it for you: contact us.

In this tutorial you install the importer in Apache Hop, import the samples that ship with PDI, read the migration report and the pre-migration audit, and run one converted pipeline. It takes about a quarter of an hour. You need neither PDI nor a database.

The samples are the assemblies/samples folder of pentaho-kettle 9.4, the last release under the Apache License 2.0: 189 transformations and 19 jobs, each showing one PDI feature.

Before you start​

  • Apache Hop 2.20.0 or later. See Install Apache Hop.
  • Java 21, which Hop needs anyway.
  • git.

In the commands below, <hop> is the folder Hop is installed in. Run them all from the same folder: Hop's scripts need full paths, so the commands use $PWD.

1. Install the plugin​

Download importer-pdi-<version>.zip from the importer's releases, once published, and unzip it into the Hop folder. The zip holds a plugins/ folder, so it lands in the right place:

unzip importer-pdi-<version>.zip -d <hop>

Start Hop GUI (or restart it). The File menu now has Import from PDI (with migration report)… and Audit a PDI project before a migration…. Hop's own Import from Kettle/PDI stays: it converts the same way, without the report.

2. Get the PDI samples​

Fetch only the samples folder of the 9.4.0.0-343 tag, not the whole PDI source:

git clone --depth 1 --branch 9.4.0.0-343 --filter=blob:none --sparse \
  https://github.com/pentaho/pentaho-kettle.git
git -C pentaho-kettle sparse-checkout set assemblies/samples/src/main/resources

The samples are in pentaho-kettle/assemblies/samples/src/main/resources/: transformations/, jobs/, and the data files the transformations read, in transformations/files/.

3. Audit it​

The audit tells you what a project holds, and its risks, before you convert anything. It writes no Hop files:

<hop>/hop-import.sh -t pdi-audit \
  -i "$PWD/pentaho-kettle/assemblies/samples/src/main/resources" -o "$PWD/pdi-audit"

Open pdi-audit/pre-migration-audit.md. Its Risks table starts with:

LevelRiskJobs
HIGHCode that calls the Pentaho API20
HIGHSteps and entries without a Hop plugin11
MEDIUMConnections with one name and different settings8
MEDIUMVariables the project does not define7
MEDIUMSteps to replace, or to install as plugins6
MEDIUMLog tables3

The audit counts transformations and jobs alike as "jobs". Each risk comes with what it means and what reduces it; the audit risks reference lists them all.

4. Import it​

<hop>/hop-import.sh -t pdi \
  -i "$PWD/pentaho-kettle/assemblies/samples/src/main/resources" -o "$PWD/pdi-samples-hop"

You can do the same from File → Import from PDI (with migration report)… in Hop GUI.

Expected result: the log ends with the Kettle import's summary:

Imported :
19 jobs
189 transformations
4 database connections where saved in metadata folder file:///…/pdi-samples-hop/metadata

and pdi-samples-hop/ holds:

  • transformations/ and jobs/, with a .hpl for each transformation and a .hwf for each job, and the data files copied next to them;
  • metadata/rdbms/: four connections, Generic Sample, SampleData, SampleData.H2 and SampleDataAdmin.H2;
  • metadata/pipeline-run-configuration/ and metadata/workflow-run-configuration/: the local run configurations;
  • connections.csv and migration/conversion-metadata.json;
  • pdi-migration-report.md.

5. Read the report​

Open pdi-samples-hop/pdi-migration-report.md. Its Totals are:

MetricCount
Jobs208
Components1070
AUTO1020
PARTIAL9
MANUAL29
UNSUPPORTED12
Flagged for review58

A "job" in the report is a transformation or a job; a component is a step or a job entry. The other sections tell you where the work is:

  • Per job: steps and flagged steps for each file. Most have 0 in the last column.
  • Most frequent source components: what the project is made of. Here, Generate Rows, Dummy and JavaScript (ScriptValueMod) steps.
  • Needs review: the 58 flagged steps and entries, with the reason. Grouped by reason:
    • MANUAL, code that calls the Pentaho API (23, in the 20 files the audit counted): User Defined Java Classes that use StepMetaInterface or org.pentaho.di, and JavaScript that uses _step_. Each one is ported to Hop's API by hand.
    • MANUAL, steps to replace (6): Excel Output (replace it with Microsoft Excel Writer), Fixed File Input (Text File Input with the file type Fixed), Edi to XML, and the message box entry.
    • UNSUPPORTED (12): HL7 Input, Automatic Documentation Output, Single Threader, XBase Input, Get Repository Names, Spark Submit and Pentaho Reporting Output have no Hop plugin.
    • PARTIAL (9): Data Validator steps, whose validations are to be checked one by one.
    • AUTO, renamed by the import (8): steps named after a file, such as process flow step 2a - input1.ktr. The import replaces .ktr and .kjb in all text, so the step is now called process flow step 2a - input1.hpl.
  • Apache Hop's Kettle import: which Hop version converted the files, and the import's summary.

These figures describe the samples, which show off PDI features one at a time. Your own project will have its own; see why there is no conversion rate.

6. Run one pipeline​

CSV Input - Reading customer data reads files/customers-100.txt, a file of 100 customers, and keeps those that match a filter.

  1. Make the output folder a Hop project, so Hop finds the connections and run configurations in its metadata/ folder:

    <hop>/hop-conf.sh -pc -p pdi-samples -ph "$PWD/pdi-samples-hop"

  2. Run the pipeline:

    <hop>/hop-run.sh -j pdi-samples -r local \
      -f "$PWD/pdi-samples-hop/transformations/CSV Input - Reading customer data.hpl"

    It fails: Could not read from "file:///…/${Internal.Pipeline.Filename.Folder}/files/customers-100.txt". PDI's ${Internal.Transformation.Filename.Directory} is renamed by Hop 2.20.0's Kettle import to ${Internal.Pipeline.Filename.Folder}, a variable Hop does not set; Hop's name is ${Internal.Pipeline.Filename.Directory}. 81 of the 208 samples use it, and the report does not flag them yet.

  3. Put Hop's name in every converted file:

    find pdi-samples-hop \( -name '*.hpl' -o -name '*.hwf' \) -exec \
      perl -pi -e 's/Internal\.Pipeline\.Filename\.Folder/Internal.Pipeline.Filename.Directory/g' {} +

  4. Run the pipeline again.

Expected result: the pipeline finishes without errors, and the log shows the rows going through:

CSV file input.0 - Finished processing (I=101, O=0, R=0, W=100, U=0, E=0)
Filter rows.0 - Finished processing (I=0, O=0, R=100, W=5, U=0, E=0)
Dummy (do nothing).0 - Finished processing (I=0, O=0, R=5, W=5, U=0, E=0)

The file has a header and 100 customers; five of them pass the filter.

Next​