Skip to main content

From import to verified project

Coming soon

The Talend importer is about to be released as open source (Apache License 2.0). Until then, we run it for you: contact us. The demo project this tutorial uses is not public yet.

In this tutorial you take a Talend project all the way: import it into a Hop project, run every converted job, compare what they write with what the Talend jobs wrote, and fix what differs until the comparison says so. It takes about an hour.

The project is a demo of receivables and collections jobs: customers, invoices, payments, a staging database, an XML export and a nightly run. Most jobs convert and run as imported; the others each show one kind of work a migration still needs. The demo comes with what the Talend jobs wrote when Talend ran them, so you need no Talend.

Before you start​

  • Apache Hop 2.19.0 or later, with the Talend importer installed. See Get started.
  • git, and a bash shell: macOS or Linux; on Windows, WSL.
  • Docker, only for the optional last step.

In the commands below, <hop> is the folder Hop is installed in.

1. Get the demo​

git clone https://forge.putki.io/putki-internal/talend-import-demo.git
cd talend-import-demo
./demo.sh setup <hop>

Expected result: setup registers the Hop project collect-demo (the hop folder) and its environment collect-demo-dev. The folder holds:

  • talend/COLLECT_DEMO/: the Talend project, 16 jobs;
  • reference/talend/: what those jobs wrote when Talend ran them;
  • hop/: the Hop project, with its input data (data/in) and the workflows that run and compare everything (demo/, compare/).

2. Import the Talend project​

./demo.sh import

This runs hop-import -t talend into the Hop project. In Hop GUI, it is File → Import from Talend… with talend/COLLECT_DEMO as the source and the hop folder as the target.

Expected result: hop/talend-migration-report.md, whose Totals are:

MetricCount
Jobs16
Components67
AUTO61
PARTIAL2
MANUAL4
Flagged for review8

The Needs review table names the components you will meet again below: a tJavaRow whose Java could not be translated, a tFilterRow that reads a Talend counter, and a Dynamic column.

3. Run every job​

./demo.sh run

run executes hop/demo/run-all.hwf: it finds the file each job runs from, runs the jobs one after the other, and writes hop/data/out/run-results.csv. A failed job does not stop the others.

Expected result: 12 jobs OK, 4 FAILED: LoadCustomers, NightlyRun (it runs LoadCustomers), LoadToQBC and NotifyAgency.

4. Compare with Talend​

In Hop GUI, open the project collect-demo with the environment collect-demo-dev, and run compare/compare-all.hwf. Or from the shell:

<hop>/hop-run.sh -j collect-demo -e collect-demo-dev -f "$PWD/hop/compare/compare-all.hwf"

The workflow checks that the Talend jobs and input data are the ones the reference was made from, then compares every file and table the Hop run wrote with the Talend reference, line by line.

Expected result: hop/data/compare/summary.csv (also in the log) starts with REFERENCE VALID. Most files are SAME. The ones LoadCustomers writes are DIFFERENT: Hop wrote only their header.

5. Use the report's Hop setting​

LoadCustomers fails on a customer without an e-mail address: Talend reads the empty field as an empty string, Hop reads it as null. The report's Hop settings section says what to do. Run again with that setting:

./demo.sh run -s HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y

Then compare again, with the same setting (add -s HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y to the hop-run command, or set it under Edit → Edit config variables in Hop GUI and restart it).

Expected result: 14 jobs OK. The summary has 14 files SAME; debtor_scores.csv and payment_alerts.csv are DIFFERENT; three files have NO TALEND REFERENCE.

Two of those jobs ran without an error and still wrote the wrong data. That is what the comparison is for.

6. Fix a job that runs but writes the wrong data​

debtor_scores.csv is DIFFERENT. hop/data/compare/diff/debtor_scores.csv shows why: Talend wrote a risk class (HIGH, MEDIUM, LOW) for every invoice, Hop wrote none. The job's tJavaRow computes the class with Talend routines; the importer could not translate that code, so it kept it as a comment in a User Defined Java Class that copies the columns.

Open hop/process/Collections/Invoices/ScoreDebtors_0_1.hpl, edit the transform RiskScore, and replace processRow() with the code under Custom Java. Run the pipeline, then compare again.

Expected result: debtor_scores.csv is SAME: 15 files match Talend.

7. Point a connection at a real database (optional)​

LoadToQBC loads the cleaned customers into SQL Server, at a host the demo does not have. Its connection reads its host, user and password from variables, with the values the Talend job had. Start the demo's SQL Server and add its settings to the environment:

docker compose --profile sqlserver up -d sqlserver-init
./demo.sh fix-qbc
./demo.sh run -s HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y

Expected result: LoadToQBC is OK: 15 jobs OK, and the 6 cleaned customers are in the server's QBC database. NotifyAgency still fails: it runs a Windows command, which Fix what the import could not convert covers with the other cases this tutorial leaves.

What you did​

You imported a Talend project, ran every job, and proved per file and per table which jobs write what Talend writes. You fixed two kinds of difference: a Hop setting, and code the importer could not translate. On your own project, the same loop applies: run, compare, fix, compare again.

Next​