Skip to main content

Talend importer

Coming soon

The Talend importer is about to be released as open source (Apache License 2.0). Until then, we run it for you: contact us.

The Talend importer converts the jobs in a Talend project into Apache Hop pipelines and workflows, and writes a migration report that lists every component that needs a person, with the reason. It will be open source (Apache License 2.0), and runs from the Hop GUI, from hop-import, or as a standalone command line.

PartWhat it doesWhere it runs
Hop pluginAdds File → Import from Talend… to Hop GUI and -t talend to hop-importInside Apache Hop 2.19.0 or later
Command lineThe same import as a single jar, for scripts and CIAny machine with Java 21
Migration reporttalend-migration-report.md in the output folderWritten by every import

The plugin and the command line run the same code, with the same options and defaults, so they convert, skip and report identically.

How it works​

  1. The importer walks the input folder and reads every job and joblet (.item files). It keeps the newest saved version of each job unless you ask for all of them.
  2. It reads the project's context groups and repository connections.
  3. For each component, the converter registered for it writes the Hop transform or action. A component with no converter, or whose converter fails, becomes a placeholder.
  4. Components joined by data links become a pipeline. When triggers (OnSubjobOk, RunIf, …) order subjobs, or the job has a tPrejob / tPostjob, workflows run the pipelines in that order.
  5. Contexts become Hop environments, repository connections become Hop connections.
  6. The report is written from what the import actually did.

What you get​

For a job called Load, in the output folder, under the same folder layout as the project:

  • Load.hpl: the job's data flow as a pipeline;
  • Load_control_flow.hwf, Load_pre.hwf, Load_post.hwf, Load_orchestration.hwf: only when the job has triggers or a pre-/post-job; then the job runs from the workflow, not the pipeline;
  • metadata/: Hop connections named after the Talend repository connections, static schema definitions, and the local run configurations;
  • environments/<context>-config.json: one Hop environment per Talend context (Default, Dev, Prod, …), with the values of all jobs;
  • talend-migration-report.md: see The migration report.

Design choices worth knowing​

  • Topology is kept. A component the importer cannot convert stays in the pipeline as a Dummy with its name and hops, so nothing disappears silently.
  • Contexts are not pipeline parameters. Hop sets a parameter to its default when a pipeline runs, which would hide the environment's value; so context values come from the environment you run with, and values a parent job passes still arrive.
  • One connection per database. Components that use the same database share one Hop connection across all jobs, named after the Talend repository connection.
  • Hop manages connections and commits. t*Close and a final t*Commit are left out and listed in the report; other commits and rollbacks become SQL actions in a workflow.
  • Java is translated where it can be, and always flagged. tJavaRow and the per-row part of tJavaFlex are translated idiom by idiom into a User Defined Java Class; what is not recognised is kept as a comment. tJavaFlex start and end code is kept for porting by hand.

For the reasoning behind each conversion, see docs/design-patterns.md in the importer's repository.

What it does not do​

  • Passwords: Talend encrypts them with a key that is not in the export. A connection's password is a variable, empty in every environment, and the report lists it. See Finish connections and passwords.
  • Embedded Java (tJava, tJavaFlex start/end code) cannot be translated reliably by any tool; the report lists each block with its code.
  • Other loops: tFileList, tFlowToIterate, and tForeach or tLoop with literal values run their iterations through an executor (see How a Talend job becomes Hop files). Any other loop becomes a placeholder; rebuild it as a Hop workflow loop. The report's appendix shows how.
  • Talend Administration Center schedules, ESB routes and services, custom components and SAP components are not converted.
  • It does not validate the result. Do that with a parallel run.

Why and how to move to Apache Hop, and how we help: putki.io/migrations, and from Talend in particular.

What we tested it on​

The test corpus is the example code of Packt's Getting Started with Talend Open Studio for Data Integration, on GitHub under the MIT licence: 134 .item files holding 49 jobs, in one project. They are teaching examples: file and MySQL I/O, tMap, filters, FTP and XML. It is a sample project, not yours; the migration report of your own import gives your figures.

ResultWhat was measured
Converted automatically90% of components (131 of 146); 94% fully or partly (137)The report of an import of the whole project with the default options. 6% (9) are left to a person, 0 cannot be converted. 49 of the 49 jobs import: 58 pipelines and 14 workflows
Runs end to end93% of the jobs the comparison could set up run to the end on Hop (38 of 41); the other 3 fail in Talend tooThe importer's compare/ harness builds and runs each job with Talaxie (the community continuation of Talend Open Studio), converts it, and runs it with Hop on the same files and MySQL database. 41 of the 49 jobs can be set up; the others need an FTP server, a message box, a database the dump does not have, or loop for good. Of the 3 that fail, 2 do not build in Talend
Same result as the original100% of the files and tables the 38 jobs leave match Talend's45 files are identical, 1 is identical apart from run-time values (timestamps) and 1 has the same rows in another order; 2 files both sides delete; the one table a job changes is identical. Filtering3 is shown side by side in Before and after

The AUTO count is high because these are teaching examples: tJava, embedded Java that no importer can translate, appears once in the whole project. On a project with Java in every job, far fewer components convert automatically; see Why there is no conversion rate.

Where to go next​