Skip to main content

How a Talend job becomes Hop files

Talend keeps a whole job in one item: its subjobs, the triggers between them, the pre- and post-job, and its iterations. Hop separates the two things a job does. A pipeline moves rows, with all its transforms running at the same time. A workflow runs things one after the other. So the importer makes one pipeline of a job that is one data flow, and a workflow with several pipelines of a job that has steps in an order.

This page explains the files you find after an import, and why each exists.

The entry file​

A job runs from its entry file, the first of these that exists:

FileWritten when the job has
<job>_orchestration.hwfa tPrejob or a tPostjob
<job>_control_flow.hwftriggers between its subjobs (OnSubjobOk, OnComponentOk, RunIf), or a table action
<job>_workflow.hwfonly file, FTP and other workflow actions
<job>.hwf or <job>.hplnone of the above: one data flow

The other files are parts the entry file runs: <job>_subjob_<first component>.hpl for each subjob, <job>_iterate_<first component> for an iteration, <job>_pre.hwf and <job>_post.hwf for the pre- and post-job. Run the entry file; a part on its own misses what runs before it.

Transforms and actions keep the names on the Talend canvas (StagingGetEODDate, tMap_1), so you can put the job and its Hop files side by side.

Subjobs and triggers become a workflow​

Each subjob, the components joined by data links, becomes a pipeline. The triggers between them become workflow hops: OnSubjobOk and OnComponentOk a success hop, OnSubjobError a failure hop, and RunIf a hop behind an action that evaluates the condition.

Pre- and post-jobs: the orchestration workflow​

A job with a tPrejob and a tPostjob becomes four workflows. In the tutorial's NightlyRun, the canvas has a pre-job with a warning, a post-job with a warning, and four tRunJobs linked by OnSubjobOk:

Hop fileTalend partWhat is in it
NightlyRun_0_1_orchestration.hwfthe jobthe entry file: runs the pre-job, the main part, then the post-job
NightlyRun_0_1_pre.hwftPrejobthe warning, as a Write to log action
NightlyRun_0_1_control_flow.hwfthe four tRunJobsone action per child job, linked by success hops
NightlyRun_0_1_post.hwftPostjobthe second warning

The post-job runs whether the main part succeeded or not, as in Talend, and the job still fails when the main part did: after a failure, the orchestration runs a copy of the post-job followed by an Abort. A workflow's result is that of its last action, so without that copy a failed job would end in success.

Child jobs: actions that run other files​

A tRunJob is not a component in Hop. It becomes an action (or, in a data flow, a Pipeline Executor) that runs the child job's entry file. Each child is converted on its own and keeps its own files; a child that is itself a workflow is run by a workflow action.

Iterations: an executor and a file of their own​

An ITERATE link (from tFileList, tFlowToIterate, or a tForeach or tLoop with literal values) runs what follows it once per row, with that row's values in globalMap. Hop runs the iterated part as a file of its own, once per row, through an executor, with the values as parameters. In the tutorial's RouteAssignments, a tFlowToIterate runs the rest of the job once per target type, and a tMap routes rows with globalMap.get("target.TYPE"):

Hop fileWhat is in it
RouteAssignments_0_1.hplthe entry file: reads the target types; a Workflow Executor runs the iterated part per row, passing the row's columns as parameters (target.TYPE); an Abort stops the job when an iteration fails, as Talend does
RouteAssignments_0_1_iterate_….hwfone iteration: the SQL action that creates the table if it does not exist, then the pipeline below
RouteAssignments_0_1_iterate_…_….hplthe data flow: the reads, the validation, the tMap and its three outputs

globalMap.get("target.TYPE") becomes ${target.TYPE}. A table action (create, drop, clear) moves out of the output component into a SQL action before the pipeline that writes the rows, where Talend runs it as the output component starts.

One tMap, several transforms​

A tMap does in one component what Hop does in several transforms. For each output, the importer writes a Calculator for the expressions it can translate (or a User Defined Java Class for the ones it cannot), a filter for the output's filter, and a Select Values that gives the output its columns; a lookup adds a Stream Lookup or a Merge Join. All of them carry the tMap's name, so tMap_1, tMap_1_tmap_calc and tMap_1_tmap_output_filter_invalid are the same Talend component.

Contexts and connections: the environment​

Talend contexts become Hop environment files, environments/<context>-config.json, and the pipelines read the context variables from the environment you run with. The database connections read their settings from variables too, with the values the Talend jobs had in those files. See Finish connections and passwords.