Skip to main content

Fix what the import could not convert

The migration report lists what the importer could not convert. This page covers what that means when you run the converted jobs: the ones that fail, and the ones that run and write the wrong data. Each case has the symptom, the cause and the fix. The examples are from the From import to verified project tutorial.

Find the second kind by comparing the output with Talend's, as in Validate a migration: a job that ends without an error is not yet a job that does what Talend did.

CaseThe jobThe report
Empty stringsfailsHop settings
ConnectionsfailsEnvironments, Passwords
Operating system commandsfailsnot flagged
Custom Javaruns, wrong dataMANUAL
Talend counters in a filterruns, wrong dataMANUAL
Dynamic child jobsruns, wrong datanot flagged
Dynamic schemasruns, wrong dataMANUAL

Empty strings: a job fails on null​

Symptom. An expression fails with a NullPointerException, for example on row1.email.toLowerCase(), for a row whose field is empty in the input file.

Cause. Talend keeps an empty string apart from null: a file input reads an empty field as "". Hop treats the two as the same unless told otherwise, so the field arrives as null.

Fix. Run with HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y, as the report's Hop settings section says: -s HOP_EMPTY_STRING_DIFFERS_FROM_NULL=Y with hop-run, or under Edit → Edit config variables in Hop GUI. Hop reads it when it starts, so it applies to everything that Hop runs; restart Hop GUI after setting it.

Connections: a job cannot reach its database​

Symptom. Cannot connect to database: an unknown host, or a login that fails.

Cause. The connection reads its host, user and password from variables, with the values the Talend job had: a test or production server you cannot reach from here, or a password Talend encrypted, which is empty.

Fix. Give the variables this environment's values in a config file of your own, after the one the import wrote. See Finish connections and passwords.

Operating system commands: a Shell action fails​

Symptom. A Shell action converted from tSystem fails with Cannot run program, or runs in a folder such as C:/collect that does not exist.

Cause. The command is carried over as it is: cmd /c copy C:\… only runs on the Windows machine it was written for.

Fix. Replace the command with the Hop action that does the same thing (Copy files, Move files, Delete files, …), with the paths as variables. Keep a Shell action only for a real program, and give its path per environment.

Custom Java: runs, but a column stays empty​

Symptom. The job runs; a column its tJavaRow computes is empty, or still holds its input value.

Cause. The importer translates the tJavaRow code it recognises. Code that calls Talend routines (TalendDate, StringHandling, …) or reads globalMap is kept as a comment in a User Defined Java Class whose processRow() copies the columns. The report lists the component as MANUAL, with the lines it could not translate.

Fix. Port the code into processRow(), reading fields with get(Fields.In, "name") and writing them with get(Fields.Out, "name").setValue(...). For example, a risk class from the days past due on an assessment date (a context variable, asOfDate):

public boolean processRow() throws HopException {
    Object[] r = getRow();
    if (r == null) {
        setOutputDone();
        return false;
    }
    Object[] outputRow = createOutputRow(r, data.outputRowMeta.size());
    java.time.LocalDate asOf = java.time.LocalDate.parse(getVariable("asOfDate", ""));
    java.time.LocalDate due = java.time.LocalDate.parse(get(Fields.In, "due_date").getString(r));
    long overdue = java.time.temporal.ChronoUnit.DAYS.between(due, asOf);
    java.math.BigDecimal amount = get(Fields.In, "amount").getBigDecimal(r);
    String risk;
    if (overdue > 60 || amount.compareTo(new java.math.BigDecimal("1000")) > 0) {
        risk = "HIGH";
    } else if (overdue > 30) {
        risk = "MEDIUM";
    } else {
        risk = "LOW";
    }
    get(Fields.Out, "risk").setValue(outputRow, risk);
    putRow(data.outputRowMeta, outputRow);
    return true;
}

Or rebuild it without Java: a Calculator for the days between two dates, then Number range or Switch / case. See Port Java components.

Talend counters in a filter: runs, but lets every row through​

Symptom. The job runs and writes rows its filter should have dropped: an alert raised although the file it checks has rows.

Cause. The filter reads a counter Talend keeps for another component, such as ((Integer)globalMap.get("tFileInputDelimited_1_NB_LINE")) == 0. A Hop pipeline cannot read another transform's row count, so the importer could not translate the condition, and left a pass-through with a MANUAL REVIEW REQUIRED note.

Fix. Count in Hop: read the source again, add a Group by that counts the rows with Always give back a result row, and put Filter rows on that count in front of the output. In a workflow, Evaluate rows number in a table or a pipeline that sets a variable does the same.

Dynamic child jobs: runs the same child every time​

Symptom. A job that runs one child job per row (a tRunJob with Use dynamic job) writes the output of the same child for every row: a customers data set that holds invoices.

Cause. The importer runs the child job the tRunJob was created with, for every row. It does not convert the expression that chooses the child, and the report does not flag it.

Fix. The iterated workflow the importer wrote receives the row's columns as parameters, the child's name among them. Make the action that runs the child take its file from that parameter, for example ${Internal.Workflow.Filename.Folder}/${row1.job}_0_1.hpl.

Dynamic schemas: runs, but writes blank lines​

Symptom. A job that copies whatever columns a file has writes empty lines.

Cause. A Dynamic column (a feature of Talend's subscription products) stands for the columns a source has at run time. Hop has no such type: the importer leaves the column out and flags the components MANUAL.

Fix. For a plain copy, use the Copy files action. For a flow that must work for any columns, use Metadata injection: a template pipeline whose fields are filled in at run time.