Multi-Threaded Jobs

Ianus supports both single and multi threaded jobs.

Single thread jobs simply start and execute tasks sequentially: every task starts and runs only after its predecessor has completed.

Multi thread jobs start tasks sequentially (so in the order they are defined) and run them in parallel. Tasks are executed on a thread pool so to optimize the system’s resource usage. The degree of parallelism can be affected by the task parameters.

While single thread mode is suitable for most of cases, there are some scenarios where the parallel tasks execution helps achieving shorter processing duration. For example:

  • Comparison of a set of large tables or files.
  • Execution of jobs on multiple independent remote systems
  • Realization of complex batch test automation processes. For example:
    1. Submit data restore job on z/OS and wait for it to end
    2. Submit data migration job on Windows and wait for it to end
    3. Submit job X on z/OS and Windows and wait for both to end
    4. Compare tables updated by the job X
    5. Copy one table from z/OS to Windows
    6. Submit job Y on z/OS and Windows and wait for both to end
    7. Compare tables updated by the job Y

Enabling Multi Threaded processing

By default, Jobs run in single thread mode. To enable multi thread processing, the job’s attribute multithread must be set to true.

<job name="MT2" multithread="true">

Controlling the degree of parallelism

There are situations where it might be necessary to control the degree of tasks parallelism as well as the order in which tasks are executed.

For example, you may want to run a set of parallel table comparison only after that two JCL submit tasks have completed.

The job attribute maxparalleltasks defines the maximum number of parallel working tasks.

The tasks attribute thread controls the way tasks are started and executed:

Value Parallelism Task Start
multi Task runs in parallel. Task starts as soon as the previous parallel task has started.
single Task runs isolated. Task starts when all previous tasks are completed.
first Task runs in parallel. Task starts when all previous tasks are completed

The default value for the thread attribute is multi.

Important

The above applies when the job attribute multithread is set to true. If not, step run sequentially.

Example 1: Single Thread job

In this example, tasks are programmed to be executed sequentially.

<job name=“ST1" multithread=“false">
  
  <tablecompare name=“A" left="QS" right="QS">
     ...
  </tablecompare>
  <tablecompare name=“B" left="QS" right="QS">
     ...
  </tablecompare>
  
  <tablecompare name=“C" left="QS" right="QS">
     ...
  </tablecompare>
</job>

This will result in a flow like this:

Diagram 1

Example 2: Multi Thread Job

In this example, tasks are programmed to run in parallel.

<job name="MT1" multithread="true">
  
  <tablecompare name=“A" left="QS" right="QS">
     ...
  </tablecompare>
  <tablecompare name=“B" left="QS" right="QS">
     ...
  </tablecompare>
  
  <tablecompare name=“C" left="QS" right="QS">
     ...
  </tablecompare>
  
</job>

This will result in a flow like this:

Diagram 2

Example 3: Complex Multi Thread job

In this example, tasks parallelism is controlled by the tasks attributes.

<job name="MT2" multithread="true">
  
  <tablecompare name=“A" left="QS" right="QS" thread="multi">
     ...
  </tablecompare>
  <tablecompare name=“B" left="QS" right="QS" thread="multi">
     ...
  </tablecompare>
  
  <tablecompare name=“C" left="QS" right="QS" thread="single">
     ...
  </tablecompare>
  
  <tablecompare name=“D" left="QS" right="QS" thread="multi">
    ...
  </tablecompare>
  <tablecompare name=“E" left="QS" right="QS“ thread="multi">
    ...
  </tablecompare>
  <tablecompare name=“F" left="QS" right="QS" thread="multi">
    ...
  </tablecompare>
  <tablecompare name=“G" left="QS" right="QS" thread="first">
    ...
  </tablecompare>
  <tablecompare name=“H" left="QS" right="QS" thread="multi">
    ...
  </tablecompare>
</job>

The script above runs as follows:

Diagram 3

Log

By default, log messages are shown and recorded in the order they are emitted. In multi threaded processing, it will result in a mix of messages from different tasks, ordered chronologically. The thread token on the log message indicates the source task.

By enabling buffered logging in the environment configuration file, logs messages of single tasks are buffered and emitted in one shot.

Error control

The job's attribute stopon, control the behavior of job in case of task errors and/or warnings.

When stopon is set to error or warning, jobs behave as follows:

Threading mode Behavior
single job stops when a tasks terminates with errors/warnings
multi as soon as a tasks terminates with errors/warnings, job waits for previously started parallel tasks to terminate and stop