Fig. 9

Parallel and distributed task execution. Three different types of executors are implemented in Watchdog: (i) execution on the local host that runs Watchdog, (ii) remote execution via SSH or (iii) cluster execution using DRMAA or the Slurm Workload Manager. a In this example, the four subtasks 1a, 1b, 2a and 2b are created by Watchdog based on tasks 1 and 2 using process blocks. Task 2a depends on 1a, and 2b on 1b. All tasks are assumed to require the same runtime. b By default, one task is executed after the other on the host running Watchdog. cWatchdog also allows parallel execution in all three execution modes (local, remote and cluster execution). d For remote execution, Watchdog establishes a SSH connection to pre-defined execution hosts and randomly distributes the tasks that should be executed to these execution hosts. e For cluster execution, the DRMAA or Slurm master receives tasks to execute and redirects them to its execution hosts. Watchdog has no influence on which execution host is used for task execution because the tasks are distributed by the internal DRMAA or Slurm scheduler. f During slave mode (supported for remote and cluster execution), tasks or subtasks that depend on each other are scheduled on the same execution host, which allows using the local disk space of the host for storage of files that are needed only temporarily but by different tasks