Hierarchical Dataflow Model with Automated File Management for Engineering and Scientific Applications

AbstractSolving modern scientific and engineering problems typically implies using multiple task- specific software applications and often a complex sequence of computations must be performed. Adopted approach to achieve the required level of automation is to use one of the many avail- able scientific and engineering workflow systems, which can be based on different workflow models. This paper introduces a workflow model targeted to provide natural automation and distributed execution of complex iterative computation processes, where the calculation chain contains multiple task-specific software applications which exchange files during the process. The proposed workflow model addresses a wide range of applications and targets complex cases when a single iteration of a top-level process may contain multiple nested execution loops. Typ- ical requirements to process automation are considered as well: execution isolation, data re-use and caching, parallel execution, data provenance tracking