Toward fine-grained online task characteristics estimation in scientific workflows

Task characteristics estimations such as runtime, disk space, and memory consumption, are commonly used by scheduling algorithms and resource provisioning techniques to provide successful and efficient workflow executions. These methods assume that accurate estimations are available, but in pro-duction systems it is hard to compute such estimates with good accuracy. In this work, we first profile three real sci-entific workflows collecting fine-grained information such as process I/O, runtime, memory usage, and CPU utilization. We then propose a method to automatically characterize workflow task needs based on these profiles. Our method es-timates task runtime, disk space, and memory consumption based on the size of tasks input data. It looks for correla-tions between the parameters of a dataset, and if no correla