Building an AI-Powered Assistant for Computational Scientists

The availability of high-performance computing (HPC) cyberinfrastructures (CI) like Ohio Supercomputer (OSC)[1] or Titan[2] enables scientists to perform computationally intensive experiments (reaching Exascale levels). These resources are expensive and hard to optimize, being shared between simple/complex tasks – long/short run times and billed per hour based on processing and memory requirements. We studied how AI injected into existing CIs could provide resource recommendations from application and system configurations, prior workflows and resulting interactions (CI traces, logs). This work examines: To what extent models trained on scaled-down datasets on commodity environments adapt to production environments, To what extent models built for specific applications can generalize to other applications within a family, and How the most appropriate model may change based on the type of data and its mix. We also describe and show the use of an automatable pipeline for generating the necessary training data and building the model