Noname manuscript No. (will be inserted by the editor) A Middleware for Efficient Stream Processing in CUDA

Abstract This paper presents a middleware capable of out-of-order execution of kernels and data transfers for efficient stream processing in the compute unified de-vice architecture (CUDA). Our middleware runs on the CUDA-compatible graphics processing unit (GPU). Us-ing the middleware, application developers are allowed to easily overlap kernel computation with data trans-fer between the main memory and the video memory. To maximize the efficiency of this overlap, our middle-ware performs out-of-order execution of commands such as kernel invocations and data transfers. This run-time capability can be used by just replacing the original CUDA API calls with our API calls. We have applied the middleware to a practical application to understand the run-time overhead in performance. It reduces exe-cution time by 19 % and allows us to process large data that cannot be entirely stored in the video memory