Design and Development of a Sketch Recognition Component for a Software IDE

Sketching is one of the most common practices applied by software engineers, developers and maintainers while designing and analyzing software. Given how intuitive it is to express ideas through sketching, it has been proposed to use sketching also to express commands in software tools such as an integrated development environment (IDE). This thesis contributes to this vision by building a transparent canvas hovering over a software IDE. This canvas catches all the touch input (originating from either a finger interaction or a pen interaction) and transforms it into visible digital ink, while allowing mouse and keyboard input to pass to the underlying tool. It then recognizes the drawn ink as shapes and passes these results to an extension of the underlying IDE. Based on the work on this thesis, the separately developed IDE extension can then add meaning to the drawings by interpreting them and executing the according commands. This thesis details the steps and decisions taken in the development of the transparent sketching canvas that provides support for digital inking while not removing much of the underlying IDE's original functionalities. The canvas locally stores the ink strokes, recognizes them as instances of pre-defined shapes, and sends the results to the IDE, where they can be interpreted further.This work was supported by the Icelandic Research Fund (Grant no. 196228