Recognition of Handwritten Mathematical Expressions

The Problem: Our goal is to automatically recognize handwritten mathematical expressions, converting them from pen strokes to formats commonly used by computer mathematics packages. This problem differs significantly from that of recognizing standard prose. In prose, significant constraints can be put on the interpretation of a character by the context it is in. For example, in “t?ree ” one can guess the missing letter. However, in mathematics few such simple constraints are present, so that in an expression such as “3+? ” the context does little to disambiguate the unknown character. The structure of mathematical expressions is also significantly more complex than that of prose. So whereas prose is written linearly from left to right, mathematical symbols can be written above, below, and inside one another, and these spatial relationships are crucial to the interpretation of the expression. This means that even if all the characters are correctly recognized, there still remains the nontrivial problem of interpreting the structure of the expression. (e.g. 245, 2 45,and24 5 are all distinct expressions with the same characters). Hence, it is difficult to apply the Markovian methods which have been applied to parsing one-dimensional texts such as standard prose or programming languages. Motivation: A great deal of technical work and education still takes place on a blackboard or in a notebook despite the powerful capabilities computers provide for computation and visualization. Typically, a scientist or engineer will do their “thinking ” on paper, and then use a computer to visualize specific quantities and relations that are importan