Dynamics of shape recognition through a minimal visuo-tactile sensory substitution interface

We tackled the question of the constitution of a perception of shape within the framework of experiments requiring the tactile discrimination and recognition of two-dimensional pictures (characters). We developed a sensory substitution device [1][2] allowing to generate binary tactile stimuli during the exploration of black-and-white images. The experimental setup is composed of a graphic tablet, a computer and a tactile stimulus generator. Tactile stimuli were produced by small electromagnetic transducers producing 250 Hz vibrations. The position of the stylus on the tablet defines the current position of a pointer on a bitmap image. Each transducer has a square receptive field with a fixed size and position defined relative to the cursor's position on the image. A transducer is activated when the corresponding receptive field contains at least one black pixel. Stylus trajectories were recorded during the tactile exploration of images. From a fundamental point of view, the major interesting property of this device is to prevent subjects from having a global and instantaneous access to shape. We have tested the assumption that the coupling between actions and sensations during the tactile exploration of such a virtual image allows discrimination and recognition of shapes. We present here the results of three different experiments. The first experiment addressed the discrimination between the character "S" and its vertical mirror image. Three naive subjects using a system with five overlapping receptive fields have discriminated quite correctly between these two shapes, showing respectively a 90%, 100% and 100% of correct responses (20 trials, no time limit). In the same experimental conditions, the second experiment addressed the question of the efficiency of recognition of uppercase characters. A unique sequence of ten characters (ITLPSVBORD) was presented to nine subjects. No feedback was given about the subject's performance. This experiment showed that recognition was efficient but also that the percentage of correct responses depends strongly on the target character. The last experiment addressed two questions : (1) the contribution of learning to the efficiency of recognition and (2) the influence of the number of contiguous receptive fields. Two groups of five subjects were presented 50 uppercase characters (taken from a library of 10 characters) three times, with an interval of one week between the three sessions (time limit = 1 minute, verbal feedback to subjects). The two groups differed by the type of receptive fields used in the experiment (one 3x3 pixel receptive field for the first group G1, and four contiguous square 3x3 receptive fields for the second group G2). Results showed a clear learning effect for both groups: 9 correct identification out of 50 for G1 and G2 (first session) compared to 25/50 (G1) and 16/50 (G2) for the last session. However, the receptive field factor was not significant. We have demonstrated that (1) this minimal device indeed gave access to the perception of shapes, (2) that learning with feedback allowed an improvement of the level of performance. Furthermore, the analysis of verbal reports and stylus trajectories showed that subjects were easily 'lost in the image'. Proprioception [3] and path (movement) integration seem not efficient enough to allow subjects to situate the receptive field(s) in the image. This observation can explain why subjects seem to have rather a sensory access to the 'topology' of characters than to their 'geometry' (a 'D' is commonly confounded with 'O'). Moreover, stylus trajectory analysis also showed that subjects used a well-identified sensori-motor repertoire composed of a very common and automatic 'oscillatory contour following' behavior and a more 'high-level' scanning behavior. We propose this experimental apparatus as a tool to study tactile perception of 2D shapes and to provide inspiration for 'active' pattern-recognition algorithms based on sensori-motor coupling rather than on disembodied reconstruction of shape. [1] Bach Y Rita, P. (1972). Brain mechanisms in sensory substitution. New York : Academic Press. [2] C.Lenay, S.Cannu, P.Villon (1997). Technology and Perception : the Contribution of Sensory Substitution Systems, Second Int. Conf. on Cognitive Technology, Aizu, Japan, Los Alamitos: IEEE, pp. 44-53. [3] J.P. Wann and S.F. Ibrahim (1992). Does limb proprioception drift ? Exp. Brain Res., 91:162-166