Watt, S. J. 1 , Diedrichsen, J. 2 & Takahashi, C. 1
1 School of Psychology, Bangor University, Wales
2 Institute of Cognitive Neuroscience, UCL, London, England
We often see and feel the same object simultaneously, and so estimates of object properties are available from vision and haptics. Here we explore whether the brain integrates information from these two sensory modalities when we use tools to interact with objects. In normal grasping, the brain does not select one sensory modality and ignore the other. Instead, information from both signals is integrated (Ernst & Banks, 2002), and exploiting the redundancy in multiple signals results in better performance than would be possible from either signal alone. For visual-haptic integration to be effective, however, the brain must solve a 'correspondence problem': it should integrate information referring to the same object, and it should NOT integrate information referring to different objects. This could be achieved by considering the similarity of signals in the two sensory modalities (Ernst, 2007; Körding et al., 2007). For example, if there is a large conflict between two size estimates, or they are separated in space, it is unlikely that they originate from the same object. Tools complicate this, however, because they systematically change the relationship between (seen) object size and the opening of the hand, as well as perturbing the spatial locations of each signal. We report several experiments showing that the brain does take these changes into account when using a tool: vision and haptics are integrated near-optimally, but only when it is appropriate to do so. We argue, therefore, that the brain combines visual and haptic information, not based on the similarity of sensory stimuli, but based on the similarity of the distal causes of stimuli, taking into account the dynamics and geometry of tools. We speculate that this is achieved by altering the forward model used to control arm movements, rather than by a specialised mechanism for tool use.