Sensory loss in either the eyes or the ears affects our ability to recognize emotions, but does loss in both eyes and ears further worsen our ability to interpret multisensory signals? In a recent paper published in the journal “Vision Research”, Minke de Boer and co-authors find that it does not. Surprisingly, in an experiment in which control participants observed movies of people speaking emotionally charged (non-sense) sentences, they showed that – while degrading sound or vision had an important influence of participants’ ability to recognize emotions – combined degradation did not make this any worse. Moreover, the found that observers could compensate well for degraded audio, but barely for degraded video. Based on eye tracking, they also found that participants’ gaze behavior adapts to degraded video and absent audio, but not to degraded audio. Information such as this is important for rehabilitation. Want to know more? You can find the paper here.
An example of a frame from one of the movies in which the visual information was degraded in a gaze dependent manner. The red dot indicates where the participant was fixating (dot not shown during the experiment).