The general goal of our research is to understand the mechanisms and principles of human perception. There are currently two related but distinct topics that define the foci of research in our group: visual perception and multi-sensory integration.

Multi-sensory Perception

At present, our research is mainly concerned with the question of how information is integrated from multiple sensory modalities into a coherent percept of the world, with a special interest in the question of how visual perception is affected by other sensory modalities. Perception has traditionally been viewed as a modular function with different sensory modalities operating largely as separate and independent modules. As a result, multisensory integration has been one of the least-studied areas of research in perception. However, accumulating evidence for cross-modal interactions in perception has led to a new surge of interest in this field making it arguably one of the fastest-growing areas of research in perception. Our research has been among those that have started a shift towards an integrated and interactive paradigm of sensory processing.

Our research tackles the question of multisensory integration at various levels:

Phenomenology: This is to find out how the different modalities interact at a descriptive level, or by investigating the phenomenology of these interactions using behavioral experiments.

Brain Mechanism: in other words, to find out which brain areas are involved in multisensory integration, what kind of circuiry (bottom-up, top-down, etc.) is involved, what kind of neuronal coding mechanism may be used for binding the signals of the different modalities (e.g., Neuronal Synchrony, spike rate, etc.), and many other questions of this kind. We have been using event-related brain potentials and functional neuroimaging to investigate these questions. We are currently collaborating with neurophysiologists making multi-electrode recordings in awake behaving monkeys.

Computational Principles:in other words, to find out what the general theoretical rules governing integration of information from different modalities are. To gain insight into these general principles, one needs to find a model that can account for the behavioral data.

Below are brief description of some of our recent studies:

Psychophysical Study of cross-modal interactions
Vision has been traditionally viewed as the dominant modality; one that operates independently of the other modalities. This general view has been changing in the light of accumulating evidence documenting the modulation of visual percepts by signals from other sensory modalities. This shift in mentality, however, has been slow up until recently, since the reports of cross-modal modulation of the visual percept have involved only small, quantitative and gradual changes in the percept. The sound-induced flash illusion, which we have recently reported (Nature 2000), on the other hand, provides ground for a decisive change in the traditional view by demonstrating a radical alteration of the visual percept, i.e. an alteration in its phenomenological quality. It also suggests that these modulations are more commonplace than previously thought.

Brain mechanisms underlying multisensory integration
We recently performed an event-related brain potential (ERP) study that uses the sound-induced flash illusion as its framework. Surprisingly, the results of this study suggest that the neural activity is already modulated by sound at the level of supposed modality-specific visual cortical areas. We have been also conducting a functional Magnetic Resonance Imaging (fMRI) study to find the specific brain areas involved in these auditory-visual interactions. The preliminary results of this study suggest that the activity in early visual cortices (V1, V2, V4) is selectively enhanced in trials in which sound alters the visual perception (i.e., when the illusion occurs) and not otherwise. This is an astonishingly early level of sensory processing, and thus, these results taken together play an important role in overturning the predominant hypothesis of modularity of sensory modalities.

Modeling multi-modal perception
Modeling studies are necessary for gaining insight into the principles underlying cross-modal interactions. None of the existing models of multimodal integration or cue combination is capable of accounting for the sound-induced illusory flash data. We have recently developed a generative statistical model that can account for the flash illusion data as well as other previous multi-modal integration phenomena. The model is based on Bayesian inference, and offers significant advantages over the previous models in that it is a predictive model (as opposed to a data-fitting model), and it can account for both conditions in which the signals of two modalities get integrated (fused) as well as those in which integration does not occur.