Contributions of the lexicon to audiovisual speech perception
Date of Completion
January 1998
Keywords
Language, Linguistics|Psychology, Experimental|Psychology, Cognitive
Degree
Ph.D.
Abstract
Two areas of research in speech perception address, respectively, the contributions of visual speech information and lexical context on phonetic categorization. However, there have been few attempts to study how these sources of information might interact. This investigation was an attempt to bridge the two domains by combining the "McGurk effect," in which observers perceive a consonant different from the one presented acoustically when it is dubbed with a discrepant video, with the "Ganong effect," in which an ambiguous segment is identified as a particular phoneme more often when it forms a word than when it forms a nonword. In the present experiments, audiovisually dubbed stimuli were created in which there was a discrepancy in the initial consonant. In different stimuli, both the auditorily-based and visually-influenced phonetic percepts formed words, both formed nonwords, or only one formed a word. In four experiments, involving speeded and unspeeded initial-consonant phonetic categorization, there were pronounced effects of lexical status; Experiments 1 and 2 revealed influences of both auditorily-based and visually-influenced lexical status on the incidence of the McGurk effect. Reaction-time analyses in Experiment 2 revealed a smaller lexical effect for fast than for slow responses. Goodness ratings of the percepts in Experiment 1 revealed that both auditorily-based and visually-influenced responses to audiovisually discrepant stimuli were perceived as poorer category exemplars than responses to congruent stimuli; video responses received higher ratings than audio responses. Experiments 3 and 4 modified the technique of Newman, Sawusch, & Luce (1997) to test whether there would be an influence on the McGurk effect due to nonword stimuli in which the two response alternatives differed in their neighborhood frequencies. There was no effect in Experiment 3, but Experiment 4 found a significant neighborhood influence on the McGurk effect, demonstrating that the effect is not due to post-perceptual response biases. The results are discussed in terms of cue-based and gesture-based theories of audiovisual integration, and in terms of autonomous and interactive models of lexical access. Potential audiovisual lexical access models, incorporating theories from both domains, are proposed. ^
Recommended Citation
Brancazio, Lawrence, "Contributions of the lexicon to audiovisual speech perception" (1998). Doctoral Dissertations. AAI9909779.
https://digitalcommons.lib.uconn.edu/dissertations/AAI9909779