Author ORCID Identifier
Date of Award
Spring 6-11-2023
Document Type
Thesis (Undergraduate)
Department
Cognitive Science
First Advisor
Mark Thornton
Second Advisor
Landry Bulls
Abstract
With several modes of expression, such as facial expressions, body language, and speech working together to convey meaning, social communication is rich in redundancy. While typically relegated to signal preservation, this study investigates the role of cross-modal redundancies in establishing performance context, focusing on unaided, solo performances. Drawing on information theory, I operationalize redundancy as predictability and use an array of machine learning models to featurize speakers' facial expressions, body poses, movement speeds, acoustic features, and spoken language from 24 TEDTalks and 16 episodes of Comedy Central Stand-Up Presents. This analysis demonstrates that it is possible to distinguish between these performance types based on cross-modal predictions, while also highlighting the significant amount of prediction supported by the signals’ synchrony across modalities. Further research is needed to unravel the complexities of redundancy's place in social communication, paving the way for more effective and engaging communication strategies.
Recommended Citation
Dormes, Brandon Javier, "Say That Again: The role of multimodal redundancy in communication and context" (2023). Cognitive Science Senior Theses. 3.
https://digitalcommons.dartmouth.edu/cognitive-science_senior_theses/3
Included in
Cognitive Science Commons, Data Science Commons, Social Psychology Commons, Theory and Algorithms Commons
