Date of Award

Spring 6-3-2026

Document Type

Thesis (Undergraduate)

Department

Computer Science

First Advisor

Dr. SouYoung Jin

Abstract

This thesis investigates the computational challenges of constructing a database for affective incongruity, exploring the difficulties in automating the collection of contradictory affective states. Capturing these incongruities is essential for advancing Vision-Language Models (VLMs) and sentiment analysis, which struggle to interpret signals deviating from basic emotional archetypes. Curating non-congruent affect provides the data necessary for models to navigate complex social contexts, which is critical for applications like automated Audio Description (AD) for the visually impaired and nuanced Human-Computer Interaction (HCI). To capture these signals, two distinct methodologies were employed. The first utilized a tripartite decomposition of video data, isolating textual, visual, and auditory streams to identify incongruity through statistical divergence across modalities. The second methodology utilized a dual-model VLM architecture: Qwen 2.5 derived a holistic understanding of the video content, while Gemini analyzed audience perception within user comments. By prompting models to reason through discrepancies between video content and viewer reactions, the system attempted to identify incongruity via social context. Both methods failed to produce a viable database. Method I proved overly sensitive to non-affective environmental noise, such as incidental audio-visual variance, while Method II was hampered by the high variance of comment data and VLM limitations in detecting non-explicit emotional cues. By documenting these null results, this thesis provides a post-mortem of automated affect detection and maps the specific technical barriers that prevent the quantification of complex human emotion.

Download

Share

COinS