Date of Award

Spring 6-5-2025

Document Type

Thesis (Undergraduate)

Department

Computer Science

First Advisor

Soroush Vosoughi

Abstract

This thesis addresses a key bottleneck in hallucination research: the scarcity and limitations of hallucination benchmark datasets. Existing datasets typically focus on a single type of hallucination and are expensive to produce due to the need for manual prompt creation and annotation. To overcome these challenges, we propose a novel mixture-of-experts (MoE) adversarial framework that actively induces hallucinations. Our framework employs three large language model (LLM) agents that iteratively and adversarially revise prompts to provoke hallucinated responses from a target question-answering model. It automates the generation of both intrinsic hallucinations (logical inconsistencies) and extrinsic hallucinations (inclusion of unverifiable external information). While experiments were conducted on prompts from the RAGTruth dataset, our framework is model-agnostic and topic-agnostic, enabling broad applicability to hallucination dataset generation. Using GPT-4o, we achieved a 96\% success rate in inducing intrinsic hallucinations and a 92\% rate for extrinsic ones. Smaller-scale experiments with other LLMs also achieved success rates above 70\%, demonstrating the framework’s robustness and generalizability. Comparisons with zero-shot prompting, which yielded significantly lower hallucination rates, underscore the necessity of our more complex methodology. Furthermore, our annotations of the hallucination-inducing strategies employed by the framework offer actionable insights for strategies in hallucination creation, detection, and mitigation. We observe that effective strategies differ across topics and models, highlighting an important direction for future research. Future research to investigate optimal prompt-editing techniques and to validate the framework’s robustness across broader datasets would be interesting to pursue to further refine our framework.

Download

Share

COinS