Author ORCID Identifier
Date of Award
Summer 6-14-2026
Document Type
Thesis (Undergraduate)
Department
Computer Science
First Advisor
Soroush Vosoughi
Second Advisor
Yaoqing Yang
Third Advisor
Adam Breuer
Abstract
Large language models (LLMs) are increasingly deployed in high-stakes scenarios, yet their tendency to encode and replicate human-like social biases threatens their safe and fair application in these settings. Current approaches for measuring these biases in LLMs often rely on large, manually annotated datasets; however, these methods often consider a limited set of languages and cultural contexts and are operationalized in ways that limit the granularity and robustness of reported metrics. This thesis aims to expand the scope of bias benchmarking along three dimensions. Firstly, we develop a human-LLM collaborative annotation framework to more efficiently construct stereotype examination datasets for low-resource languages and cultures. We apply this framework to construct EspanStereo, a stereotype examination dataset of 2,690 instances specific to five countries in the Spanish-speaking world. Secondly, we show that preprocessing-based debiasing techniques can redistribute, rather than eliminate, encoded biases in LLMs in ways that are frequently invisible to current benchmarks. Finally, we develop a semantics-based debiasing method with an associated gold-standard benchmark that offers a more granular means of stereotype examination than fixed-sequence benchmarks.
Recommended Citation
Guerrerio, John J., "Towards Robust and Holistic Bias Benchmarking for Large Language Models" (2026). Computer Science Senior Theses. 62.
https://digitalcommons.dartmouth.edu/cs_senior_theses/62
