Date of Award

Spring 6-4-2026

Document Type

Thesis (Undergraduate)

Department

Computer Science

First Advisor

Nikhil Singh

Abstract

When assessing the extent to which someone truly understands the topics they discuss, you might ask them a series of questions that examine their knowledge from different angles. As Large Language Models (LLMs) are increasingly integrated into real-world tasks, they are often required to reason through ambiguous and uncertain scenarios that rarely have a single correct answer. In these settings, their usefulness is contingent on more than accuracy alone. Rather, it depends on the strength of the reasoning that underlies their responses. This begs the question: how might we extend these types of tests to LLMs? We formalize this informal test for LLMs by introducing a consistency framework built on two meaning- preserving transformations: commutativity (reordering the options) and a contrapositive- style reverse inference (swapping the premise and the chosen conclusion). We apply this to four LLMs across six everyday decision domains. Our results show that full-consistency rates vary widely (46–66%). Across every model, consistency rises monotonically with the perceived likelihood of the relationship between premise and choice. In other words, the more likely a relation is rated within a prompt, the more consistent the models’ responses to transformed inputs become. As the plausibility of a relation falls, consistency declines, and with it, the coherence of the LLM’s reasoning and world model. We interpret this as a signature of autoregressive training: low-frequency regions of the relational space are represented less coherently than high-frequency ones. Interestingly, we find that small, non- proprietary models only exhibit a significant increase in consistency when the relation is rated as likely or very likely, thereby treating relationships that are merely plausible or technically possible similarly to impossible relations. Such insights provide implications for trusting LLM judgment in long-tail or otherwise ambiguous social settings.

Download

Share

COinS