Date of Award

Spring 6-4-2026

Document Type

Thesis (Undergraduate)

Department

Computer Science

First Advisor

Nikhil Singh

Abstract

Large Language Models are increasingly deployed in systems that assume more information leads to better decisions: RAG pipelines inject retrieved documents, tool-augmented agents call external APIs, and multi-model architectures route signals between specialized components. Whether this holds when both the information and the outcomes are noisy, and decisions compound over time, has received limited empirical attention. This thesis tests that assumption on LLM-based trading agents. Using StockBench (Chen et al., 2025) as our benchmark, we ran 150 backtests across 2 frontier LLMs (Kimi K2 and Qwen3-235B-Instruct), 12 augmentation strategies, and 5 random seeds, over 82 trading days — a period that includes a tariff-driven crash and V-shaped recovery. Following StockBench, the agents traded the top 20 Dow Jones Industrial Average constituents on a daily basis. We injected forecasts from Kronos, a time-series foundation model, and sentiment analyses from FinGPT, a parameter-efficient large language model fine-tuned for financial text analysis. We then measured portfolio outcomes (cumulative returns, max drawdown, Sortino ratio) and analyzed the models’ reasoning traces to investigate how each model processed the injected signals. In our experiments, augmentation outcomes varied more with signal format and model identity than with signal accuracy: a Kronos forecast with 49.3% directional accuracy (not significantly different from random) hurt K2 by 2.94pp (p=0.016) and helped Qwen by 1.75pp (p=0.069). Two Kronos model sizes that disagree on 37.2% of predictions produced portfolios differing by only 0.40pp and 0.09pp in mean return. The 5.34pp spread between K2’s worst and best strategy appears to be associated with differences in delivery format, gating regime, and narrative context — not signal accuracy: the reasoning logs suggest numerical forecasts were associated with anchoring-like behavior, while narrative signals did not produce the same visible restructuring in the chain-of-thought traces. A conditional gate that withheld forecasts during high-volatility regimes was associated with a 3.12pp swing. Adding narrative context to the gated signal produced K2’s best result (+3.10%, positive on all 5 seeds). These are exploratory findings from two models and one market period; no full-period comparison survives multiple-comparison correction. However, the pattern is consistent across 150 runs, multiple metrics, and converging evidence. If these patterns replicate, they would suggest that in LLM trading systems, the receiver’s processing behavior may matter as much for outcomes as the quality of the injected context.

Download

Share

COinS