Author ORCID Identifier

https://orcid.org/0000-0001-5608-5224

Date of Award

Spring 5-2026

Document Type

Thesis (Ph.D.)

Department or Program

Engineering Sciences

First Advisor

Dr. Eugene Santos, Jr.

Abstract

The \emph{Markov decision process} (MDP) has long served as the canonical model for sequential decision-making. However, it assumes that the reward function is Markov with respect to a given state representation---an assumption that often does not hold in practice. Instead, agents typically only perceive streams of observations and actions and must infer the latent structure according to which reward unfolds over time. From this perspective, reward prediction is initially non-Markov, reflecting a mismatch between the agent's current representation and the underlying structure of the environment.

In this thesis, we advance the view that reward is not simply a signal to be optimized within a fixed representation, but an organizing principle through which representation itself is learned, thereby enabling the subsequent optimization of behavior. To formalize this perspective, we introduce the \emph{abstract reward Markov decision process} (ARMDP). In an ARMDP, the reward function is not specified a priori but instead learned from scalar feedback and encoded as a reward automaton whose states capture the reward-relevant structure of the interaction history. Decoupled from the underlying environment dynamics, this representation forms a reusable world model of reward whose step-wise predictions of scalar feedback signal immediate misalignment with the true latent reward structure, in contrast to representations learned from long-horizon return, where such signals are confounded by slow credit assignment. We show how agents can acquire such models through an active-learning process that iteratively proposes, tests, and refines hypotheses through interaction with the environment. We call this process \emph{reward representation learning} (RRL).

The thesis also addresses a deeper fundamental question usually left implicit in the MDP: what does the reward scalar mean? To answer this, we develop an information-theoretic interpretation of reward as an \emph{information residual}, defined by the difference between an agent's present certainty and its anticipated certainty horizon. This yields a probabilistic semantics in which reward has a fixed scale measured in nats, and motivates a horizon-sensitive \emph{behavioral cloning} (BC) algorithm that outperforms standard baselines.

Taken together, this thesis presents a coherent account of reward which we apply in \emph{reinforcement learning} (RL) and \emph{inverse reinforcement learning} (IRL) settings, enabling efficient solutions beyond the standard MDP formalism.

Download

Share

COinS