Date of Award
Spring 5-8-2025
Document Type
Thesis (Master's)
Department or Program
Computer Science
First Advisor
Soroush Vosoughi
Second Advisor
Lorie Loeb
Third Advisor
John Bell
Abstract
This thesis presents a method for teaching large language models (LLMs) to perform embodied multi-task planning using simulation data and a lightweight training technique called proxy-tuning. Traditional LLMs are strong in language understanding but struggle with tasks that require reasoning and planning in real-world environments. To address this, we construct a dataset based on VirtualHome, a simulator that models household activities using input commands. We extract individual tasks from the simulator and combine them into multi-task plans, optimizing for coherence and efficiency. These plans are formatted as Q&A pairs to train models to generate step-by-step instructions that complete multiple tasks at once. Instead of directly fine-tuning large models, which is computationally expensive, we first fine-tune smaller models and then transfer their learned behavior to a larger model using proxy-tuning. This technique adjusts the output of the large model based on the differences between a fine-tuned and an unfine-tuned smaller model, without changing the large model’s parameters.
Recommended Citation
Wang, Sirui, "Growing Through Simulated Realities: Proxy-Tuning LLMs with Embodied Multi-task planning" (2025). Dartmouth College Master’s Theses. 220.
https://digitalcommons.dartmouth.edu/masters_theses/220
