Date of Award
Spring 6-4-2025
Document Type
Thesis (Undergraduate)
Department
Computer Science
First Advisor
Nicholas C. Jacobson
Second Advisor
Andrew Campbell
Abstract
As wearable sensing becomes increasingly pervasive, a key challenge remains: how can we generate natural language summaries from raw physiological signals such as actigraphy - minute-level movement data collected via accelerometers? In this work, we introduce MotionTeller, a generative framework that natively integrates minute-level wearable activity data with large language models (LLMs). MotionTeller combines a pretrained actigraphy encoder with a lightweight projection module that maps behavioral embeddings into the token space of a frozen decoder-only LLM, enabling free-text, autoregressive generation of daily behavioral summaries.
We construct a novel dataset of 54,383 ⟨actigraphy, text⟩ pairs derived from real-world NHANES recordings, and train the model using cross-entropy loss with supervision only on the language tokens. MotionTeller achieves high semantic fidelity (BERTScore-F1 = 0.924) and lexical accuracy (ROUGE-1 = 0.722), outperforming prompt-based baselines by 7% in ROUGE-1. The average training loss converges to 0.38 by epoch 15, indicating stable optimization. Qualitative analysis confirms that MotionTeller captures circadian structure and behavioral transitions, while PCA plots reveal enhanced cluster alignment in embedding space post-training. Together, these results position MotionTeller as a scalable, interpretable system for transforming wearable sensor data into fluent, human-centered descriptions, introducing new pathways for behavioral monitoring, clinical review, and personalized health interventions.
Recommended Citation
Zhang, Aiwei; Pillai, Arvind; Campbell, Andrew; and Jacobson, Nicholas C., "MotionTeller: Multi-modal Integration of Wearable Time-Series with LLMs for Health and Behavioral Understanding" (2025). Computer Science Senior Theses. 77.
https://digitalcommons.dartmouth.edu/cs_senior_theses/77
