Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs
Vishal Pramanik, Maisha Maliha, Nathaniel D. Bastian, Sumit Kumar Jha
Recommendation Score
Research context
Topics
Alignment & Safety
Paper type
Benchmark
Best for
Useful for both
arXiv categories
Why It Matters
HETA introduces the first Hessian-based attribution method for autoregressive LLMs, capturing non-linear causal dependencies in token generation—essential for building reliable, interpretable generative systems in production.
Abstract
Attribution methods seek to explain language model predictions by quantifying the contribution of input tokens to generated outputs. However, most existing techniques are designed for encoder-based architectures and rely on linear approximations that fail to capture the causal and semantic complexities of autoregressive generation in decoder-only models. To address these limitations, we propose Hessian-Enhanced Token Attribution (HETA), a novel attribution framework tailored for decoder-only language models. HETA combines three complementary components: a semantic transition vector that captures token-to-token influence across layers, Hessian-based sensitivity scores that model second-order effects, and KL divergence to measure information loss when tokens are masked. This unified design produces context-aware, causally faithful, and semantically grounded attributions. Additionally, we introduce a curated benchmark dataset for systematically evaluating attribution quality in generative settings. Empirical evaluations across multiple models and datasets demonstrate that HETA consistently outperforms existing methods in attribution faithfulness and alignment with human annotations, establishing a new standard for interpretability in autoregressive language models.