LACE: Lattice Attention for Cross-thread Exploration
Yang Li, Zirui Zhang, Yang Liu, Chengzhi Mao
Recommendation Score
Research context
Topics
Fine-tuning & PEFT, LLM Reasoning
Paper type
System
Best for
Best for builders
arXiv categories
Why It Matters
LACE enables LLM reasoning paths to share insights via cross-thread attention, dramatically reducing redundant failures and improving solution quality—essential for building robust, scalable reasoning systems.
Abstract
Current large language models reason in isolation. Although it is common to sample multiple reasoning paths in parallel, these trajectories do not interact, and often fail in the same redundant ways. We introduce LACE, a framework that transforms reasoning from a collection of independent trials into a coordinated, parallel process. By repurposing the model architecture to enable cross-thread attention, LACE allows concurrent reasoning paths to share intermediate insights and correct one another during inference. A central challenge is the absence of natural training data that exhibits such collaborative behavior. We address this gap with a synthetic data pipeline that explicitly teaches models to communicate and error-correct across threads. Experiments show that this unified exploration substantially outperforms standard parallel search, improving reasoning accuracy by over 7 points. Our results suggest that large language models can be more effective when parallel reasoning paths are allowed to interact.