Microsoft AI Introduces Belief State Transformer (BST): Enhancing Goal-Conditioned Sequence Modeling with Bidirectional Context
Transformer models have transformed language modeling by enabling large-scale text generation with emergent properties. However, they struggle with tasks that require extensive planning. Researchers have explored modifications in architecture, objectives, and algorithms to improve their ability to achieve goals. Some approaches move beyond traditional left-to-right sequence modeling by incorporating bidirectional context, as seen in models trained on past and future information. Others attempt to optimize the generation order, such as latent-variable modeling or binary tree-based decoding, though left-to-right autoregressive methods often remain superior. A more recent approach involves jointly training a transformer for forward and backward decoding, enhancing the model’s ability to maintain compact belief states.
Further research has explored predicting multiple tokens simultaneously to improve efficiency. Some models have been designed to generate more than one token at a time, leading to faster and more robust text generation. Pretraining on multi-token prediction has been shown to enhance large-scale performance. Another key insight is that transformers encode belief states non-compactly within their residual stream. In contrast, state-space models offer more compact representations but come with trade-offs. For instance, certain training frameworks struggle with specific graph structures, revealing limitations in existing methodologies. These findings highlight ongoing efforts to refine transformer architectures for better structured and efficient sequence modeling.
Researchers from Microsoft Research, the University of Pennsylvania, UT Austin, and the University of Alberta introduced the Belief State Transformer (BST). This model enhances next-token prediction by considering both prefix and suffix contexts. Unlike standard transformers, BST encodes information bidirectionally, predicting the next token after the prefix and the previous token before the suffix. This approach improves performance on challenging tasks, such as goal-conditioned text generation and structured prediction problems like star graphs. By learning a compact belief state, BST outperforms conventional methods in sequence modeling, offering more efficient inference and stronger text representations, with promising implications for large-scale applications.
Unlike traditional next-token prediction models, the BST is designed to enhance sequence modeling by integrating both forward and backward encoders. It utilizes a forward encoder for prefixes and a backward encoder for suffixes, predicting the next and previous tokens. This approach prevents models from adopting shortcut strategies and improves long-term dependency learning. BST outperforms baselines in star graph navigation, where forward-only Transformers struggle. Ablations confirm that the belief state objective and backward encoder are essential for performance. During inference, BST omits the backward encoder, maintaining efficiency while ensuring goal-conditioned behavior.
Unlike forward-only and multi-token models, the BST effectively constructs a compact belief state. A belief state encodes all necessary information for future predictions. The BST learns such representations by jointly modeling prefixes and suffixes, enabling goal-conditioned text generation. Experiments using TinyStories show BST outperforms the Fill-in-the-Middle (FIM) model, producing more coherent and structured narratives. Evaluation with GPT-4 reveals BST’s superior storytelling ability, with clearer connections between prefix, generated text, and suffix. Additionally, BST excels in unconditional text generation by selecting sequences with high-likelihood endings, demonstrating its advantages over traditional next-token predictors.
In conclusion, the BST improves goal-conditioned next-token prediction by addressing the limitations of traditional forward-only models. It constructs a compact belief state, encoding all necessary information for future predictions. Unlike conventional transformers, BST predicts the next token for a prefix and the previous token for a suffix, making it more effective in complex tasks. Empirical results demonstrate its advantages in story writing, outperforming the Fill-in-the-Middle approach. While our experiments validate its performance on small-scale tasks, further research is needed to explore its scalability and applicability to broader goal-conditioned problems, enhancing efficiency and inference quality.
Check out the Paper. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 80k+ ML SubReddit.
Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.
