Long context windows in large language models (LLMs) increasingly strain computational resources, as accumulated tokens from documents, conversation histories, and reasoning processes grow. Traditional compression methods either reduce model accuracy or fail to speed up real-world applications effectively. A research collaboration from NYU, Columbia, Princeton, University of Maryland, Harvard, and Lawrence Livermore National Laboratory introduces Latent Context Language Models (LCLMs), a breakthrough encoder-decoder design that compresses the input context before decoding, significantly cutting memory and compute demands. LCLMs achieve up to 16x compression, making inference nearly 9 times faster on long-context benchmarks while maintaining competitive accuracy compared to existing methods. The architecture combines a lightweight encoder with a larger decoder, trained on a vast dataset with a novel mix of continual pre-training, fine-tuning, and reconstruction tasks to balance compression and performance. Integrated easily into existing pipelines, LCLMs allow models to handle much longer contexts more efficiently and cost-effectively. This advance offers enterprises a powerful tool to manage growing context lengths without sacrificing speed or accuracy, though careful tuning for retrieval-augmented generation systems remains necessary. The models and code are openly shared on HuggingFace and GitHub.
Back
