Londeree Technologies
Unlocking In-Context Learning at Scale
Artificial intelligence has taken remarkable strides, with models like generative pre-trained transformers (GPTs) demonstrating emergent behaviors that were once unimaginable. Among these is in-context learning—the ability to absorb new skills or facts within the scope of an interaction, simply by being shown directions or examples in a prompt. This powerful capability hints at the potential for AI to adapt and generalize in real time. Yet in-context learning today is transient; every new interaction discards the insights gained by resetting the model’s context. To move beyond this limitation and unlock a fundamentally new approach to model optimization, infinite context models are the key.
Mission
Our mission is to build scalable AI systems powered by infinite-context models, enabling real-time adaptation through continuous learning and distributed in-context learning across clusters.
Infinite-Context Models
Current AI models struggle with processing long sequences due to computational and memory constraints. While attention-based systems like transformers excel at handling limited (though large) context windows, their computational and memory requirements grow quadratically with input length, making them unsuitable for tasks requiring vast or indefinite context. This limitation hinders the ability of AI systems to process large documents, rich media streams, or high-bandwidth sensor data in a single pass.
Infinite-context models aim to overcome these constraints by providing a scalable approach to sequence processing. Such models would enable efficient operation over arbitrarily long sequences without degrading performance or overwhelming computational resources. This creates opportunities for a wide range of applications, including persistent agents with a knowledge of their history with a user, understanding entire datasets in context, processing real-time video or audio streams, and integrating information from complex multimodal environments. The ability to seamlessly operate over extended or even infinite horizons represents a paradigm shift in AI capability, allowing models to maintain coherence and relevance across far larger spans of data than is currently possible.
Beyond large-scale sequence processing, infinite-context models also lay the groundwork for unlocking the full potential of in-context learning at scale. This means that we can gain the benefits of in-context learning and leverage it as a new pre-training mechanism.
In-Context Learning
Unlike traditional training paradigms, which rely on gradient-based updates to permanently alter a model’s parameters, in-context learning enables models to operate by memorizing information to adapt to what is presented in context. Persistent in-context learning, made possible by architectures with long or infinite context windows, takes this concept further by enabling models to retain a functional memory of prior interactions, effectively bridging the gap between adaptability and online learning.
The implications of persistent learning extend far beyond simple memorization. For instance, models equipped with episodic memory could identify what they know and, just as importantly, what they do not know, improving their capacity for self-assessment and decision-making. With carefully designed training processes, in-context learning could provide mechanisms to evaluate the trustworthiness of information, a critical step toward aligning AI systems with human values and the elimination of hallucination. Persistent learning also makes the process of crafting AI systems more akin to onboarding a teammate: instead of curating datasets and retraining models, developers and users could interact with them directly to impart new knowledge or skills, vastly improving usability and deployment efficiency.
In-context learning is not just an emergent behavior of modern AI but a capability with profound implications for how we train, deploy, and interact with models. By operating entirely within the forward pass, in-context learning eliminates the need for backpropagation during adaptation, enabling models to learn and apply new information with a fraction of the computational overhead. Additionally, because in-context learning occurs without gradient based parameter updates, a single exposure to relevant data or instructions can lead to immediate and effective adaptation.
Another feature is its ability to leverage input modalities as a source of training information, bypassing the need for explicit labels. For instance, consider a model trained to perform OCR. During training, the model learns a mapping from image to text, but it does not internalize the information contained in the documents themselves. However, during operation, in-context learning allows the model to learn this information implicitly—because it is able to recite the document’s contents. This capability opens the door to pretraining models on modalities that they were never trained to generate.
We’re Making this Possible
Current AI models face significant limitations due to the quadratic scaling of attention mechanisms, which restricts their ability to process long sequences efficiently. These models require memory and computation that grow exponentially with input size, forcing them to truncate or summarize inputs—losing critical information along the way. To address this, we’ve identified key criteria that any model must meet to qualify as an infinite-context model capable of enabling in-context learning at scale. Among the architectures evaluated, a subclass of linear transformers has emerged as a promising candidate due to its ability to process sequences efficiently without the quadratic overhead of attention. However, while linear transformers address many core challenges, they still fall short in critical areas we aim to overcome through ongoing research and development. In upcoming articles, we will explore these challenges in depth, outlining how we are designing scalable memory mechanisms, optimizing large-scale vector operations, and pioneering parallelization strategies for recurrent systems—all essential components of true infinite-context models.
Looking Ahead
In the coming months, we’ll share insights into the technical breakthroughs driving our work, including the design requirements for infinite context models, the development of core memory mechanisms, strategies for handling large vectors and matrices, and innovations in parallelizing recurrent systems. These articles will provide a deep look at how we are addressing the engineering and research challenges that make this vision possible.
We invite you to join us on this journey. Whether you’re a researcher, developer, investor, or simply someone passionate about the future of AI, there are countless ways to contribute and collaborate. Together, we can build the systems that will redefine how machines learn, adapt, and apply knowledge in the real world.
Our mission is clear: to create AI systems that push the boundaries of scalability and capability, unlocking infinite-context learning to transform the field. We aim to bring about a future where AI is not only smarter but also more adaptive, efficient, and persistent. The path ahead is challenging, but with each step, we move closer to realizing a new era of artificial intelligence—one where the possibilities are truly limitless.