Context and Constraints

The first public release of ChatGPT had a context window of 2048 tokens. Magic claims its LTM-2-Mini Model has a context window of 100 million. In a world where more parameters means more performance, you’d think a larger context window should similarly lead to better performance.

Not necessarily.

Good design is the product of constraints. Constraints define the problem and drive the solution. This is as true in industrial design as it is in software architecture.

Design depends largely on constraints … [and] is one of the few effective keys to the Design problem: the ability of the Designer to recognize as many of the constraints as possible; his willingness and enthusiasm for working within these constraints. Constraints of price, of size, of strength, of balance, of surface, of time, and so forth. Each problem has its own peculiar list.

— Ray & Charles Eames, 1972

Restricted context windows represent one of the key constraints in agentic systems design. Too small and we fail to include essential information. Too large and we lose the signal of our intent amongst the noise. This not only increases our LLM costs but too much context also degrades model performance.

This constraint pushes us towards designing agentic systems as the orchestrations of sub-agents. The limited context window of today’s models forces us to divide systems into smaller, more focused components.

What appears to be a limitation turns out to be a strength. Smaller, more focused components can help reduce the negative effects of model non-determinism, limiting the instances of ‘hallucination’ and other undesirable outputs. Smaller, more focused components make modification of our systems easier and can sometimes help us identify scenarios where generative AI isn’t the best solution - when, for example, sub-tasks like classification are better handled by deterministic processes.

Limited context windows are features, not bugs. Embracing this can help us design better agentic systems.