LLM Context Window Size: The New Moore’s Law?

At a recent panel on AI startups, I was asked by an aspiring entrepreneur the extent to which current LLM context window sizes were a limitation on certain LLM use cases and might constrain certain AI startups.

In layman’s terms, “context window size” (also referred to as “context length”) can be thought of as an LLM’s “short-term memory”.

In the case of a chat agent, it constrains the amount of prior conversation history that the chat agent can look back on to figure out what to say next. The smaller the context window, the more “forgetful” of earlier conversation the agent appears to be.

My answer to the question, which the panel was broadly unanimous on, is that the context window size is not a constraint to worry about.

Workarounds

Firstly, there are workarounds to the context window limit which are effective for most use cases.

Extending the above analogy, the common approach here is to shunt the things that need to be remembered to “long-term memory”, perhaps in the form of a traditional database or a vector database, and then call them back into the LLM’s “short-term memory” as needed. This definitely introduces some additional complexity in implementation but it works.

However, the bigger point is that context window size appears to be growing incredibly quickly so it is unlikely to be a constraint, other than in extreme edge cases.

By The Numbers

I wanted to get some specifics here so I collated a table of some of the most well-known LLMs with their release date and context window sizes.

It turns out that this data is surprisingly hard to collate. Firstly, it’s hard to find historical data on the earlier models. Secondly, popular models are frequently revised, and sub-versions released, with increases (and sometimes decreases) in context window size between sub-versions. Lastly, lightweight and/or faster versions of models (“turbo” versions for OpenAI) are released after the original versions, but can have smaller context windows, presumably to increase performance. (You can see the data table at the bottom below.)

With those caveats, here’s a scatter plot of context window size against release date for a bunch of well-known LLMs from Amazon, Anthropic, Google, Meta, OpenAI, and X.ai.

The first thing to note here is that the Y-axis is logarithmic.

Although there are gaps and variances between vendors, it appears that we’ve been seeing a more or less consistent growth over time.

Specifically, in the 6 years from 2019 to 2025, we’ve seen an increase in context window size of roughly 3 orders of magnitude - i.e. a ~1,000x increase.

A New Moore's Law?

In 1975, Gordon Moore, the co-founder of Fairchild Semiconductor and Intel, predicted that the number of components per integrated circuit would continue to double every two years. This prediction came to be known as “Moore’s Law” and has more or less held true until today.

Are we seeing a similar phenomenon now? Will this trend continue?

Is there a new “law” here whereby context window size will continue to grow by an order of magnitude (10x) every 2 years?

Why Does it Matter?

To recap, a larger context window means a longer “short-term memory”.

An example of a simple benefit of a larger context window is the ability to summarize larger amounts of text in a single invocation of the LLM, without having to segment up the data into chunks and then cross-correlate insights across those chunks.

However, in the broadest sense, the larger the context window, the more of an application’s state can be kept in the “short-term memory”. Ultimately, each user’s entire interaction from their very first touch can be held in the LLM’s context window, eliminating entirely the need for any “long-term memory” in the form of separate databases and systems-of-record.

The Cost

Of course, increases in context window size don’t come without costs.

As the amount of data sent to the LLM grows, the computational cost of executing the LLM also grows. Additionally, if we truly want to use LLM state as the source-of-truth and permanent record for an application, then we have to persist that LLM state indefinitely and securely.

But, we can probably bet that these issues will also diminish over time as we see improvements in model architecture, LLM algorithms, and, of course, Moore’s Law itself!

The Data

It’s interesting to note here that not all of the vendors are growing their context window at the same rate. Google is being most aggressive whereas, at the other end of the scale, Anthropic has been keeping context window size relatively flat.

I’m guessing this is a deliberate choice vs a limitation in capability. Different vendors have different philosophies on LLM usage and are optimizing for different use cases.