How Much Memory Does Your Agent Actually Need?

A year after tool-using agents became the default way to build on top of large language models, one question still gets answered by habit rather than measurement: how much memory should the agent keep? Many teams start with a one-size-fits-all decision — "give it the whole conversation" or "store everything in a vector index" — and only discover the cost after the context window starts dictating the agent's behavior. Memory is not a resource you add late in the project; it is a budget you allocate from the first prompt. The practical question is not whether your agent has memory, but how much of each kind it needs, and how you would know.

The Memory Pyramid: Why Capacity Is Not Uniform

The word "memory" hides a pyramid of different retention systems. At the bottom sits the raw transcript: every user message, tool call, and model reply since the agent started. Above that sits task state — the small set of variables, intermediate plans, and tool outputs that define what the agent is currently doing. Higher still are long-term facts: user preferences, repository conventions, API quirks learned in previous sessions. Finally, there is the model's own context window, which is not memory at all but the short-lived workspace where everything else competes for space.

These layers behave differently under load. The transcript grows linearly with each step and often contains duplicates — every retry of a tool call re-sends the previous error message. Task state grows with the complexity of the current goal but shrinks when the goal completes. Long-term facts grow slowly and, once written, tend to stay relevant. Confusing these layers is the most common cause of memory over-allocation. A team that sees a bloated context usually assumes it needs a bigger external store, when in fact it needs to stop replaying old tool outputs.

The only honest way to know which layer is consuming your agent's budget is to instrument the agent before you optimize it. That is the discipline this article walks through: a minimal, dependency-lean measurement harness that tells you where your memory budget is actually going.

Four Loads on the Same Budget

Every token in your agent's context window belongs to one of four loads. Understanding the split is the difference between guessing and engineering.

Conversational load is the recent back-and-forth between the user and the agent. It feels essential, but most of it is not. Once the user's request has been understood, the exact wording of earlier clarifying questions rarely needs to be replayed verbatim.

Tool-output load is the serialized result of every function the agent called. This is the most common hidden bloater. JSON payloads from APIs are verbose, nested, and repetitive. A single tool call that returns 40 records can consume more tokens than the rest of the conversation combined, and the agent often calls the same tool twice with slightly different parameters.

Episodic load is the agent's memory of what it already tried and what the outcome was. Some of it is necessary — you do not want the agent to repeat a failed approach. Most of it, however, can be compressed into a single line: "tried direct SSH, failed with permissions error, switched to API."

Contextual load is everything else the model needs to behave correctly: the system prompt, tool definitions, few-shot examples, and formatting instructions. This load is constant. It does not grow with the session, but it reduces the space available for the other three.

When practitioners ask "how much memory does my agent need?", they are usually asking about the sum of these loads. The useful answer is a breakdown, not a total.

What IBM Research's ALTK-Evolve Tells Us

The question is not merely a playground concern. On 18 August 2026, IBM Research published a post titled "How Much Memory Does Your Agent Actually Need?" on the Hugging Face blog, under the ALTK-Evolve initiative. The title alone is notable: an industry research group explicitly asked how much memory matters, rather than assuming that more is better.

I will not summarize the blog post claim-by-claim here, because the purpose of this article is to give you a practical instrument, not a recap. The relevant takeaway, as I read it, is that memory requirements are not static. An agent that unlocks new tools or gains access to new repositories will change the ratio of tool-output load to conversational load. A memory configuration that is generous at deployment time becomes wasteful after the agent's tooling evolves. The project name "ALTK-Evolve" suggests exactly this evolutionary view: memory should be tuned continuously, not set once.

If you treat memory sizing as a measurement problem, you can make any agent — regardless of framework — justify its memory footprint with numbers. That is what the rest of this article shows you how to do.

Requirements

Before you can measure your agent's memory, you need four things:

1. Access to the agent's request flow — you must be able to log each exchange between the agent, its tools, and the model. In practice, this means a small wrapper around your existing agent loop, or a logging hook if your framework provides one.
2. A way to estimate token counts — your LLM provider's SDK usually exposes usage data in the response object. If not, use a rough heuristic: count characters and divide by four for English text, or use your provider's tokenizer.
3. A Python environment — the measurement tool below uses only the standard library plus one small validation package, so it works in any Python 3.9+ setup.
4. Representative workloads — measure across real tasks, not synthetic ones. A memory audit of a question-answering agent will look nothing like an audit of a coding agent.

You do not need a vector database, a dedicated observability platform, or a change of framework. You need a notebook and a sense of what each number means.

Step-by-step installation: build a memory meter

The tool we will build is deliberately small: a logger that records each memory-related event in the agent's lifecycle and prints a summary of where tokens went. It does not replace your agent's existing memory; it watches it.

First, create a dedicated directory so the measurement tool stays isolated from the agent you are auditing.

mkdir agent-memory-meter && cd agent-memory-meter

Next, create a virtual environment to keep the Python dependencies from interfering with the system interpreter.

python -m venv .venv

Activate the environment. On macOS and Linux, use the source command.

source .venv/bin/activate

If you are on Windows, the command differs slightly; use the activation script inside the Scripts folder instead. Once the environment is active, install a single dependency for data validation.

pip install pydantic

Now create the measurement script. It defines a MemoryEvent record and a logger that aggregates the events.

# How Much Memory Does Your Agent Actually Need?
import json
import time
from pydantic import BaseModel


class MemoryEvent(BaseModel):
    timestamp: float
    step: int
    kind: str            # "user", "tool_output", or "assistant"
    tokens_in: int       # approximate prompt tokens consumed by this event
    tokens_out: int      # approximate completion tokens produced
    payload_chars: int   # length of the raw payload in characters


class MemoryLogger:
    def __init__(self):
        self.events = []

    def record(self, step, kind, tokens_in, tokens_out, payload):
        self.events.append(MemoryEvent(
            timestamp=time.time(),
            step=step,
            kind=kind,
            tokens_in=tokens_in,
            tokens_out=tokens_out,
            payload_chars=len(payload),
        ))

    def summary(self):
        total_prompt = sum(e.tokens_in for e in self.events)
        total_completion = sum(e.tokens_out for e in self.events)
        total_chars = sum(e.payload_chars for e in self.events)
        return {
            "events": len(self.events),
            "total_prompt_tokens": total_prompt,
            "total_completion_tokens": total_completion,
            "payload_kb": round(total_chars / 1024, 2),
        }


def main():
    log = MemoryLogger()
    log.record(1, "user", 120, 0, "List all open issues assigned to me")
    log.record(2, "tool_output", 0, 0,
               '{"issues": [{"id": 12, "title": "fix login", "assignee": "ada"}]}')
    log.record(3, "assistant", 0, 240,
               "There is one open issue assigned to you: fix login.")
    print(json.dumps(log.summary(), indent=2))


if __name__ == "__main__":
    main()

The script logs three kinds of events: a user message, a tool output, and an assistant reply. In your real agent, you would call log.record at each of those points, using the token counts returned by your model provider. Run the script to confirm the tool works.

python memmeter.py

You should see a JSON summary with event counts, token totals, and the payload size in kilobytes. The numbers will be small for this toy example; the value comes when you wire the logger into a real agent.

Usage examples: reading the numbers

Once the logger is attached to your agent, let it run on a representative set of tasks. After a few sessions, inspect the summary and ask four questions.

How large is the tool-output load compared to the conversational load? If tool outputs dominate, your agent is spending its memory budget on raw API responses. The fix is not a bigger context window; it is narrower tool schemas or a summarization step that condenses a tool result into its relevant fields before it re-enters the conversation. Your measurement tells you, in kilobytes and tokens, whether you are solving the right problem.

How many events reference the same fact? The logger records step numbers, so you can spot repeated tool calls with nearly identical payloads. If the same issue list is fetched three times with different filter parameters, the agent's memory is being filled with near-duplicates. Consider caching tool results by arguments, so the memory only holds the first response and the cache key.

Is the completion output large? Large assistant responses are often re-sent verbatim in later steps of the conversation. If completion tokens dominate the summary, your agent is talking to itself more than it is talking to the user. Trim chain-of-thought outputs before they are appended to the transcript.

How much of the context window would these totals consume? Compare the measured totals against the context limit of the model you deploy. If you are at 10% of the limit, you have headroom and should not add an external memory store yet. If you are at 90%, the fix is reduction before expansion — compress the tool outputs first, and only then consider a long-term store for facts that survive across sessions.

A useful complementary check is the compression ratio: divide the sum of payload_chars by the total token count. If the ratio is high — many characters per token — your payloads are verbose and highly compressible. If it is low, your memory contains dense content that summarization would damage, and you should instead stop some content from entering memory at all.

Why the Default Answer Is Usually Wrong

The default answer to "how much memory does my agent need?" is usually phrased as a multiple of the context window: "enough to hold the whole session," or "a vector store with everything." Both answers avoid the real question, which is about the composition of the memory, not its volume.

Ambient context — the system prompt and tool definitions — is cheap to reason about because it is constant. Session memory is where the waste hides. An agent that calls a search tool, receives 600 lines of JSON, and then reasons about two lines from that result is effectively paying for a memory much larger than the information it uses. The alternative is not to give it more memory, but to give it less irrelevant memory, measured with a tool like the one above.

This is also why the ALTK-Evolve perspective matters. As the agent evolves — new tools, new data sources, new user habits — the memory profile changes. A configuration tuned for a two-tool agent will be wrong for a ten-tool agent. The only sustainable practice is to keep the meter attached, run it periodically, and let the numbers drive the allocation.

Conclusion

Your agent's memory requirement is not a single number you can look up or benchmark once. It is a distribution across conversation, tool output, episodic history, and constant context — and that distribution shifts as the agent changes. The practical answer to "how much memory does your agent actually need?" is: enough to complete the task without re-fetching the same data, and no more. The only way to find that point is to measure.

The lightweight logger presented here gives you a baseline in a few minutes. Attach it to your agent, collect data from real workloads, and act on the breakdown: compress tool outputs, deduplicate repeated events, and reduce before you expand. As IBM Research's question on the Hugging Face blog suggests, memory deserves the same empirical treatment as every other part of the system. Treat it as a budget to be measured, not a feature to be maximized, and your agent will run faster, cost less, and reason about what actually matters.