Thinking of ACE? We Can Do It with Fewer Tokens

Thinking of ACE? We Can Do It with Fewer Tokens

When we design an AI agent, the question that dominates every review is the same: "Will it fit?" Will the system prompt fit? Will the retrieved evidence fit? Will the twelve intermediate reasoning steps fit? Usually, the answer we choose is to buy a bigger context window. But there is a quieter, often better alternative: teach the agent to spend fewer tokens in the first place.

That alternative is the subject of "Thinking of ACE? We Can Do It with Fewer Tokens," a technical post published by IBM Research on the Hugging Face blog on 2026-08-11 at https://huggingface.co/blog/ibm-research/altk-evolve-sldd. The post argues that agentic context engineering — the discipline of managing what an agent reads and writes — does not have to be synonymous with ever-larger context budgets. This article is a practical companion to that idea. It explains why token efficiency is the backbone of reliable agents, and it walks through a small, concrete toolkit for measuring and shrinking your own agent's token consumption.

Before going further, a note on scope. The verified facts about the source are its title, its publication date, and its URL. The engineering interpretation that follows is my own, and the code examples are a generic setup rather than excerpts from the post.

Why Context Engineering Is a Token Problem, Not a Memory Problem

An agent is a loop. It reads, reasons, calls tools, reads the results, reasons again, and repeats until it produces an answer. Each pass through the loop adds tokens to the next pass:

• The system prompt is paid for on every turn.
• Retrieved documents are usually injected in full.
• Tool outputs are often returned verbatim, no matter how large.
• The agent's own chain-of-thought is saved and replayed so that it can stay coherent.

The result is that a simple task with three tool calls can produce tens of thousands of tokens of context even when the useful information could fit in a few hundred. That is not a memory problem — the model can remember all of it. It is a budget problem. Every extra token adds cost, latency, and noise. And noise is not harmless: in practice, a model is often much worse at extracting a single relevant fact when that fact is buried in irrelevant text. Engineers sometimes call this the "lost in the middle" effect, and it gets worse as contexts grow.

Agentic Context Engineering (ACE), as the name suggests, treats context as something we design and maintain, not something that simply accumulates. A well-engineered agent context is compact, current, and sufficient. It contains only what the next step needs, not everything the agent has ever seen. This is where the IBM Research post makes its case: with disciplined context engineering, you can run agentic workloads with far fewer tokens — without surrendering task quality. The title of the post is the thesis: "We can do it with fewer tokens."

The Core Playbook: Six Habits of Token-Efficient Agents

The following principles are a generic synthesis of ACE best practices. They are consistent with the direction of the source post, but they are not a summary of its internal details.

1. Prefer structured state over raw history. Instead of replaying the full transcript at every step, maintain a compact status block: what the agent knows, what it has tried, what failed, and what comes next.

1. Trim tool outputs at the boundary. The tool returns what it returns; the context does not have to contain all of it. Cut the payload before it ever enters the conversation.

1. Summarize before you persist. Each time the agent completes a subtask, compress the outcome into a one-line ledger entry. Let the details die after they have served their purpose.

1. Retrieve narrowly and often. Instead of injecting a large corpus at the start of the session, fetch small chunks just in time, close to the moment they are needed.

1. Budget the system prompt. Every sentence in the prompt competes with working memory. If a rule can be shortened without losing precision, shorten it. If an example is redundant, delete it.

1. Measure everything. You cannot manage token consumption you cannot see. Count tokens per step, per loop, and per session until the habit becomes automatic.

These habits are cheap to implement and immediately effective. The rest of this article shows the concrete mechanics.

Requirements

To follow the examples in this article, you need:

• Python 3.10 or newer.
• A virtual environment tool such as venv.
• The Hugging Face libraries transformers and datasets.
• Optionally, tiktoken for token counting with OpenAI-compatible models.
• An API key only if you plan to test against a hosted model; all of the examples run locally.

The setup deliberately uses open, local tools. Token measurement does not require a GPU, and you can run every example on a laptop during a coffee break.

Step-by-Step Installation

First, create an isolated virtual environment so that the packages we install do not interfere with the rest of your system:

python -m venv ace-env

This creates a folder named ace-env containing its own Python binary and library directory.

Activate the environment. The command differs by operating system; on Linux and macOS:

source ace-env/bin/activate

On Windows:

ace-env\Scripts\activate

After activation, your shell prompt should show (ace-env) at the beginning.

Upgrade pip to the latest version so that dependency resolution behaves well:

pip install --upgrade pip

Install the core packages. transformers gives us tokenizers and model utilities, while datasets is handy for loading small evaluation corpora:

pip install transformers datasets

If you work with OpenAI-style APIs and want to count tokens using the same tokenizer family the API uses, install tiktoken as well:

pip install tiktoken

Verify the installation by importing the libraries and printing their versions:

python -c "import transformers; import datasets; print('transformers', transformers.__version__); print('datasets', datasets.__version__)"

If you installed tiktoken, verify it the same way:

python -c "import tiktoken; print('tiktoken', tiktoken.__version__)"

The setup is complete. There is no server to start and no model to download; token counting works entirely offline.

Usage Examples

1. Count tokens before you design

The first habit is measurement. Before you can decide whether a context is bloated, you need a reliable way to count the tokens in your prompts and tool outputs. The following function uses a Hugging Face tokenizer:

from transformers import AutoTokenizer

def count_tokens(text: str, model_id: str = "gpt2") -> int:
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    return len(tokenizer.encode(text, add_special_tokens=False))

example = "RESULT: invoice_2024_05_221.pdf contains 4 line items totaling $12,500."
print(count_tokens(example))

The gpt2 tokenizer is a commonly used local stand-in for many modern models, and it runs without an API key. If your target model is known, replace model_id with that model's tokenizer name.

2. Trim tool outputs at the boundary

When a tool returns a large JSON blob, it is tempting to inject the whole thing. A simple trimming function keeps the head and tail, where important metadata usually lives, and removes the middle:

def trim_tool_output(output: str, keep: int = 400) -> str:
    if len(output) <= keep:
        return output
    half = keep // 2
    return (
        output[:half]
        + f"\n...[trimmed {len(output) - keep} characters]...\n"
        + output[-half:]
    )

large_http_body = '{"status": "ok", "items": [' + ','.join(f'{{"id": {i}}}' for i in range(1000)) + ']}'
print(trim_tool_output(large_http_body, keep=300))

This preserves the structure and the boundaries of the payload while discarding the repetitive middle section. In an agent loop, apply this function at the point where the tool result is inserted into the context, not later. Once the raw output has been trimmed, do not also keep the untrimmed version somewhere "just in case" — that defeats the purpose.

3. Replace conversation history with a status ledger

The most common token leak in agentic systems is the full transcript. Instead of storing every message, tool call, and reflection, maintain a short ledger that is updated after every step:

class AgentLedger:
    def __init__(self, max_entries: int = 5):
        self.entries = []
        self.max_entries = max_entries

    def record(self, step: int, action: str, conclusion: str) -> None:
        self.entries.append(f"step {step}: ran {action} -> {conclusion}")
        self.entries = self.entries[-self.max_entries:]

    def render(self) -> str:
        return "\n".join(self.entries)

ledger = AgentLedger(max_entries=3)
ledger.record(1, "search", "found 3 candidates")
ledger.record(2, "read_doc", "candidate A missing license")
ledger.record(3, "verify", "candidate B is valid")
ledger.record(4, "summarize", "ready to answer")
print(ledger.render())

The agent loses access to the raw reasoning of the first three steps, but it retains what matters: what it did and what it concluded. For many tasks, this is enough for coherent long-horizon behavior. The token cost of the ledger grows linearly with the number of entries — not with the length of the full transcript. This is the single most effective change you can make to an existing agent.

4. Build a compact system prompt

System prompts are paid for on every turn, so their token cost is multiplied by the number of loops the agent runs. A good prompt is precise: it states the agent's role, the format of each step, and the budget rules. A bad prompt buries those instructions in examples and caveats. A minimal template demonstrates the principle:

SYSTEM_PROMPT = """You are a research assistant.
Work in steps. After each step, write a single line:
[step N] <action> -> <conclusion>
Keep the ledger under 10 lines. Do not recap old steps."""

print(count_tokens(SYSTEM_PROMPT))

The exact wording will differ for your task, but the discipline is the same: if a sentence can be removed without changing observed behavior, remove it. When in doubt, run a small A/B test with and without the sentence and compare task success rates, not your intuition.

Putting It Together

When you combine these practices, the effect is cumulative. A typical agent run might look like this:

• The system prompt is 150 tokens instead of 800.
• Tool outputs are trimmed to 400 characters each instead of 5,000.
• The conversation history is a five-line ledger instead of a 40-turn transcript.
• Retrieval pulls three short chunks instead of ten long documents.

The same task now runs with an order of magnitude fewer tokens. Latency drops because input processing is faster. Cost drops because input pricing is multiplicative across turns. And in many cases, quality improves, because the model is no longer forced to search for a signal in a sea of noise.

There are limits. Some tasks genuinely require long documents in context, and aggressive compression can lose the exact wording the agent needs. Citation-heavy legal analysis, for instance, may demand verbatim source text. The goal is not zero tokens; it is the smallest context that reliably completes the task. That is the practical definition of ACE done right: not "how much can the model hold" but "how little does the task actually need."

A useful exercise is to take one of your existing agents, run it on a standard test case, and count the total tokens consumed end to end. Then apply the four examples above one at a time, re-running the test case after each change. You will quickly discover which step of your pipeline is the biggest consumer — and whether it was ever earning its place in the context.

Conclusion

Agentic context engineering is often framed as a problem of scale: bigger models, bigger windows, bigger budgets. The IBM Research post "Thinking of ACE? We Can Do It with Fewer Tokens" makes the opposite point, and the practical toolkit in this article follows the same direction. Token efficiency is not a performance optimization you apply after the agent is built. It is a design constraint you apply from the first prompt.

Start by measuring what your agent actually spends. Then trim what it does not need. Then replace raw history with structure. You will almost certainly find that the agent was carrying far more than it ever used — and that fewer tokens, used deliberately, are enough.