Building Reliable Data Analytics Agents: Lessons from the KDD Cup

Building Reliable Data Analytics Agents: Lessons from the KDD Cup

Data analytics agents sit in an awkward place in the current wave of AI systems. A chatbot that produces a plausible paragraph has succeeded if the paragraph is useful. An analytics agent that produces a plausible number has succeeded only if the number is correct — and correctness in data work is not a matter of style. It is a matter of joins, filters, types, nullability, and time zones.

The KDD Cup, the long-running data mining competition series associated with the SIGKDD community, is a useful frame for this problem. Competition tasks are adversarial in a very specific way: they hand you a dataset you did not design, a question you did not write, and a scoring function you cannot negotiate with. NVIDIA's AI Blog post, "Building Reliable Data Analytics Agents: Lessons from the KDD Cup," uses that setting to examine what it takes for an agent to be trusted with real analytical work rather than merely appear to do it.

This article takes the same theme — reliability as the core engineering problem — and turns it into a build guide: what to install, how to structure the pipeline, and how to test it. One clarification up front: the source is a vendor engineering write-up, and the implementation patterns below are general, reproducible practice motivated by that theme. They are not claims about the internal details of any specific system described there.

Why analytics agents fail differently

Most agent failures in conversational settings are annoying. Most agent failures in analytical settings are silent. An agent that hallucinates a column name usually gets an error. An agent that joins two tables on the wrong key often does not. It returns a number, the number looks reasonable, and the number is wrong.

Three failure modes dominate:

Schema drift and hallucination. The model assumes a column called user_id when the table has uid. Without introspection, the model fills gaps with plausible names.

Semantically wrong but syntactically valid queries. An inner join silently drops rows that a left join would have kept. A WHERE clause on a nullable column removes records the analyst expected to see.

Non-reproducibility. The same question asked twice produces two different queries and two different answers. There is no artifact to audit.

Every practice below exists to close one of these gaps. The unifying idea is simple: treat the agent as a pipeline with verifiable stages, not as a prompt that emits answers.

Requirements

To follow along you need:

• Python 3.11 or newer.
• A Unix-like shell (macOS, Linux, or WSL). The commands assume bash or zsh.
• Access to any instruction-following LLM through an HTTP API or a local server. The examples below are provider-neutral and refer to a model identifier you supply.
• Enough RAM to hold your working dataset, or a database engine that can spill to disk. No GPU is required for the scaffolding itself.

The stack is deliberately boring: Pandas for in-memory data, DuckDB for SQL over files without a server, Pydantic for typed tool contracts, Tenacity for retry logic, and pytest for regression tests.

Step-by-step installation

Start by creating an isolated environment so the agent's dependencies do not collide with your system Python.

python3 -m venv .venv

Activate it. On Windows, use .venv\Scripts\activate instead.

source .venv/bin/activate

Upgrade pip before installing anything, since older versions resolve dependencies less predictably.

python -m pip install --upgrade pip

Install the core dependencies. DuckDB runs analytical SQL directly against CSV and Parquet files, which removes an entire class of "the agent pointed at the wrong database" bugs during development.

pip install "pandas>=2.0" duckdb "pydantic>=2" pyyaml tenacity pytest openai

Create the project skeleton. Keeping tools, validation checks, and evaluations in separate directories makes it obvious where a new piece of logic belongs.

mkdir -p analytics_agent/tools analytics_agent/checks analytics_agent/evals data artifacts
touch analytics_agent/__init__.py

Write a configuration file rather than hardcoding model names and paths. This is the single change that most improves reproducibility, because every run is described by an artifact you can commit.

cat > config.yaml <<'YAML'
model:
  id: "<your-model-id>"
  temperature: 0
  max_repair_attempts: 3
data:
  source: "data/orders.csv"
  sample_rows: 20
limits:
  max_rows_returned: 5000
  statement_timeout_seconds: 30
YAML

Export your API credentials as environment variables so they never enter the repository. Never commit keys to a config file.

export ANALYTICS_AGENT_API_KEY="<your-key>"

Set temperature to 0 in the config. It does not guarantee determinism across providers, but it removes one source of variance and makes regressions easier to attribute.

Grounding the agent in the actual schema

The first rule of a reliable analytics agent: it must never guess what the data looks like. Before any question is answered, profile the source and put the profile in the prompt context.

import duckdb

con = duckdb.connect()
con.execute("CREATE VIEW t AS SELECT * FROM read_csv_auto('data/orders.csv')")

print(con.execute("DESCRIBE t").df())
print(con.execute("SUMMARIZE t").df())

DESCRIBE gives column names and types; SUMMARIZE adds null counts, distinct counts, and basic distributions. Passing both to the model costs a few hundred tokens and eliminates most column-hallucination failures.

Sample real rows too, but sample deterministically — a fixed seed or an ORDER BY with a LIMIT — so the same question produces the same context on every run.

Constraining the tool surface

An agent is only as reliable as the actions it can take. Define a small, typed set of tools and refuse everything else.

from pydantic import BaseModel, Field

class DescribeTable(BaseModel):
    table: str = Field(..., description="Name of a registered view.")

class SampleRows(BaseModel):
    table: str
    limit: int = Field(20, ge=1, le=100)

class RunSQL(BaseModel):
    sql: str = Field(..., description="A read-only SELECT or WITH statement.")
    rationale: str = Field(..., description="Why this query answers the sub-question.")

Four tools — describe, sample, run SQL, and a final answer tool — cover a surprising amount of analytical work. Every additional tool multiplies the space of ways the agent can go wrong.

Making execution safe and deterministic

Generated SQL must be read-only and bounded. Keyword filtering is a useful first gate, but be honest about its limits: it is not a security boundary. Pair it with a read-only database connection.

FORBIDDEN = ("insert", "update", "delete", "drop", "alter", "create", "attach", "copy")

def safe_sql(sql: str) -> str:
    stripped = sql.strip().rstrip(";")
    lowered = stripped.lower()
    if not lowered.startswith(("select", "with")):
        raise ValueError("Only SELECT and WITH statements are permitted.")
    if any(kw in lowered for kw in FORBIDDEN):
        raise ValueError("Statement contains a forbidden keyword.")
    return stripped

Even a SELECT can be expensive, so set a statement timeout and a hard row cap. An agent that returns 500,000 rows has not answered the question; it has moved the problem downstream.

Validating every result before it reaches the user

This is the step most teams skip, and it is the one that matters most. After each query executes, run programmatic assertions on the result. Failures should trigger a repair loop, not a confident answer.

def assert_sane(df, min_rows=1, max_null_fraction=0.2, unique_key=None):
    assert len(df) >= min_rows, f"Expected >= {min_rows} rows, got {len(df)}"
    if unique_key:
        assert df[unique_key].is_unique, f"{unique_key} is not unique after the join"
    for col in df.columns:
        frac = df[col].isna().mean()
        assert frac <= max_null_fraction, f"{col} is {frac:.0%} null"

The unique_key check is particularly valuable. It catches the classic many-to-many join that inflates row counts and quietly doubles every sum in a downstream aggregation.

Wrap the generate-validate-repair sequence in a bounded retry loop. Give the model the exact assertion error and let it revise; cap the attempts so a confused agent does not loop forever.

from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=1, max=8),
    retry=retry_if_exception_type(ValueError),
)
def generate_and_execute(question, schema_context):
    sql = safe_sql(propose_sql(question, schema_context))
    df = con.execute(sql).df()
    assert_sane(df)
    return df

Tracing and replay

For every run, append a structured record to a JSONL file: the question, the schema context hash, the generated SQL, the validation outcome, the final answer, and timing.

import json, time, hashlib

def log_event(path, **fields):
    fields["ts"] = time.time()
    with open(path, "a") as f:
        f.write(json.dumps(fields) + "\n")

log_event("artifacts/trace.jsonl", question=q, sql=sql, rows=len(df))

This is unglamorous and it is the difference between debugging by reasoning and debugging by guessing. When an answer is challenged three weeks later, the trace is the artifact that resolves the dispute.

Usage examples

A minimal command-line entry point makes the agent testable in CI and in a terminal. Keep the interface narrow.

python -m analytics_agent.run \
  --config config.yaml \
  --question "What was the monthly revenue trend for the top three regions in the last year?"

Run against a local Parquet file instead of CSV by changing data.source in the config — the profiling and tooling layers do not change.

python -m analytics_agent.run --config config.local.yaml --question "Which customers have not ordered in 90 days?"

For interactive exploration, keep the profiling step warm and reuse the DuckDB connection so repeated questions do not re-read the file.

Evaluating reliability instead of vibes

Reliability is a measurable property, not a feeling. Build a golden set of twenty to fifty questions with answers you have verified by hand or by an independent query. Score each run on three axes: did it produce the right answer, did it produce it with a valid query, and did it produce it within the cost budget.

pytest analytics_agent/evals -q --tb=short

A useful test asserts determinism explicitly — run the same question twice and compare the resulting DataFrames.

def test_repeatable(agent, question):
    a = agent.ask(question)
    b = agent.ask(question)
    assert a.equals(b), "Agent output is not reproducible at temperature 0"

Track the pass rate over time. A drop after a prompt or model change is a signal; without the eval set, it is an invisible regression that surfaces as a wrong dashboard number months later.

Open limits worth naming

Three constraints resist easy fixes. First, LLM-generated SQL can be correct on your sample and wrong on edge cases the sample never contained, so coverage of the eval set matters more than its size. Second, keyword-based SQL filtering is defense in depth, not defense; isolation belongs at the database level. Third, "reliable" is domain-relative — a 95% pass rate is excellent for exploratory analysis and unacceptable for regulatory reporting. Decide the threshold before you build, not after.

Conclusion

The lesson the KDD Cup setting makes concrete is that analytics agents are evaluated like data science submissions, not like chat responses: against held-out questions, with no partial credit for confidence. Reliability comes from the scaffolding around the model — schema grounding before generation, a narrow typed tool surface, read-only bounded execution, programmatic validation of every result, bounded repair loops, and a regression suite that catches drift. None of these components is exotic, and none of them is optional if the output is a number someone will act on. Start with the profiling step and the validation assertions; they deliver the largest reliability gain for the least effort, and everything else builds on top of them.

Source: "Building Reliable Data Analytics Agents: Lessons from the KDD Cup," NVIDIA AI Blog — https://developer.nvidia.com/blog/building-reliable-data-analytics-agents-lessons-from-the-kdd-cup