Stop Ranking Agent Configs by Average Score

Stop Ranking Agent Configs by Average Score

In the race to build the most effective AI agents, many teams default to a single, simplistic metric: the average score across a set of evaluation tasks. Whether you're fine-tuning a language model, configuring retrieval-augmented generation (RAG) pipelines, or optimizing tool-use logic, the temptation to rank agent configurations by their mean performance is almost irresistible. But this approach is fundamentally flawed. Average scores hide variance, mask failure modes, and reward configurations that are "good enough" at everything but excellent at nothing. In this article, we'll explore why average-score ranking is misleading, how to move beyond it, and provide concrete steps to implement a more robust evaluation framework.

The Problem with Averages

Consider a simple scenario: you have two agent configurations, A and B, tested on 10 diverse tasks. Config A scores 95 on 9 tasks but fails catastrophically on one task, scoring 0. Config B scores 70 on all 10 tasks. The average score for A is 85.5, while B's average is 70. Based on average alone, you'd rank A higher. But in a production system, A's catastrophic failure could mean losing a customer, corrupting a database, or generating harmful content. B, while less flashy, is reliable across the board. The average score completely obscures this critical trade-off.

This isn't just a theoretical concern. In practice, agent evaluation often involves tasks with varying difficulty, edge cases, and long-tail scenarios. Averages compress this rich information into a single number, discarding distribution shape, variance, and worst-case behavior. As noted in discussions on platforms like Towards Data Science, the limitations of mean-based metrics are well-known in machine learning, yet they persist in agent evaluation due to convenience.

Why Agents Amplify the Problem

Agents differ from traditional ML models in key ways that make average-score ranking even more dangerous:

• **Sequential dependencies**: An agent's actions in early steps affect later outcomes. A single bad decision can cascade into total failure.
• **Tool use**: Agents often call external APIs, databases, or web services. A configuration that works 95% of the time may fail in critical edge cases.
• **Non-determinism**: Due to temperature settings, sampling, or external factors, the same config may produce different results on the same input.

Average scores hide these dynamics. A config that sometimes fails spectacularly may still have a high average if it succeeds often enough. But in production, "often enough" isn't good enough.

Requirements

Before we dive into implementation, let's outline what you'll need to follow the examples in this article.

• **Python 3.9+** installed on your system.
• **pip** package manager.
• Basic familiarity with Python, command-line tools, and JSON.
• Optionally, **Docker** if you want to run evaluations in isolated environments.

We'll use several open-source libraries:

• `pandas` for data manipulation
• `numpy` for statistical computations
• `matplotlib` and `seaborn` for visualization
• `scipy` for statistical tests

You'll also need a dataset of agent evaluation results. For demonstration, we'll create synthetic data, but you can replace it with your own.

Step-by-Step Installation

1. Set Up a Virtual Environment

First, create a dedicated environment to avoid dependency conflicts.

python3 -m venv agent-eval-env
source agent-eval-env/bin/activate  # On Windows: agent-eval-env\Scripts\activate

2. Install Required Packages

Install the core libraries we'll use for analysis and visualization.

pip install pandas numpy matplotlib seaborn scipy

This installs everything needed for the examples below.

3. Verify Installation

Run a quick check to ensure everything works.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
print("All imports successful!")

If no errors appear, you're ready to proceed.

Moving Beyond Average Scores

Instead of ranking by mean, we'll adopt a multi-faceted evaluation approach. The key ideas are:

1. **Report distribution statistics**: Include median, interquartile range (IQR), minimum, and maximum scores. 2. **Analyze failure modes**: Identify specific tasks where configs fail and why. 3. **Use statistical tests**: Compare configs with confidence intervals, not just point estimates. 4. **Visualize performance**: Use box plots, violin plots, and heatmaps to reveal patterns.

Let's implement these steps with concrete code.

Creating a Synthetic Evaluation Dataset

First, we'll generate a dataset that mimics real agent evaluation results. In practice, you'd load your own CSV or JSON file.

import pandas as pd
import numpy as np

np.random.seed(42)
n_tasks = 20
n_configs = 5

# Generate task names
tasks = [f"task_{i}" for i in range(n_tasks)]

# Create synthetic scores for each config
# Config A: high on most, catastrophic on one
config_a = np.random.normal(90, 5, n_tasks)
config_a[0] = 0  # catastrophic failure

# Config B: consistent, moderate
config_b = np.random.normal(70, 3, n_tasks)

# Config C: high variance
config_c = np.random.normal(80, 20, n_tasks)

# Config D: bimodal (some tasks great, some poor)
config_d = np.concatenate([np.random.normal(95, 2, 10), np.random.normal(40, 10, 10)])

# Config E: mediocre but stable
config_e = np.random.normal(60, 2, n_tasks)

# Assemble into DataFrame
data = pd.DataFrame({
    'task': tasks * n_configs,
    'config': ['A']*n_tasks + ['B']*n_tasks + ['C']*n_tasks + ['D']*n_tasks + ['E']*n_tasks,
    'score': np.concatenate([config_a, config_b, config_c, config_d, config_e])
})

print(data.head())

Computing Summary Statistics

Now let's compute meaningful statistics beyond the average.

summary = data.groupby('config')['score'].agg([
    'mean', 'median', 'std', 'min', 'max',
    lambda x: x.quantile(0.25),
    lambda x: x.quantile(0.75)
])
summary.columns = ['mean', 'median', 'std', 'min', 'max', 'q25', 'q75']
summary = summary.sort_values('mean', ascending=False)
print(summary.round(2))

Notice how Config A has the highest mean (around 85.5) but a min of 0. Config B has a lower mean (70) but a min of 64. If you only looked at means, you'd pick A. But the min reveals A's fatal flaw.

Visualizing Distributions

A picture is worth a thousand averages. Let's create a box plot.

import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(10, 6))
sns.boxplot(data=data, x='config', y='score', order=['A','B','C','D','E'])
plt.title('Score Distribution by Agent Configuration')
plt.ylabel('Score')
plt.xlabel('Configuration')
plt.show()

The box plot immediately shows Config A's outlier at 0. Config C has huge spread. Config D is bimodal (two distinct clusters). Configs B and E are tight but low. This visual tells you far more than a table of averages.

Statistical Comparison with Confidence Intervals

Instead of ranking by mean, use confidence intervals to compare configs.

from scipy import stats

def mean_confidence_interval(data, confidence=0.95):
    a = 1.0 * np.array(data)
    n = len(a)
    se = stats.sem(a)
    h = se * stats.t.ppf((1 + confidence) / 2., n-1)
    return np.mean(a), np.mean(a)-h, np.mean(a)+h

ci_df = data.groupby('config')['score'].apply(
    lambda x: mean_confidence_interval(x)
).apply(pd.Series)
ci_df.columns = ['mean', 'ci_lower', 'ci_upper']
print(ci_df.round(2))

Config A's confidence interval is wide (from about 73 to 98) because of the outlier. Config B's interval is narrow (68 to 72). Overlapping intervals suggest the configs may not be statistically different, despite mean differences.

Analyzing Failure Modes

Identify which tasks cause failures. We'll flag any score below a threshold.

threshold = 50  # Define failure threshold
failures = data[data['score'] < threshold]
print("Failures (score < 50):")
print(failures.groupby(['config', 'task']).size().unstack(fill_value=0))

Config A fails only on task_0. Config C fails on several tasks. Config D fails on tasks 10-19. This pinpoints exactly where each config struggles.

Usage Examples

Example 1: Comparing Two Configs

Suppose you're deciding between Config A and Config B. Instead of comparing means, run a two-sample t-test or Mann-Whitney U test.

config_a_scores = data[data['config'] == 'A']['score']
config_b_scores = data[data['config'] == 'B']['score']

t_stat, p_value = stats.ttest_ind(config_a_scores, config_b_scores)
print(f"T-test: t={t_stat:.3f}, p={p_value:.3f}")

# Non-parametric alternative
u_stat, p_value_u = stats.mannwhitneyu(config_a_scores, config_b_scores)
print(f"Mann-Whitney U: U={u_stat:.0f}, p={p_value_u:.3f}")

A high p-value (>0.05) means you cannot conclude the configs are different, even if means differ.

Example 2: Creating a Radar Chart for Multi-Metric Evaluation

Define multiple metrics (e.g., accuracy, latency, robustness, safety) and plot them.

# Simulate multiple metrics
metrics = ['accuracy', 'latency', 'robustness', 'safety']
config_metrics = {
    'A': [90, 50, 60, 20],
    'B': [70, 80, 85, 90],
    'C': [80, 60, 70, 60]
}

import numpy as np
import matplotlib.pyplot as plt

angles = np.linspace(0, 2*np.pi, len(metrics), endpoint=False).tolist()
angles += angles[:1]

fig, ax = plt.subplots(figsize=(8,8), subplot_kw=dict(polar=True))
for config, values in config_metrics.items():
    values += values[:1]
    ax.plot(angles, values, label=config)
    ax.fill(angles, values, alpha=0.1)
ax.set_xticks(angles[:-1])
ax.set_xticklabels(metrics)
ax.legend(loc='upper right')
plt.show()

Config A excels at accuracy but fails at safety. Config B is balanced. The radar chart makes trade-offs obvious.

Example 3: Automating the Evaluation Pipeline

Create a Python script that loads results, computes statistics, and generates a report.

# save as evaluate_agents.py
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats

def evaluate_agents(csv_path, output_dir='./eval_results'):
    import os
    os.makedirs(output_dir, exist_ok=True)
    
    df = pd.read_csv(csv_path)
    
    # Summary stats
    summary = df.groupby('config')['score'].agg(['mean','median','std','min','max'])
    summary.to_csv(f'{output_dir}/summary.csv')
    
    # Box plot
    plt.figure()
    sns.boxplot(data=df, x='config', y='score')
    plt.savefig(f'{output_dir}/boxplot.png')
    
    # Failure analysis
    failures = df[df['score'] < 50]
    failures.to_csv(f'{output_dir}/failures.csv')
    
    print(f"Results saved to {output_dir}")

if __name__ == '__main__':
    evaluate_agents('results.csv')

Run it with:

python evaluate_agents.py

Conclusion

Ranking agent configurations by average score is a dangerous oversimplification that hides variance, masks catastrophic failures, and rewards mediocrity. Instead, adopt a richer evaluation framework that includes distribution statistics, visualizations, confidence intervals, and failure mode analysis. The code and examples provided here give you a practical starting point to implement this in your own projects.

By moving beyond averages, you'll make more informed decisions, build more robust agents, and avoid costly surprises in production. Remember: the goal isn't to find the config with the highest mean—it's to find the config that reliably performs well across the scenarios that matter most.