What Building Shippy Taught Us About Building Agents

What Building Shippy Taught Us About Building Agents

Building reliable AI agents is one of the hardest problems in applied machine learning today. Over the past year, our team developed Shippy, an open-source agent designed to automate software shipping workflows—from code review to deployment. What started as a simple proof-of-concept turned into a deep education in agent architecture, error handling, and system design. This article shares the practical lessons we learned, complete with concrete code examples and configuration steps you can apply to your own agent projects.

The Problem We Set Out to Solve

Shipping software involves dozens of repetitive tasks: linting, running tests, updating changelogs, bumping versions, and triggering deployments. Most teams use CI/CD pipelines, but those pipelines are brittle—they break when a test flakes, a dependency changes, or a config file moves. We wanted an agent that could not only execute these steps but also recover from failures, communicate status, and adapt to new project structures without manual intervention.

The goal was ambitious: build an agent that could clone a repo, understand its build system, run tests, fix common issues, and open a pull request with the fix—all without human prompting beyond the initial goal.

Requirements

Before we dive into the implementation, here are the core requirements for building an agent like Shippy:

• **Python 3.10+** – The agent logic is written in Python, leveraging async patterns for parallel task execution.
• **Git** – The agent needs to clone, commit, and push to repositories.
• **OpenAI API key** – We used GPT-4 for reasoning and code generation, but any LLM API works.
• **Docker** (optional) – For running tests in isolated environments.
• **A task queue** – We used Redis + Celery for handling long-running tasks asynchronously.
• **Node.js 18+** – Only required if your agent needs to interact with JavaScript/TypeScript projects.

Step-by-Step Installation

Let's walk through setting up a minimal version of Shippy on your local machine. This will give you a working agent that can clone a repo, analyze its structure, and propose a fix for a failing test.

1. Create a Virtual Environment and Install Dependencies

First, set up an isolated Python environment:

python3 -m venv shippy-env
source shippy-env/bin/activate

Install the core dependencies:

pip install openai gitpython pydantic httpx redis celery

These packages provide the LLM interface (`openai`), Git operations (`gitpython`), data validation (`pydantic`), HTTP requests (`httpx`), and async task management (`redis`, `celery`).

2. Configure Environment Variables

Create a `.env` file in your project root:

OPENAI_API_KEY=sk-your-key-here
GITHUB_TOKEN=ghp_your-token-here
REDIS_URL=redis://localhost:6379/0

Load these in your Python code:

import os
from dotenv import load_dotenv

load_dotenv()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
GITHUB_TOKEN = os.getenv("GITHUB_TOKEN")
REDIS_URL = os.getenv("REDIS_URL")

3. Set Up the Agent Core

Create `agent.py` with the basic agent class:

from openai import OpenAI
from git import Repo
import json

class ShippyAgent:
    def __init__(self, api_key: str, github_token: str):
        self.client = OpenAI(api_key=api_key)
        self.github_token = github_token

    def clone_repo(self, repo_url: str, local_path: str) -> Repo:
        """Clone a GitHub repository to local path."""
        repo = Repo.clone_from(
            repo_url,
            local_path,
            env={"GIT_ASKPASS": "echo", "GITHUB_TOKEN": self.github_token}
        )
        return repo

    def analyze_project(self, repo_path: str) -> dict:
        """Analyze the project structure to understand build system."""
        import os
        files = os.listdir(repo_path)
        structure = {}
        if "package.json" in files:
            structure["type"] = "node"
        elif "pyproject.toml" in files:
            structure["type"] = "python"
        elif "Cargo.toml" in files:
            structure["type"] = "rust"
        else:
            structure["type"] = "unknown"
        return structure

    def generate_fix(self, error_log: str, project_type: str) -> str:
        """Use LLM to generate a fix for a given error."""
        prompt = f"""
        You are a senior software engineer. Given this error log from a {project_type} project, suggest a code fix.
        Error log:
        {error_log}
        Provide the fix as a diff or code change.
        """
        response = self.client.chat.completions.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.2
        )
        return response.choices[0].message.content

4. Add Task Queue for Long-Running Operations

Create `tasks.py` for background processing:

from celery import Celery
from agent import ShippyAgent

app = Celery('shippy', broker='redis://localhost:6379/0')

@app.task
def fix_repo(repo_url: str, error_log: str):
    agent = ShippyAgent(
        api_key="your-openai-key",
        github_token="your-github-token"
    )
    local_path = "/tmp/repos/" + repo_url.split("/")[-1].replace(".git", "")
    repo = agent.clone_repo(repo_url, local_path)
    structure = agent.analyze_project(local_path)
    fix = agent.generate_fix(error_log, structure["type"])
    # Apply fix and create PR (simplified)
    with open(f"{local_path}/fix.patch", "w") as f:
        f.write(fix)
    return {"status": "fix_generated", "patch_file": "fix.patch"}

Run the Celery worker:

celery -A tasks worker --loglevel=info

Usage Examples

Now let's see Shippy in action with real-world scenarios.

Example 1: Fixing a Failing Python Test

Suppose you have a Python project where tests are failing due to a missing import. Run the agent:

from agent import ShippyAgent

agent = ShippyAgent(
    api_key="sk-your-key",
    github_token="ghp_your-token"
)

error = """
ModuleNotFoundError: No module named 'requests'
File: tests/test_api.py, line 3
"""

result = agent.generate_fix(error, "python")
print(result)
# Output: Add `import requests` at the top of test_api.py, or add `requests` to requirements.txt

Example 2: Automating a Code Review

The agent can also review pull requests. Here's a minimal PR reviewer:

def review_pr(repo_url: str, pr_number: int):
    # Clone the PR branch
    repo = agent.clone_repo(repo_url, "/tmp/pr-review")
    # Get diff
    diff = repo.git.diff("main", "feature-branch")
    # Ask LLM to review
    prompt = f"Review this diff and list any issues:\n{diff}"
    response = agent.client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

Example 3: Running in Docker for Isolation

To run tests safely without affecting your host system, use Docker:

docker run --rm -v $(pwd):/workspace python:3.11 bash -c "cd /workspace && pip install -r requirements.txt && pytest"

Integrate this into the agent:

import subprocess

def run_tests_in_docker(repo_path: str) -> str:
    result = subprocess.run(
        ["docker", "run", "--rm", "-v", f"{repo_path}:/workspace", "python:3.11",
         "bash", "-c", "cd /workspace && pip install -r requirements.txt && pytest"],
        capture_output=True, text=True
    )
    return result.stdout + result.stderr

Key Lessons Learned

1. Agents Need Robust Error Recovery

The first version of Shippy assumed every step would succeed. It didn't. We learned to implement retry logic with exponential backoff. For example, if a Git clone fails due to network issues, retry up to three times:

import time
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
def clone_with_retry(repo_url, local_path):
    return Repo.clone_from(repo_url, local_path)

2. Context Is Everything

LLMs are powerful but lack project-specific knowledge. We found that providing the agent with a project's README, recent commit history, and CI configuration dramatically improved fix quality. We added a "context gathering" step that reads these files before generating any response.

3. Test in Production-Like Environments

Running agents on your laptop is fine for prototyping, but production requires isolation. Docker containers for each task prevented state pollution between runs. We also added timeouts to prevent runaway agents:

from concurrent.futures import TimeoutError

with timeout(seconds=120):
    result = agent.run_task()

4. Human-in-the-Loop Is Still Essential

Despite impressive results, the agent occasionally produced nonsensical fixes. We added a review step where the agent generates a PR, but a human must approve it before merging. This balances automation with safety.

5. Monitoring and Logging Are Non-Negotiable

We instrumented every agent action with structured logging. This allowed us to debug failures and improve prompts over time. Use Python's `logging` module with JSON formatter:

import logging
import json

class JsonFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps(record.__dict__)

logger = logging.getLogger("shippy")
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)

Conclusion

Building Shippy taught us that AI agents are not magic—they are carefully designed systems that combine LLM reasoning with robust engineering practices. The most important lesson is that agent reliability comes from infrastructure, not just the model. Retry logic, context gathering, isolation, and human oversight turned a fragile prototype into a useful tool.

If you're building your own agent, start small. Clone a repo, run a test, fix a single issue. Then add error handling. Then add monitoring. Then add the next feature. The path from demo to production is paved with edge cases, but the journey is worth it. Agents that can ship software autonomously are not science fiction—they are just good engineering.

*For further reading on agent design patterns, see the Hugging Face Blog, OpenAI News, Microsoft AI Blog, and Anthropic News.*