How OpenAI’s Performance Factory Keeps ChatGPT Fast
OpenAI built the Performance Factory, an autonomous system using Codex-powered workflows to find and fix performance regressions. This system automates investigations, proposes fixes, and verifies recovery, enabling the company to track over 1,700 investigations in three months. The factory ensures ChatGPT remains fast by shifting from manual labor to automated agent factories.
Tags
Quick summary
OpenAI built the Performance Factory, an autonomous system using Codex-powered workflows to find and fix performance regressions. This system automates investigations, proposes fixes, and verifies recovery, enabling the company to track over 1,700 investigations in three months. The factory ensures ChatGPT remains fast by shifting from manual labor to automated agent factories.
How OpenAI’s Performance Factory Keeps ChatGPT Fast
The pace of modern software development has outstripped traditional performance engineering methods. As systems grow in complexity, manual investigations into slowdowns become unsustainable. To address this, OpenAI developed the Performance Factory, an autonomous system designed to find and fix performance regressions at unprecedented speed. This system leverages Codex and large language models to automate the entire lifecycle of performance engineering, from detecting an alert to verifying a fix in production.
The Evolution from Manual Labor to Agent Factories
OpenAI’s approach to performance engineering has evolved through distinct stages, moving from manual labor to a fully automated factory model. Initially, engineers manually examined metrics, logs, and traces to investigate slowdowns. This process was time-consuming and required deep domain knowledge, often causing relevant signals to be missed. To mitigate this, the team built over 30 performance tools, including CPU and allocation profilers, to assist with investigations.
The introduction of Codex marked a shift toward partnership. Engineers could ask questions like "Why did this get slower?" and collaborate with Codex to search codebases, find relevant metrics, and follow interactions across services. However, engineers still had to steer the process, deciding which tools to use and how to guide the investigation.
To further automate, OpenAI captured the investigation process into a shared skill called perf-forensics. This skill instructs Codex on available tools, when to use them, and how to work through a slowdown. The system can now investigate whether deployments or configuration changes explain a slowdown without human guidance for every step.
The final evolution involved moving these investigations to the cloud. Previously, engineers would copy alerts into Codex and paste findings back into Slack. Now, alerts automatically trigger Codex sessions in the cloud. The agent investigates, posts findings, and the workflow continues autonomously, reducing the burden on engineers to sift through threads to identify which regressions need attention.
The Four Components of the Performance Factory
The Performance Factory is structured around four core components that enable end-to-end agentic workflows. These components work together to track task states, assign work, execute code, and respond to external events.
- Task Service with Configurable Workflows: Engineers define workflows by adding configuration files and prompts written in Markdown. These configurations list the states a task can move through, allowed transitions, and steps requiring human intervention.
- Dispatch Loop: This component periodically checks which tasks are ready for the next step. It assigns an agent to a task by starting a Codex turn with the specific instructions for the task's current state.
- Codex in the Cloud: Once assigned, the agent runs in a cloud environment with access to the same tools an engineer would use, including code repositories, metrics, logs, traces, CPU profiles, and memory snapshots.
- Subscriptions: These wake agents when specific events occur, such as a GitHub review comment posting or a deployment completing. This ensures tasks can wait for human approval or external dependencies before resuming.
This architecture allows OpenAI to run work in parallel without relying on engineers' laptops or manually managing individual cloud development boxes. Each task maintains its state and findings, allowing work to pass between agents and engineers and resume after code reviews or deployments.
The Regression Workflow: From Alert to Recovery
The first workflow implemented within the Performance Factory is the Regression Workflow. This workflow automates the process of fixing regressions, moving a task through stages such as "needs investigation," "root caused," "fix proposed," and "resolved."
When a performance alert fires, the workflow opens a task. An agent investigates the slowdown, records findings in the task, and proposes a fix for an engineer to review. After deployment, another agent picks up the same task to verify that performance has returned to baseline.
In just over three months, the Regression Workflow tracked more than 1,700 investigations and root caused over 600 cases. This equates to approximately 17 automated investigations every day. The system effectively handles the full loop: investigating, authoring the fix, and verifying mitigation.
Finding the Root Cause and Adversarial Review
The initial step in the Regression Workflow is finding the root cause. The agent analyzes which metrics worsened, when the change occurred, and which service requests were affected. The perf-forensics skill provides Codex with access to the tools performance engineers use and a playbook for answering these questions.
The agent looks back through data to determine when a metric first degraded, even if an alert fires later. It compares groups of requests to distinguish between actual slowdowns and other factors.
Once a cause is identified, an adversarial review strengthens the diagnosis. This step ensures the proposed fix is robust before it reaches engineers for implementation. OpenAI notes that fixing the regression is often the hardest part, requiring careful consideration of the proposed solution.
Lessons Learned and Future Applications
Building the Performance Factory has provided several key lessons for engineering teams adopting agentic workflows.
- Protect Engineers’ Attention: By automating the investigation and initial diagnosis, the system frees engineers to focus on high-level decision-making and complex problem-solving rather than sifting through alerts.
- Evaluate Changes to Agents: OpenAI emphasizes the importance of evaluating how changes to agents affect their performance. This includes rigorous testing of investigations to ensure accuracy.
- Evaluate Agent-to-Engineer Handoffs: The system must effectively manage the transition between automated agents and human engineers, ensuring context is preserved and handoffs are smooth.
- Turn Recurring Failures into Self-Repair: The ultimate goal is to transform common optimization challenges into workflows that run regularly across services, enabling self-repair capabilities.
Looking ahead, OpenAI plans to apply these principles to broader applications and proactive monitoring. The Performance Factory demonstrates how Codex-powered systems can evolve from simple assistants to autonomous factories capable of maintaining system performance at scale.
Conclusion
OpenAI’s Performance Factory represents a significant advancement in performance engineering. By integrating Codex into a structured system of workflows, dispatch loops, and cloud-based execution, OpenAI has automated the detection and resolution of performance regressions. The system has already proven its value, handling thousands of investigations and root causes in a short period. As the field of agentic development continues to grow, the lessons learned from the Performance Factory offer a blueprint for other teams seeking to maintain high performance in complex, rapidly evolving systems.
Source: https://openai.com/index/how-openai-performance-factory-keeps-chatgpt-fast



