agent-optimization

Agent Optimization

A practical prompt, checklist, and playbook for making LLM agents faster without lying to yourself with benchmarks.

This repo is for engineers who already have an agent and want to improve latency, cost, reliability, or delivery quality without reward-hacking the metric.

Origin: this prompt was distilled from the blog post Never optimize the number: seven principles for faster LLM agents.

Quick Start

Copy the prompt:

• prompts/agent-optimization-prompt.md

Paste it into the agent, coding assistant, or reviewer that is about to optimize your system.

Then give it:

1. The goal metric you want improved.
2. The current baseline and how it was measured.
3. A small sample of real traces, failures, or eval cases.
4. The correctness and delivery gates that must not regress.
5. The files, tools, or dashboards it may inspect.

Use this starter request:

Use the Agent Optimization Prompt.

Goal: reduce median latency and token cost for this agent without reducing delivery quality.
Baseline: [metric, sample size, date range, harness].
Correctness gates: [tests/evals/manual checks].
Known constraints: [model fixed? no new dependencies? production path?].

First produce an audit. Do not propose prompt changes until deterministic levers are exhausted.

What It Optimizes For

• Honest measurement before and after every change
• Real replay data instead of demo prompts
• Delivery and correctness before speed
• Deterministic fixes before prompt/model tweaks
• Fewer wasted turns, retries, and generated tokens
• Fast paths that degrade to partial output, not silence
• Fresh blind validation before claiming a win

What It Is Not

This is not an automatic prompt optimizer, benchmark leaderboard, or magic system prompt.

It is a work protocol. It tells an agent or engineer how to optimize an agent without cheating:

• no weakened gates
• no "median of successful runs"
• no tuning to the examples you already saw
• no speedups that only work because output got smaller or worse

Core Principle

Never optimize the number. Optimize the thing the number measures.

If the metric improved and the user's experience did not, the optimization failed.

Files

Use As An Agent Skill

The skills/agent-optimization/ folder is self-contained. It follows the common agent skill shape: a directory with SKILL.md, optional references/, and optional agent metadata.

For Codex, copy the skill folder into your Codex skills directory:

mkdir -p ~/.codex/skills
cp -R skills/agent-optimization ~/.codex/skills/

For Claude surfaces that support custom skills, use the same skills/agent-optimization/ folder as the skill source. If the surface expects an upload, zip that folder and upload it as a custom skill.

Then invoke it by asking for agent optimization work, for example:

Use the agent-optimization skill to review this agent latency plan.

The skill includes its own copy of the prompt and checklist under skills/agent-optimization/references/.

Compatibility notes:

• Codex skills: https://developers.openai.com/codex/skills
• Claude Code skills: https://docs.anthropic.com/en/docs/claude-code/skills

When To Use

Use this when:

• an agent is slow or expensive
• an eval score improved but user value is unclear
• an agent keeps retrying, re-deriving, or regenerating large artifacts
• a prompt patch fixed one case but may have fit the test set
• a team wants an adversarial review before shipping an optimization

Do not use it when:

• you have no baseline
• you cannot define delivery or correctness
• you only want a prettier prompt
• you are unwilling to keep failed runs in the record

The Short Version

1. Measure before believing.
2. Replay real traffic.
3. Count delivery first, latency second.
4. Find the bottleneck before touching prompts.
5. Exhaust deterministic levers before touching the model.
6. Compile repeated reasoning into code.
7. Degrade inexpressible asks to partial output, never silence.
8. Validate on fresh blind data.
9. Report like an adversary is reading.

Citation

If this helps your work, cite the source post:

Abhishek Gahlot, "Never optimize the number: seven principles for faster LLM agents", 2026. https://abhishek.it/blog/never-optimize-the-number

License

MIT. Use it, adapt it, and keep the honesty rules intact.