am-i-nerfed

Am I nerfed

A small local "intelligence" test for your favourite model. Measures score on six tasks of requested difficulty, constrained by either time or tokens. Shipped as a CLI and an Agent Skill. Meant to answer the morning dilemma: "Am I crazy, or is it my model?"

Useful for:

• A quick sanity check when struggling to interact with the model, such as when suspected of being "nerfed".
• Evaluating the same model across different days.
• Comparing different models.
• Comparing different reasoning efforts for the same model.

Install

npm install --global am-i-nerfed
npx skills add marinsokol5/am-i-nerfed
am-i-nerfed init # builds your private task bank

init generates the tasks and answers on your machine from a random seed, so no model has seen them before.

Run

CLI

Starts a fresh (no personal instructions or earlier conversation) Claude Code or Codex session through your existing login (reuses the existing CLI binaries) to perform the assessment and stops it at the deadline (either the token budget or the time budget). It also properly records all the input (model, reasoning effort, ...) and output (transcript, reasoning tokens, ...) parameters. More controlled environment than the Skill below. Support for other coding agents would be easy to add.

Default is medium difficulty and 180 seconds.

am-i-nerfed run --agent codex --model gpt-6-astra --effort high --difficulty hard --seconds 360
am-i-nerfed run --agent claude --model claude-opus-5-5 --effort medium --difficulty easy --max-output-tokens 10000

Skill

Conducts the assessment inside whatever conversation invoked the skill, running against the current agent, model, reasoning effort and any prior context in that conversation. Only supports a time limit: the execution happens outside our CLI, so there is no way to track tokens or interrupt the coding agent mid-answer. The full instructions the agent follows are in SKILL.md.

/am-i-nerfed medium 180

What it measures

The tasks are logic puzzles that test reasoning and comprehension, not memory, coding or image generation. It's a very basic proxy of a model's capabilities and cannot reliably be used for any serious claims or comparisons.

Each task is worth a sixth of the score and gives partial credit. Only answers saved before the deadline or budget count. The solver may only use the assessment commands: no code, browsing or files.

Tasks are created deterministically from a random seed during am-i-nerfed init or am-i-nerfed reset, so both the tasks and their answers differ for everyone.

Initial evaluation

Scores are percentages per level, then the average of the three. Overall is the mean of the time-limit and token-budget averages at high effort; models are ranked by it. All three sweeps use no system prompt and a 30-second grace period; token sweeps also allow 1,000 grace tokens and stop a run at twice its budget.

Run on 2026-10-05, one run per cell, Claude models through Claude Code and OpenAI models through Codex, all on the same private task bank. Full results are in eval/RESULTS.md; reproduce them with the harness in eval/.

Opus 4.6 scored 0 because on every level it reasoned in one long pass and never saved an answer before the deadline or budget, despite being asked to do so.

Interesting findings

1. Anthropic models are really bad at following simple instructions -> A year ago they were already a bit "stubborn", but threatening them (or Anthropic, or their "family") reliably made them follow instructions to the letter. Now even that doesn't work, and they ignore simple instructions whatever we try. We tried getting them to do short reasoning loops or check their current budget (time or token) frequently, or submit initial answers to all tasks before diving deep into something, or to submit answers one by one, or to use the budget fully. Basically we tried to help the model perform well on the exam, since sometimes it would just give up and quit. It's very hard to get them to do anything they don't want to do. OpenAI models have been an absolute breeze to work with in comparison. We removed the system prompt (Claude Code still adds a short identity line and a few reminders; the direct API runs had none at all) and played with various executions (in total for sure 200+) across the Claude Code harness and direct API usage, and across various reasoning efforts. The behaviour stayed the same everywhere, so it is most likely baked in during training.
2. Anthropic models need to provide a summary at all cost -> They seem to be purposely defensive with any kind of budget because they are (same as above) trained to provide a closing summary at all times, both API and Claude Code harness.
3. OpenAI needs to call Nvidia pronto -> Anthropic inference speed across our testing has been about 4-6x faster than OpenAI's. Personally, the difference is quite obvious in daily usage of Opus 5.5 compared to Astra, even on fast mode, it's night and day difference in response speed.
4. Fable 5.1 is not reliably testable at the moment -> Its safety classifier is too trigger-happy and refuses this assessment quoting cybersecurity reasons ("This request triggered restrictions on violative cyber content and was blocked under Anthropic's Usage Policy."), and Claude Code then silently falls back to Opus 4.8. This tool marks such a run as failed, since the requested Fable model is not the one being evaluated.

Tenets

There is a plethora of published benchmarking and daily nerf trackings; so why on God's green Earth do we need this one. Well, I wanted something that's:

1. Individual -> Showing results for your own model in your own coding agent, since this is what you actually have to work with at the moment. It's sometimes hard to trust general benchmarking (even when independent) due to various A/B tests, separate behaviour per account (one can be throttled due to significant usage already, one can be whitelisted because of marketing reasons), different harness on top of them, different backend serving them (Anthropic servers vs Bedrock/AWS servers), different behaviour per time of day (USA working hours are widely considered to be peak time).
2. Quantifiable -> Single number that tries to answer the question of "how smart is my model this morning".
3. Cheap -> My guiding principle was around 1% of 5h token window in basic Claude/Codex subscriptions. I think we did worse than that but hopefully not by much. Last I invoked /am-i-nerfed medium 180 on my Opus 5.5 medium, it made around 3% damage to my Claude Pro 5h budget.
4. Subscription-first -> Although direct API usage gives much more power to the harness, the subscription-based coding agents (Claude Code and Codex namely) are more likely to be hit with sudden unexplainable model degradation, so it was imperative that the end solution works for them.
5. Trustworthy -> The score has to be somewhat repeatable (in this non-deterministic agent world we live in) and roughly rank the models similar to the public benchmarks available.

FAQ

Why logic puzzles? Given limited token budget, we just can't test for everything, logic puzzles arose as a proxy for intelligence. They have exact answers, and a model can't look up or remember the answer; it has to work it out. It does mean that we are not testing for any other area: coding, knowledge, taste, writing, image generation, tool use, etc.

How many tokens does it use? A medium run uses roughly 5,000–15,000 output tokens.

Is a time limit fair when models can't tell time? It's a somewhat unfair constraint, due to favouring faster backends, but it is easily measurable by our CLI tool (whatever the model/harness/agent) and easily understandable by a human user invoking this command. Despite tokens becoming the new currency, time is still something universally familiar among humans, and timeboxing agent execution to N minutes feels very natural. The timer is exposed to the model through the skill (might switch it to be an MCP tool in the future but don't see a reason so far) and it's crazy how fast models picked up to work with this constraint, especially OpenAI ones.

How noisy is one run? Theoretically quite noisy, practically I have been getting relatively similar scores per each model. If separate runs are around 5-10 point difference between each other, with the underlying model being exactly the same, I think the tool would then do a good job. It's anyway meant as a very very rough sanity check. Basically that you don't suddenly get a Sonnet 5-type of model when using Opus 5.5-type.

How comparable are the scores? Not comparable to other benchmarks or other users of this tool, since everyone gets different tasks. But your individual tasks are frozen post-initialization (running am-i-nerfed reset would get you a new task bank) so you should be able to roughly compare them to your previous runs (you can use am-i-nerfed history list to see previous runs).

License

MIT