What it is
Fitness Report asks a question the existing MCP benchmarks do not: not which model is best at using tools, but which server is worth using. It connects to a remote MCP server, generates a task suite from that server’s own tool schemas, drives the tasks with a real agent, and publishes first-try success with a confidence interval, per-tool failure attribution, and a replayable recording of every frame that crossed the wire.
Before any of that, validity gates run against the harness’s own generated tasks. When the measurement itself is invalid, the run publishes a refusal instead of a number, and the refusal names the gate that stopped it. That behavior is the product. A leaderboard you can trust has to be able to say: this measurement was not valid.
The first pass scored nothing
I ran fourteen public MCP servers, the ones reachable without credentials, including servers from several companies whose names you would recognize. Every single one refused.
Seven refused at the null baseline gate, which measures whether a model with no tools at all can pass the tasks. It could, on 75 to 100 percent of them. The task generator had been writing encyclopedia questions. A crypto data server got asked for Bitcoin’s genesis date, which the model already knew, rather than the current price, which only the server can supply. Seven more refused for producing too few admissible tasks to measure anything. One failed the construct gate outright.
That last one is the finding I would not have gotten any other way. A major cloud provider’s documentation server advertises five tools, and every single call to any of them is rejected at the HTTP layer by its own gateway, with the message “Http operation is not supported for gateway protocol type MCP.” I confirmed it with plain curl outside the harness. The published tape carries fifty-four recorded rejections, each naming the tool and the status. The server looks healthy in a tool list and cannot be driven at all.
Every server is named on the board, with its tape attached. I am describing findings here rather than reciting a list, but nothing is anonymized: the point of publishing the recordings is that you do not have to believe me.
Making the tasks harder, never the gates easier
The fix had one rule: when a gate fails, you fix the tasks, not the gate. Loosening a threshold to get a prettier board would have made the whole thing worthless.
So the generator was rewritten to produce tasks that require the server: live and volatile data, verbatim quotes from long-tail content, entity lookups a model cannot recall. Every candidate task now faces a cold probe from a no-tools model before anything is paid for, and any task the cold model answers correctly is deleted with its reason recorded. The gate thresholds never moved.
It worked. Across the reruns, all three null models scored zero out of twelve on every suite. Same servers, same gates, harder tasks. Seven servers then earned scores ranging from 75 to 100 percent first-try success, and the spread is real: the server at 75 percent has three recorded failures you can replay, and the ones at 100 percent have none.
One run resolved through the extension protocol, which is the part I am most pleased with. A server’s reference pass came in at 10 of 12, below the threshold but not statistically decisive. Rather than guessing, the harness bought two pre-registered batches of six more tasks, pooled the counts to 22 of 24, and only then let the gate resolve. The batch size and the maximum number of extensions are fixed in code before the first call and reachable by no command-line flag, because an extension budget you can raise mid-run is just optional stopping with extra steps.
The harness caught me too
Partway through I noticed my own spending ledger was using a flat estimate for the task generator’s cost, because nothing measured it. When I made it measure, the real figure was nearly three times my guess. Every total I had reported was low.
I fixed the accounting and relabelled the affected numbers as floors rather than totals, which is the same discipline the product sells. The published site now says “at least” wherever a cost is known to be incomplete, and says nothing numeric at all where it cannot be derived. Finding that in my own bookkeeping, in a project about refusing to publish numbers you cannot support, was a useful moment.
How it is built
TypeScript on Node, running from a Raspberry Pi, paired with Claude Code throughout. The validity gates are ported from evalgate, the tooling that came out of my earlier FOIL work, including the rule that a gate you have to remember to honor is not a gate. Sessions are recorded in the mcp-tape format and replayable in a viewer at mcpreplay.dev, so every finding on the board links to the exact frames that justify it rather than asking you to take my word for it. The leaderboard is a static site on Cloudflare Pages.
Total spend across all 32 runs, scored and refused: about 23 dollars.
Why it matters
There are more than a hundred thousand MCP servers listed across the public registries, ranked by stars and listing age. Not one of those leaderboards runs the servers. Meanwhile the serious academic benchmarks in this space all measure the model, not the server, which leaves the question a person integrating a server actually has unanswered.
The harder problem is that an evaluation which never refuses is not measuring anything. It has no way to distinguish a server that works from tasks that were too easy, and the failure mode is invisible: an invalid environment produces results that look perfectly analyzable. My first pass would have published fourteen scores if I had let it. They would have been measuring the model’s memory, not the servers, and nobody reading the board could have told the difference.
So the refusals stay on the board next to the scores, each naming the gate that stopped it, with the evidence attached.