What it is
FOIL started as a response to Anthropic’s Frontier Red Team article on emerging multiagent systems. I set out to measure how much an AI agent’s decision depends on each of its information sources. The project killed its own original thesis in the first week, and what survived is more useful: a demonstration that the order of a model’s inputs can matter more than their content, and the tooling to catch that before it corrupts a study.
The headline number: two orderings of four byte-identical scout reports, differing by a single adjacent swap, scored 1.000 and 0.005 on the correct answer. Same sentences, same characters, only the sequence changed.
The failure record is the point
I built eight evaluation environments across eleven pre-registered studies. Five environments were invalid, and every invalid one produced results that looked perfectly analyzable. One passed a rigorous structural gate over 200 seeds and was still wrong, because the scoring key disagreed with what the model, reasonably, concluded. I was one write-up away from publishing that as a finding.
Everything is published at the same standard: the kills, the gate failures, a claim I had to retract after checking 1,855 real agent sessions, and a model sweep where the pre-registered rule forbade the trend claim the numbers would have tempted me into. Every hypothesis, threshold, and failure outcome was committed to version control before its data existed.
The tools
Two zero-dependency Python libraries came out of the failures rather than the findings.
orderprobe is a runtime check that needs no ground truth: permute a decision’s inputs, and if the answer moves, stop and escalate. In a pre-registered test it caught 77 percent of errors while flagging 29 percent of decisions. Its limits ship in the README, because stability is not a guarantee: 3 of 34 stable answers were still wrong.
evalgate is four checks that prove an eval environment can measure what you claim before you pay to run it, plus all five of my invalid environments packaged as fixtures so you can test your own validity gate against environments known to deserve failure. Its gate planner computes, for free, the sample sizes that three of my own studies failed for lacking.
Why it matters
Multiagent AI systems assemble context from many sources, and the order those sources arrive in is usually an accident of scheduling. If the conclusion depends on that order, part of the answer is an artifact of the scheduler. The sweep result sharpens it: across four model generations, accuracy rose 28 points and the inversions did not go away. Whatever newer models bought, it was not immunity to input order.
The deeper argument is about research practice. Negative results and invalid environments are usually discarded, which means every new team pays to rediscover them. Publishing mine, as data and as importable test fixtures, cost nothing extra and is the part of this project most likely to save someone else money. Total spend for all of it: about 55 dollars.