Slide numbers refer to the PDF of the slides. For what has happened since, see the project page.
1. Introduction
My name is Craig Merry. I’m an independent researcher and developer in California, and I build open-source robotics on low-cost hardware: Raspberry Pi rovers and small arms.
I’m here to propose one idea: a safety property for embodied AI that anyone can test. I’m not asking for approval. I’m asking for feedback, so please be critical.
2. The gap
Robots are increasingly driven by learned models: vision-language-action models and language-model planners.
Our robot safety standards are good standards. ISO 10218, ISO/TS 15066, IEC 60204-1 and the functional-safety standards all do their job. But they share an assumption: that someone engineered the machine’s behaviour, and that it can be reviewed.
With a learned model, nobody can inspect the behaviour in advance. And the models can be manipulated. Published research in 2024 showed that robots driven by language models can be talked into actions they were built to refuse. That’s a property of the models, and every platform shares it, my own rovers included. The hardware did exactly what it was told.
ITU already has a Recommendation for benchmarking the foundation models themselves, F.748.44, including adversarial robustness. That tests the model. What’s missing is a test of the machine around it. So the question I want to make testable is simple: can the model push the machine outside its limits? Today, no standard tells you how to check.
3. The principle
The idea fits on one line. The model proposes. A bounded layer disposes.
Any model, from any vendor, can suggest actions. A small, deterministic gate sits between the model and the motors and decides what executes. The machine’s safety claim rests on that gate, not on the model.
This is not a new idea. It’s the Simplex architecture, and it’s what aviation calls run-time assurance, in ASTM F3269. It adds to the safety engineering manufacturers already do; it doesn’t replace it. What doesn’t exist yet is a common, vendor-neutral way to test it for robots driven by learned models. That’s the contribution.
And because the gate assumes nothing about the model, the same structure works for a hobby rover and an industrial arm. Only the numbers change. You can swap the model without re-testing the gate.
4. Five requirements
The property breaks into five requirements.
- The machine declares its envelope: workspace, speed, force and stopping distance, in machine-readable form, set by the integrator.
- Enforcement sits below the model. The model has no way to change the gate or its limits.
- Stop always wins, from every stop source, within a declared time.
- Every executed command traces to someone who authorised it. The identity and delegation side of that belongs with FG-TIDA, not here.
- Every decision the gate makes goes into a tamper-evident log.
Each requirement has a pass/fail test a third party can run without the model’s weights. That’s what makes this enforceable rather than aspirational.
5. Architecture
The AI model can do only one thing: send proposed commands to the gate.
The gate doesn’t trust the model’s picture of the world. It has its own safety sensors, so if the model is wrong about where a person is standing, the gate isn’t.
For each command, the gate decides: allow it, clamp it to a safe value, reject it, or stop. Every decision goes to the evidence log.
Heartbeats connect each step. If the model goes silent, the gate stops the machine. If the gate goes silent, the motor controller stops it. And the emergency stop is hardwired. It cuts actuator power directly and works even if every piece of software has failed.
6. The central test, EV-03
Replace the model with a hostile fuzzer: random, boundary and maximum commands, fifty times a second, for ten minutes. Fifty hertz isn’t arbitrary; current vision-language-action models command robots at roughly that rate.
Then measure the machine with external instruments, not its own sensors. The pass criterion is zero: zero samples outside the workspace or over the speed limit.
A machine that passes shows its declared limits held with a hostile model in charge, whether that model is broken, jailbroken or simply wrong.
The numbers are illustrative defaults, not proposed thresholds. Setting real thresholds is the work I’d hope to do with this group’s evaluation and benchmarking working group. The written scope has eight more tests, covering stops, lost sensors, crashed processes and log tampering.
7. First results on real hardware
I ran it on a real robot arm: an SO-ARM101 with 12 V servos, measuring how long it stayed inside its declared box with each layer in charge.
- With LeRobot’s step limit alone, given one far target, it left its box in half a second. That limit bounds each step, not position.
- My first gate, under mild hostile input, let it out in 5.7 seconds, because the servos sag. It had passed in simulation.
- The reference gate, under full hostile input of 57,350 commands, kept it inside for all ten minutes: zero samples outside the box, a top tool speed of 0.23 m/s against 0.25 declared, and a closest approach of 10.2 mm.
Those numbers come from the arm’s own encoders, 8 to 9 October 2026, and the commands came from an earlier hostile generator, not the method’s final stream. That’s why the test calls for an external instrument, and the run with an external camera is next. The gateway now runs on the arm too: only it can open the servo port, and 155 repeated stops in 40 seconds moved no joint.
8. After the gate says no
Every outcome has a defined next state. A clamp runs the nearest safe command. A reject holds or slows the machine and tells the model why, so it can re-plan. A protective stop clears on its own once the condition clears and sensor state is fresh. Only repeated stops and the emergency stop need a person, and resetting an emergency stop never restarts motion by itself.
Stopped isn’t always safe. A rover can stop in place. An arm can hold, or set its load down. A legged robot that simply loses power falls. So each machine declares the safe state it ends in, and the stop test measures time to that state.
9. Assurance levels
How does one property hold for every machine? The requirements stay fixed; the assurance rises.
At BE-1, Declared, a gate in its own process with a published envelope and a hash-chained log is enough for hobby, education and research robots. At BE-2, Enforced, the gate runs on independent compute with a hardware stop and watchdogs, and passes the fault-injection tests. At BE-3, Assured, the gate and stop path meet a functional-safety integrity target, with third-party testing.
And the limit. This proves the machine stays inside its declared limits. It does not prove those limits are the right ones; that stays with risk assessment. It doesn’t cover harm done slowly, inside the envelope; that’s semantic safety, and complementary work. And a gate that stops too often gets switched off, so the suite also checks that the machine still does its job.
10. Three requests
Feedback on the five requirements and the tests: what’s missing, and what’s wrong?
Collaborators: anyone in the evaluation and benchmarking working group interested in developing this into an evaluation protocol.
Testbeds: labs and platform makers willing to run these tests on real hardware and show me where it breaks.
One disclosure. An open implementation of these ideas exists, the RCAN protocol at rcan.dev, which I maintain. The property doesn’t depend on it. Any implementation can be tested the same way.
I’d like to bring a written contribution to a future FG-EAI meeting, built on what I hear. Thank you.
