At 6:37 AM on a Monday morning, I sent a WhatsApp message: “What’s the status of the re-org plan?”

By 7:30 AM — less than an hour later — we had:

• Created 3 new GitHub repositories
• Extracted a complete cognitive architecture (HOPE recurrence loop, MambaWave neural architecture, memory system, safety kernel) from a 560-file dumping ground into a clean package structure
• Deleted 429 junk files
• Written 24KB of architecture documentation
• Passed 86 tests on real hardware
• Ported an entire training UI
• And the robot brain was running on a Raspberry Pi 5, thinking with Google’s Gemini 2.0 Flash, responding in 860ms

The “we” in this story is me and my AI agent — an instance of Anthropic’s Claude running inside Clawdbot, a framework that gives AI agents persistent memory, tool access, and the ability to spawn sub-agents. Here’s what happened, how the architecture works, and why we built it the way we did.

The Problem: A Brilliant Mess

ContinuonOS is the brain for a personal robot — a 6-foot-reach mobile manipulation platform with dual arms, an OAK-D depth camera, and a Hailo-8 AI accelerator, all running on a Raspberry Pi 5. The vision: a robot that lives in your home, learns from every interaction, and — above all — does no harm.

But the codebase had a problem. The real innovations — the HOPE cognitive loop, the MambaWave neural architecture, the memory consolidation system — were buried inside ContinuonXR, a 1,874-file repository that had become a dumping ground. Debug scripts, training logs, a copy of Gradle 8.7, an unrelated website (worldtapeai.com), and the actual robot brain code all lived side by side.

ContinuonOS, the clean rewrite, existed but was mostly stubs. A 38-line “HOPE dynamics” module that counted keywords. A 63-line safety kernel. An 87-line memory system. The architecture was right, but the implementation was a chatbot wrapper around the Claude API.

We needed to extract the real brain from the mess and transplant it into the clean body. And we needed to reorganize an entire GitHub org while we were at it.

The Architecture: One Brain, Many Shells

Before diving into the reorg, let me explain what ContinuonOS actually is.

The “One Brain, Many Shells” Principle

                    ┌─────────────────────────┐
                    │      ContinuonOS         │
                    │   (The Brain — Pi 5)     │
                    │                          │
                    │  Brain A ← Mind          │
                    │  Brain B ← Executor      │
                    │  Runtime ← Infrastructure │
                    └────────┬────────────────┘
                             │ REST / gRPC / RCAN
              ┌──────────────┼──────────────────┐
              │              │                  │
        Android XR     Flutter Mobile      Web Dashboard
         (Shell)         (Shell)            (Shell)

ContinuonOS is the single brain. Client apps — Android XR headset, Flutter mobile app, web dashboard — are “shells.” They’re thin interfaces that capture sensors and display output. The brain does all reasoning, memory, and safety checking. This means the robot’s intelligence isn’t tied to any particular interface. You can talk to it through an XR headset, a phone, or a terminal. Same brain, same memories, same safety guarantees.

The Ring Architecture

The brain processes information through concentric rings, from fastest/safest (Ring 0) to slowest/most creative (Ring 6):

Ring	Name	Latency	Purpose
0	Safety Kernel	<1ms	Veto unsafe actions. Always runs. Cannot be bypassed.
1	Vision Dreamer	~10ms	Camera frames → tokens
2	Liquid Reflex	~5ms	Fast reactive motor control
3	Mamba Brain	~50ms	Mid-latency SSM reasoning
4	HOPE Recurrence	~200ms	Deliberative cognitive loop
5	CMS Memory	async	Episodic → semantic → procedural
6	MambaWave	varies	Unified neural architecture

Critical design principle: Outer rings can degrade without affecting inner rings. Ring 0 (Safety) never turns off. If MambaWave crashes, the safety kernel still vetoes dangerous commands. If the network drops, local inference keeps working.

Brain A / Brain B: Mind vs. Body

This split is deliberate and philosophical.

Brain A (~7,000 lines) is the mind. It thinks, remembers, learns, decides. It contains the HOPE recurrence loop, the memory system, the safety kernel, and the MambaWave architecture. It’s the part that gets smarter over time.

Brain B (~500 lines) is the executor. It’s deliberately small and auditable. It takes approved actions from Brain A, executes them physically, and logs what happened. Brain B can refuse Brain A’s commands if they violate safety bounds.

Why keep them separate? Because the mind should be free to explore, imagine, and learn — but the body needs hard limits that can’t be talked around. No amount of clever reasoning should bypass the safety kernel. The executor is the last line of defense.

The HOPE Recurrence Loop: How the Robot Thinks

HOPE stands for Hierarchical Oscillatory Processing Engine. It’s the cognitive loop that runs continuously, processing each input through six steps:

1. Encode — Convert input (text, sensor data, reward signals) into a latent representation
2. Read CMS — Query the Continuous Memory System for relevant memories
3. State Update — Update the internal state vector: h_t = f(h_{t-1}, encoded_input, memories)
4. Write CMS — Store new experiences back into memory
5. Learn — On-device parameter update (Hebbian-like gradient step)
6. Decode — Wave-particle rollout produces action or response

The “wave-particle” decoder is one of the more interesting pieces. It runs two parallel paths — a “wave” path (global, spectral, FFT-based) and a “particle” path (local, MLP-based) — and mixes them through a learned gate. The intuition: sometimes you need broad global context (wave), sometimes you need precise local action (particle). The gate learns when to use each.

A Lyapunov stability monitor ensures the dynamics stay bounded. If the system starts to diverge, it triggers graceful degradation rather than letting outputs blow up. The HOPE loop ran 100 steps in 43ms on the Pi 5 during testing — 0.4ms per step, well within budget.

MambaWave: A Custom Neural Architecture

MambaWave fuses two ideas that aren’t usually combined:

Mamba SSM (State Space Models) — excellent at modeling temporal sequences with linear-time complexity. The robot needs to track what happened over time.

Spectral processing (FFT) — excellent at capturing global patterns across the entire input. The robot needs to understand context.

MambaWave runs both in parallel and mixes them through a learned gate:

Input → [Mamba SSM path] ──┐
                            ├── Gated Mix → Output
Input → [FFT Spectral path]─┘

Each ring of the architecture gets a different MambaWave configuration:

• Fast loop (Ring 2, reflexes): 64-dim hidden, 1 block, <5ms
• Mid loop (Ring 3, reasoning): 128-dim, 2 blocks, <50ms
• Slow loop (Ring 4, HOPE): 256-dim, 4 blocks, <200ms
• Seed loop (Ring 6, full model): 512-dim, 8 blocks, <500ms

PyTorch is an optional dependency. When not installed, Brain A falls back to cloud LLM inference (Gemini, Claude) without MambaWave. This means the robot works out of the box — cloud for reasoning, with the option to run a local neural model when you train one.

The Memory System: How the Robot Remembers

The Continuous Memory System (CMS) is a three-tier architecture modeled on human memory:

L0: Episodic Memory (decay rate 0.9) — Every interaction becomes an RLDS episode. Most episodes fade quickly. This is your working memory, your recent experiences.

L1: Semantic Memory (decay rate 0.99) — Patterns extracted from episodes during “sleep cycles” (consolidation). Facts, knowledge, learned associations. Fades slowly.

L2: Procedural Memory (decay rate 0.999) — Learned skills and behaviors. Barely fades at all. Requires human approval for promotion. This is critical: the robot can’t teach itself dangerous skills without your sign-off.

During idle periods, the consolidation module runs a “sleep cycle”:

1. Scans L0 episodic memories
2. Extracts recurring patterns
3. Promotes significant patterns to L1
4. Applies decay to low-importance memories
5. Proposes L2 skill promotions (awaiting approval)

There’s also a spatial memory module with an occupancy grid, A* pathfinding, landmark tracking, and object memory. The robot builds a map of its environment over time.

The Safety Kernel: Do No Harm

This is the most important part of ContinuonOS, and it’s designed to be boring. No clever tricks, no emergent behavior, no learning. Just hard rules that cannot be overridden.

The safety kernel includes:

• Physical bounds — Velocity caps, cliff detection, workspace limits, joint torque limits
• Anti-subversion — Prompt injection detection, jailbreak pattern matching, tamper-evident audit log
• Work authorization — Permission levels for different action types. Destructive actions require explicit approval
• Protocol 66 — Emergency full shutdown. All actuators disabled, system enters lockdown, human intervention required to restart
• Continuous monitoring — CPU temperature, memory usage, force sensor readings, watchdog heartbeat

The principle is simple: Ring 0 always wins. If Brain A reasons brilliantly that it should do something unsafe, Ring 0 vetoes it. If MambaWave’s neural output suggests a dangerous action, Ring 0 blocks it. If someone tries to jailbreak the robot through conversation, Ring 0 detects the pattern and flags it.

The safety kernel is a singleton — there’s exactly one instance, initialized at boot, running until shutdown. It cannot be replaced, mocked, or bypassed by any other module.

The Inference Pipeline: Cloud-First, Local-Ready

ContinuonOS uses a pluggable inference backend. Right now, the robot thinks with Google Gemini 2.0 Flash via API — fast (860ms average response), capable, and effectively free with a Google AI Pro subscription.

But the architecture supports four backends:

Backend	Use Case
Gemini (default)	Fast cloud inference, good for everyday conversation
Claude (Anthropic)	Higher-quality reasoning when needed
LlamaCpp	Fully offline local model — no internet required
Mock	Testing and CI

The vision: start with cloud inference (Gemini), accumulate training data as RLDS episodes, train a custom MambaWave model in the cloud, and deploy it back to the device via OTA update. Over time, the robot shifts from cloud-dependent to locally intelligent.

The Toolchain: How AI Built an AI

Here’s where the meta gets interesting. The entire reorganization was orchestrated through WhatsApp messages to an AI agent.

The Stack

Clawdbot — An open-source framework that wraps Claude (or other LLMs) with persistent memory, tool access, and sub-agent spawning. It runs on the Pi 5 itself, connecting to me via WhatsApp. When I message it, Claude processes my request with full access to the filesystem, git, shell commands, and the ability to spawn background workers.

Anthropic Claude (Opus 4.5) — The main agent brain. It reads code, understands architecture, makes decisions, and writes clean Python. It ran as my primary agent through Clawdbot.

Sub-agents — Clawdbot can spawn isolated sub-agent sessions for parallel work. During the reorg, up to 3 sub-agents ran simultaneously:

• One creating the continuon-proto repository
• One cleaning ContinuonXR (deleting 429 files)
• One porting the HOPE architecture

Each sub-agent gets its own session, works independently, and reports back when done. The main agent coordinates, reviews, and handles merge conflicts.

Google Gemini 2.0 Flash — Powers the robot brain’s inference (not the development agent). ContinuonOS calls the Gemini API for its reasoning, making the robot’s “thoughts” powered by a different model than the one that wrote its code.

Claude Code OAuth — We initially tried to use Claude Code’s OAuth token for the robot brain, but discovered the standard Anthropic Messages API doesn’t support OAuth authentication. That’s what led us to add the Gemini backend — a bug that became a feature.

The Development Flow

Craig (WhatsApp) → Clawdbot → Claude (Opus 4.5)
                                    │
                              ┌─────┼──────┐
                              │     │      │
                        Sub-agent  Sub    Sub
                        (proto)   (XR)  (HOPE)
                              │     │      │
                              └─────┼──────┘
                                    │
                              Git push → GitHub

The entire session — from first message to all 7 phases complete — was a conversation. I described what I wanted in plain language. Claude analyzed the codebase, made architectural decisions, wrote code, ran tests, and pushed to GitHub. When something broke (like the HOPE config mismatch after Phase 3), it diagnosed and fixed the issue.

The sub-agent pattern was crucial for speed. Phases that would normally be sequential — create schemas, clean up old code, port brain modules — ran in parallel. Three sub-agents porting HOPE, MambaWave, and CMS simultaneously, each confined to non-overlapping directories to avoid merge conflicts.

The Reorg: Seven Phases in One Morning

Here’s what actually happened, phase by phase:

Phase 0: Get It Running

ContinuonOS was already set up as a systemd service from the night before. We switched the inference backend from mock to Gemini, discovered the Anthropic API key was depleted, added the Gemini backend (writing a complete GeminiBackend class with the google-genai SDK), and had the robot responding in 1.4 seconds.

Phase 1: Shared Schemas (continuon-proto)

Created a new repo with protobuf schemas extracted from ContinuonXR, repackaged under the continuon.* namespace. Four proto packages: RLDS episodes, gRPC brain service, RCAN types, and edge manifests. Plus pip-installable Python dataclasses so ContinuonOS can use the schemas without needing a protobuf compiler.

Phase 2: Clean ContinuonXR

Tagged the current state as v0-pre-reorg (for safety), then deleted 429 files: debug scripts, verify scripts, training logs, temp directories, an entire copy of Gradle 8.7 that somehow got committed, and worldtapeai.com. Created a MIGRATION.md documenting 12 items that need to move to other repos.

Phase 3: Extract the Brain (The Big One)

This was three sub-agents running simultaneously:

HOPE Architecture — 12 new modules replacing a 38-line keyword counter. The real 6-step recurrence loop with FFT-based spectral processing, Lyapunov stability guarantees, intrinsic curiosity, and on-device learning. Pure numpy — no PyTorch required.

MambaWave — 9 modules with the SSM + spectral fusion architecture. PyTorch optional — the package imports cleanly without it, raising clear errors only when you try to run torch operations.

CMS Memory + Safety Kernel — 11 modules total. Memory went from a skeleton to a real system with search/scoring, consolidation, spatial memory, and approval-gated procedural learning. Safety went from basic velocity checks to a complete Ring 0 with anti-subversion, authorization, Protocol 66, hardware monitoring, and workspace bounds.

Phase 4: Infrastructure

gRPC server on port 50051 (5 RPCs for streaming robot state, teleop commands, capability manifests). RCAN/mDNS discovery broadcasting _rcan._tcp.local. so client apps can find the robot automatically. GitHub Actions CI workflow (created but needs a token scope fix to push).

Phase 5: Cloud Backend

Scaffolded continuon-cloud with episode ingest (FastAPI), training pipeline (Vertex AI), model packaging (OTA bundles), and Firestore command relay. The full episode pipeline: device → cloud upload → validation → training → model update → device.

Phase 6: Documentation

A 24KB ARCHITECTURE.md with ASCII diagrams covering every layer. Ecosystem map showing data flows between repos. Rewritten README. Updated reorg plan with completion status.

Phase 7: Testing

86 tests passing in 0.35 seconds. Hardware detection confirmed Hailo-8 NPU, OAK-D camera (MyriadX), Bluetooth, GPIO, SPI, USB speaker. HOPE loop stress test: 100 steps in 43ms at 48.5°C — well under the 80% CPU / 75°C limits. Chat pipeline averaging 860ms responses.

Bonus: Trainer UI Port

Ported the entire training web interface (20 files, ~12.5K lines): hardware auto-detection, OAK-D depth camera integration, Hailo-8 NPU scanning, drive/arm controls, and RLDS episode recording. All adapted to use ContinuonOS packages.

The Principles: Why This Architecture

Three principles guide every design decision in ContinuonOS:

1. Live With Humans

This robot is designed to share space with people. Not in a factory, not in a lab — in a home. That means:

• It should be approachable, not intimidating
• It should learn from natural interaction, not from programming
• It should adapt to your preferences, not the other way around
• Its memory system mirrors human memory because that’s what makes sense in a human environment

2. Do No Harm

The safety kernel exists because this principle isn’t negotiable. Protocol 66 exists because sometimes you need an absolute shutdown. The approval-gated procedural memory exists because the robot shouldn’t learn dangerous behaviors without human oversight.

Ring 0 always wins. Not sometimes. Not usually. Always.

3. Learn Continuously

Every interaction is training data. The robot gets better over time — not through software updates, but through experience. The episode pipeline captures everything: what was said, what was observed, what was done, what the HOPE state looked like, what safety flags were raised.

Over weeks and months, the CMS consolidates this into knowledge and skills. The cloud training pipeline refines the neural model. The robot that greets you in six months is smarter than the one you meet today — because it’s been learning from living with you.

What’s Next

The physical robot is waiting for parts. Mecanum wheels, SO-ARM101 dual arms, V-slot aluminum mast, ESP32-C6 for the face display — all ordered, not all arrived. While we wait:

• Accumulate training data — ContinuonOS runs 24/7 on the Pi, generating RLDS episodes from every conversation
• Rename ContinuonXR → continuon-clients — reflects its new scope as the client app repo
• Train the first MambaWave model — once we have enough episodes, run the cloud training pipeline
• OAK-D integration — the camera is connected but needs pipeline testing with the trainer UI
• Hailo-8 inference — the NPU is ready, just needs model compilation for on-device vision

The architecture is proven. The tests pass. The brain is thinking. Now we build the body.

---