Skip to main content

Why the ContinuonAI Experiments Feel Different

12 min read By Craig Merry
ContinuonAI ContinuonAI Robotics Lab Notes HOPE Cognitive Architecture

Day 1 was swapping the Jetson Nano for a Raspberry Pi 5 and sketching the skeleton of a robot brain that could learn on-device. Since then, I’ve been reviewing what we actually shipped—and just pulled the latest code. The scope shifted: we’re not just building a robot anymore. We’re testing a complete cognitive architecture on a Pi 5. Here are the hypotheses that emerged.

Hypothesis 1: Mission statements should be code, not slides

Most robotics teams write their safety rules in a slide deck and hope engineers remember them. We’re testing the opposite: what if the robot literally can’t boot without loading its mission statement first? Every rule—“don’t harm people,” “prefer local reasoning before cloud calls,” “cap API spend at $5/day”—gets merged into the startup sequence before any motor can twitch. The hypothesis: if ethics and constraints are executable artifacts, they can’t drift from the implementation.

Early result: it works. Gemma inherits the same context as the servo controller. The $5 budget isn’t a Slack reminder; it’s a guardrail the scheduler checks before reaching out to Gemini.

Hypothesis 2: Hardware tutorials fail because they’re tribal knowledge

I’ve set up enough Raspberry Pis to know that “just follow the forum post” means four hours of debugging I²C permissions and camera enumeration. So we wrote a validation script that runs the entire bring-up checklist: does the servo controller respond? Can we grab a depth frame? What’s the timestamp skew between the camera and the servo pulse?

The hypothesis: if bring-up is one command that outputs pass/fail, anyone can reproduce the robot. Target: under 5 milliseconds of sensor-to-actuator latency before we trust autonomy. So far, the script catches I²C misconfigurations and missing camera drivers before they turn into mystery bugs.

Hypothesis 3: Discovery should be zero-config

Most home robots require you to SSH in, find the IP, and type it into a control app. We’re testing mDNS/Zeroconf with a UDP fallback: the robot broadcasts its name, firmware version, and API endpoints every five seconds. Open the app, and nearby robots just appear in a list.

The hypothesis: if discovery is automatic, tele-op becomes as easy as AirDrop. This week, it worked across my home network. Next test: multiple robots competing for the same subnet.

Hypothesis 4: Cloud-optional beats cloud-required

Gemini is brilliant, but Wi-Fi isn’t always reliable in a garage. We added a local Gemma 3 Nano model that runs on the Pi 5’s CPU with quantized weights. The hypothesis: a 2B-parameter model that’s always available beats a 1.5T model that times out. The Gemma setup maintains chat history, answers safety questions, and degrades gracefully when transformers aren’t installed (mock responses keep the UI alive).

Early data: latency jumps from 200ms (Gemini cloud) to 1–2 seconds (Gemma local), but when the internet drops, local wins every time.

Hypothesis 5: Real-world robot data includes sound

Most manipulation datasets are silent: RGB, depth, joint angles, done. We’re capturing synchronized audio alongside every frame—voices, ambient noise, the creak of a servo under load—and logging the millisecond delta between the audio block and the video timestamp. The hypothesis: if we treat home labs like messy reality instead of clean benchmarks, the model learns context cues that lab datasets miss.

When the recorder sees a delta over 5ms, it warns us. We’re aiming for tight enough sync that a spoken “stop” command lands in the same step as the hand gesture.

Hypothesis 6: Dashboards double as scientific instrumentation

The ContinuonAI Flutter app communicates with the ContinuonBrain Python API server over HTTP/WebSocket at 0.5 Hz (every two seconds), requesting a structured status payload that includes robot mode state, detected hardware manifests, motion gate status, and resource utilization metrics. The dashboard renders this as a deterministic card layout—no animations, no stochastic UI elements—so a screenshot at timestep t provides complete experimental context for that observation.

The hypothesis: if telemetry is deterministic and structured, dashboards become instruments, not decoration. When comparing ten hallway navigation runs, I can diff screenshots to isolate which runs had the OAK-D depth camera online, which ran under thermal throttling, and which hit the emergency stop threshold. The UI becomes part of the experimental apparatus.

Measured observables:

  • Mode state: {idle, manual_control, manual_training, autonomous, sleep_learning, emergency_stop}
  • Hardware manifest: detected sensors (I²C address, device type) and actuators (PWM channel assignments)
  • Resource monitors: CPU % (1-min avg), memory MB (RSS), thermal °C, battery %
  • Learning state: active checkpoint ID, validation rate %, episodes logged

This structure makes the dashboard falsifiable: if I claim “Run #7 failed because the servo controller was offline,” the screenshot either shows servo_controller: null or it doesn’t.

Hypothesis 7: Constraints make intelligence scientific

We’re not bolting features onto the robot; we’re enforcing hard constraints that shape the learning dynamics. The hypothesis: if you design intelligence like a controlled experiment—with budgeted compute, rollback checkpoints, and falsifiable invariants—you get reproducible results instead of “it worked once but I don’t know why.”

Constraint hierarchy:

Fast loop (control-rate, 20–50 Hz):

  • Parameter updates Δθ bounded by ‖Δθ‖₂ ≤ ε_fast (typically ε_fast = 10⁻⁴)
  • Runs on every observation; updates bias terms, normalization stats, small memory vectors
  • No disk I/O, no network calls, no gradient computation
  • Enforced: loop time < 50ms or skip update

Mid loop (episodic, ~10 minutes idle):

  • Bounded LoRA training: max_steps ≤ 500, max_wall_time ≤ 300s, max_memory ≤ 1.5 GB
  • Reads N episodes from local RLDS (N ∈ [16, 256])
  • Gradient steps limited: total_flops < budget_flops
  • Promotion gate: reject if safety_violations > 0 OR avg_action_delta > 0.25
  • Enforced: thermal < 75°C, battery > 40%, no active teleop

Slow loop (overnight, consolidation):

  • Compact memory: merge episodes, prune low-relevance entries
  • Update semantic CMS level: write consolidated context embeddings
  • Checkpoint rotation: factory → last_good → current
  • Enforced: time budget ≤ 2 hours, thermal < 70°C, network idle

Rollback protocol:

If validation_rate < threshold_min OR lyapunov_energy diverges:

  1. Halt learning
  2. Restore checkpoint: last_good → current
  3. Flag for operator review
  4. Require manual approval before resuming learning

This isn’t feature engineering. It’s experimental design: every learning loop has measurable invariants, falsifiable promotion criteria, and deterministic rollback.

Hypothesis 8: Design for one, architect for many

Right now, it’s one DonkeyCar on my desk. But the roadmap pushes toward fleet tooling: shared safety protocols, OTA adapter updates, and a control surface that can manage twenty robots. The LAN discovery service already broadcasts enough metadata (firmware version, capabilities, endpoints) that a second Pi 5 rig would auto-register.

The hypothesis: if you architect for N>1 from day zero, the jump to fleet-scale is a deploy step, not a rewrite. We’re not there yet, but the telemetry hooks and constraint system are designed to scale.

Hypothesis 9: One brain, many shells

This week’s most interesting commit was the “hybrid architecture” PRD. Instead of building separate control apps for web, mobile, desktop, and XR, we’re testing a different model: one robot brain (ContinuonBrain on the Pi 5) accessible from any shell—Flutter on your phone, browser failover on your laptop, native XR when you’re wearing a headset.

The hypothesis: if the brain exports a consistent API and every shell shares the same safety contracts, the control surface becomes disposable. When the native app crashes, the web client takes over. When Wi-Fi drops, the XR headset’s local cache keeps logging. When you’re away from the lab, the mobile app still shows live telemetry.

Early architecture decisions that support this:

  • The brain broadcasts discovery on mDNS so any shell can find it
  • Emergency stop and mode switching work identically across all clients
  • RLDS recording can happen browser-side (IndexedDB) or native (SQLite), but the schema is identical
  • Offline detection and retry queues are built into the client package, not the shell UI

The marketing page now describes ContinuonBrain as a system that “runs everywhere—on-device, on-robot, and in the cloud.” That’s not marketing speak; it’s the actual design constraint. If we’re right, you’ll be able to drive the robot from an iPhone in the garage, switch to browser control when the battery dies, and still have XR recordings upload seamlessly when you plug the headset back in—because the brain stayed consistent and the shells stayed thin.

Hypothesis 10: A cognitive architecture fits on a Pi 5

This morning I pulled the latest code and found something I wasn’t expecting: we shipped a complete implementation of HOPE (Hierarchical Online Predictive Encoding), a wave-particle hybrid brain with a formal Continuous Memory System specification. Not a prototype—a production-ready cognitive architecture with Lyapunov stability proofs, optimized for the Pi 5.

The hypothesis: if you design a hybrid dynamical system from first principles—wave dynamics for global coherence, particle dynamics for local adaptation, hierarchical memory with provable bounds—you can run real cognition on edge hardware without cloud dependency.

The HOPE architecture

State dynamics (wave-particle hybrid):

At timestep t, given observation x_t, previous action a_{t-1}, and reward r_t:

  1. Encode input: e_t = Encoder(x_t, a_{t-1}, r_t) ∈ ℝ^{d_e}

  2. Query CMS memory: q_t = Q_ψ(s_{t-1}, e_t), retrieve context c_t via attention over L+1 memory levels

  3. Wave subsystem (global linear dynamics):

    • w_t = A_w w_{t-1} + B_w [e_t || c_t]
    • Properties: eigenvalues λ(A_w) lie in stability region, provides temporal coherence
  4. Particle subsystem (local nonlinear updates):

    • p_t = σ(W_p [s_{t-1} || e_t || c_t] + b_p)
    • Local feature extraction, nonlinear transforms, attention routing
  5. Hybrid state update:

    • s_t = w_t + p_t (additive coupling of wave + particle components)
  6. Output decoder: y_t = Decoder(s_t, c_t) → predictions, actions, or policy logits

  7. CMS write: Update memory levels M^(ℓ) with exponential decay d_ℓ and attention-weighted write

Continuous Memory System (CMS)

Memory hierarchy: L+1 levels (ℓ = 0, 1, …, L) ordered by decay rate:

  • Level ℓ: M_t^(ℓ) ∈ ℝ^{N_ℓ × d_ℓ} (memory matrix), K_t^(ℓ) ∈ ℝ^{N_ℓ × d_k} (key matrix)
  • Decay coefficients: d_0 > d_1 > … > d_L (faster levels decay more aggressively)
  • Timescale separation: ℓ=0 is episodic (~seconds), ℓ=L is semantic (~days)

Read operation (content-addressable retrieval):

For each level ℓ, compute attention weights:

  • α_t^(ℓ) = softmax(K_t^(ℓ) q_t / √d_k) ∈ ℝ^{N_ℓ}
  • Retrieve context: c_t^(ℓ) = (α_t^(ℓ))^T M_t^(ℓ)
  • Hierarchical mixing: c_t = Σ_{ℓ=0}^L β_t^(ℓ) U^(ℓ) c_t^(ℓ)

Write operation (bounded memory growth):

Memory update with exponential decay and gated write:

  • M_{t+1}^(ℓ) = (1 - d_ℓ) M_t^(ℓ) + η_t w_t (v_t)^T
  • Decay term (1 - d_ℓ) ∈ (0,1) ensures bounded memory: ‖M_t^(ℓ)‖_F ≤ C for all t
  • Write gate η_t ∈ [0,1] learned via prediction error or novelty signal
  • Value v_t = V_φ(s_t, e_t) computed by value network

Formal guarantees:

  1. Bounded growth: ‖M_t^(ℓ)‖_F ≤ ‖M_0^(ℓ)‖_F / d_ℓ + C_write / d_ℓ²
  2. Exponential decay of old memories: influence of M_τ decays as (1 - d_ℓ)^{t-τ}
  3. Lyapunov stability: V(s_t) = ‖s_t - s*‖² + Σ_ℓ γ_ℓ ‖M_t^(ℓ) - M*^(ℓ)‖_F² is non-increasing under learned dynamics

Nested learning (slow parameter adaptation)

While fast state (s_t, M_t) updates at control rate, slow parameters Θ adapt via low-rank updates:

  • Θ_{t+1} = Θ_t + α ΔΘ_LoRA
  • Rank constraint: rank(ΔΘ) ≤ r (typically r=8)
  • Validation gate: reject update if validation_loss increases or safety_score decreases

What’s actually running

  • HOPE Brain: d_s=256, d_w=256, d_p=128, num_levels=3, cms_sizes=[64,128,256]
  • Pi 5 performance: >10 steps/second, <2GB memory, INT8 quantization enabled
  • Agent Identity: Introspects body (sensors/actuators), memory (episode count), design (mission), shell (Pi 5 vs Jetson)
  • Hierarchical LLM fallback: 67% of queries answered by HOPE directly; Gemma 2B/4B fallback; Gemini cloud as last resort
  • Background Learning: Autonomous memory consolidation, bounded LoRA updates, checkpoint rotation with validation gates

Empirical validation: 12 conversations stored, adapters autonomously promoted, Lyapunov energy shows convergence over 6,000+ autonomous training steps. The learning dashboard plots energy decay, validation rate, and memory utilization in real-time.

This isn’t a robotics project anymore. It’s a cognitive architecture research project with formal proofs, reproducible experiments, and a chassis that happens to have wheels.

The common thread

Every experiment shares one assumption: autonomy should feel more like a lab protocol than a product sprint. Safety rules are non-negotiable. Hardware setup is scripted. Data provenance is tracked. Intelligence budgets are capped. Dashboards log conditions. The brain is portable, the shells are disposable. And now: cognition itself is reproducible, testable, and formally specified.

The HOPE implementation includes complete mathematical derivations, Lyapunov stability proofs, and computational complexity analysis. The CMS memory update rule has formal specifications for bounded growth and hierarchical decay. The background learner enforces strict resource budgets and checkpoint rotation. Every hypothesis—from mission-as-code to one-brain-many-shells to edge-first cognition—traces back to testable, falsifiable claims.

If this works, ContinuonAI won’t just be a robot—it’ll be a reproducible cognitive architecture accessible from any device, where anyone can validate the Pi 5 deployment, watch HOPE learn in real-time, and replicate the same memory consolidation experiments we’re running. That’s the hypothesis worth chasing.

Next up: wire the dual-arm servos when they arrive, test the first autonomous hallway run with HOPE’s world model predicting obstacles, and see if the hierarchical memory system can recall spoken commands from hours ago. And maybe the most exciting test: let the background learner run overnight, check the checkpoint files in the morning, and measure if the validation rate improved without any cloud calls. If the experiments stay this weird—and this rigorous—I’m in.