Skip to content

PRD: Gemma 4 brain integration into Vector AI system #1

@chatde

Description

@chatde

Overview

The Vector AI Brain at /Volumes/AI-Models/vector/ already has a production-grade autonomous system:

  • Priority-based BehaviorArbiter (SAFETY > ONBOARD > AI > IDLE)
  • Persistent MemoryBank with Jaccard scene similarity
  • LearningEngine for autonomous environment discovery
  • MissionController (SecurityPatrol, ObjectCensus, RoomCheck, PresenceWatch)
  • SpatialMemory for room mapping
  • VoiceHandler for voice commands

Current AI stack: LLaVA (vision) + Llama 3.1 (personality) — two separate models, no memory context injection.

Goal

Replace both models with Gemma 4 (single call) that:

  1. Sees through Vector's camera (multimodal vision)
  2. Gets injected with full MemoryBank context — every prior observation, known objects, room history
  3. Returns unified nav decision + personality reaction + memory note in one JSON response
  4. Plugs transparently into the existing BehaviorArbiter + MemoryBank + MissionController

Architectural decisions

  • Swap mechanism: GEMMA4=1 env var in rover_brain.py — zero changes to existing code paths
  • Model: gemma4:e4b via Ollama REST at http://localhost:11434
  • Context injection: MemoryBank.get_recent(5) + known_objects + SpatialMemory summary prepended to every prompt
  • Output contract: JSON with keys: scene, objects, nav_decision, danger_level, reaction, memory_note
  • Patcher: patch_all_models() monkey-patches mission_controller._ask_vision and learning_engine LLaVA calls
  • Fallback: if Gemma 4 unavailable, falls back to original AIBrain (LLaVA)

Delivered so far

  • gemma4_brain.py — Gemma4Brain class + patch_all_models() + standalone CLI
  • rover_brain.py — GEMMA4=1 swap point added
  • gemmi_processor.py — Gemma 4 replaces LLaVA as Ollama fallback
  • gemma4_companion.py — lightweight companion loop using existing SDK config

What remains

See child issues for implementation slices.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions