My AI assistant on Meta Quest 3: how it works and a prompt to build yours

Meta Quest 3 (browser, WebXR)
        │  secure WebSocket (wss), local network
        ▼
Server on my computer (FastAPI)
  1. whisper.cpp ── your voice to text       (local)
  2. router ─┬── chat and simple requests ── Ollama, small model (local)
             └── reasoning, planning, code ── cloud model
  3. Kokoro ── the answer to speech          (local)
        │
        ▼
Audio back to the headset + floating panels

(Note: this isn't literally the prompt I used to build Edison; I built it over several sessions, iterating. It's a summary of the architecture and what I learned, meant to help your coding agent build you a first version.)

I want to build a voice AI assistant called {{ASSISTANT_NAME}} that lives in my Meta Quest 3 in mixed reality. I talk to it, it answers out loud, and floating panels appear in my room. Work in phases: don't move on to the next one until the current one meets its acceptance criteria, and when you finish each phase, tell me exactly what I need to test with the headset on (you can't test WebXR on the hardware).

ARCHITECTURE
- Frontend: a web page with Three.js + WebXR ("immersive-ar" session with passthrough) opened from the Quest browser. No native app, no Unity, no XR frameworks on top.
- Backend: FastAPI (Python 3.11+) running on my computer ({{COMPUTER_OS}}), on the same wifi network as the headset.
- STT: whisper.cpp compiled locally, with my hardware's acceleration. Input: 16 kHz mono WAV.
- TTS: Kokoro, local, voice {{TTS_VOICE}} in {{LANGUAGE}}. Output: 24 kHz WAV.
- LLM with a router: {{LOCAL_MODEL}} via Ollama (OpenAI-compatible API) for chat and simple things; {{CLOUD_LLM}} for anything that needs reasoning, planning or writing code.
- Transport: WebSocket (wss) between headset and backend; POST /voice as an alternative.
- Configuration with pydantic-settings and .env. No keys in the repo. Code in English.

NON-NEGOTIABLE REQUIREMENTS
- HTTPS is mandatory: WebXR and the microphone don't work on the Quest without HTTPS. Use mkcert and generate a certificate that includes localhost, 127.0.0.1 and my computer's IP on the local network. Put the IP in a config variable and create a script to regenerate the certificate if the IP changes.
- Nothing exposed to the internet: no public tunnels. CORS limited to the local network.
- A start script that brings up backend and frontend, and a setup script that compiles whisper.cpp, downloads models and creates the virtual environment.
- A desktop mode: the same page must work in the computer's browser with mouse and keyboard, to debug without the headset. Add debug keyboard shortcuts.
- Measure and log the latency of each stage (STT, LLM, TTS) on every turn. Expose it in GET /health along with the status of each component.
- Split the frontend into modules from the start (scene, audio, panels, network); don't put everything in a single index.html.

PHASE 1: voice pipeline on the computer
- Separate stt, llm and tts services. POST /voice: audio in, audio out. GET /health.
- A terminal test script with two modes: text (I type and it answers with audio) and voice (I record with the computer's microphone).
- Short system prompt: answers of 2 or 3 sentences max, plain text with no markdown or emojis (the output is heard, not read).
Criteria: I talk through the terminal and hear the answer; /health shows all three components green; the log shows latency per stage.

PHASE 2 (MVP): talking to it from the Quest
- HTTPS page with an "Enter MR" button that starts the immersive-ar session.
- Microphone capture in the browser (push to talk), sent over WebSocket as binary frames, a text message to mark the end, audio response and status messages. Protocol with simple prefixes (for example STT:, LLM:, DONE:, ERROR:). Automatic reconnection with ping/pong.
- A floating assistant panel with its status (listening, transcribing, thinking, speaking) and the last answer. Panels are planes with a canvas as texture.
- A simple avatar (for example, an icosahedron with a shader) that changes with the status and reacts to the amplitude of the answer's audio using an AnalyserNode.
Criteria: with the headset on I enter MR, see the panel in my room, talk to it, hear the answer, and the panel shows what it understood and what it answered. If the WebSocket drops, it reconnects on its own.

PHASE 3: local/cloud router
- Routing function with configurable modes: off, local-first, local-only and cloud-only. Default: local-first.
- "Complex" heuristic: long query, reasoning words (architecture, refactor, debug, plan) or questions about my personal stuff (projects, tasks, status). Classify by intent, not only by exact name: whisper mangles proper names.
- The local model gets its own system prompt, short and very directive, with mandatory literal phrases for what it doesn't know. 1B models make things up with long prompts.
- A fallback chain that never returns an error to the user: local -> cloud -> second cloud option. Log which backend actually answered and why. Endpoint GET /llm/status.
- If system memory goes over a threshold, skip the local model for that turn. Use a short keep_alive in Ollama to free RAM.
Criteria: simple chat answers locally in about a second with the model already loaded; a complex query goes to the cloud; if I stop Ollama, it keeps answering; /llm/status shows the last decision.

PHASE 4: extra panels and voice tools
- A tools layer that intercepts the transcribed text BEFORE the LLM using patterns: "show me the weather", "my tasks", "close everything". If it matches, it opens the panel and answers with a short confirmation, without going through the LLM. CONTENT: event over WebSocket.
- A generic content panel system: image, markdown, weather (can be mocked at first), task list and conversation log.
- Interaction: drag, resize and close panels; scroll with the controller thumbstick and with pinch; save the layout in localStorage.
Criteria: I say "show me the weather" and the panel appears in under 2 seconds; I move a panel, reload, and it stays where I left it; I can scroll in MR without a mouse.

PHASE 5 (optional): wake word and delegation
- Wake word with voice detection in the browser.
- Delegate long tasks to {{CLOUD_LLM}} or to a coding agent in the background, and have the assistant tell me out loud when it's done.

KNOWN ISSUES ON THE QUEST: handle them from the start
- When entering an immersive session, the browser can suspend the AudioContext and the microphone stops capturing. After starting the session, call resume(), keep a silent buffer looping so it doesn't get suspended again and, if the microphone stream died, call getUserMedia again.
- When resizing a canvas used as a texture, recreate the CanvasTexture.
- Don't use createConicGradient (it isn't available in every Quest browser). Iframes with CSS3DRenderer don't show in immersive mode: plan a fallback.
- To debug on the headset, use the remote devtools (chrome://inspect with the headset connected) and leave prefixed logs, for example [audio].