Skip to content

Architecture ​

System diagram ​

┌─────────────────┐           ┌─────────────────────────────────────┐
│  Browser        │  WebRTC   │  protovoice container (GPU 0)        │
│  (mic + spk)    │◄─────────►│                                      │
│                 │           │  FastAPI :7866                       │
│                 │◄──HTTP───►│    /api/offer (POST/PATCH)            │
└─────────────────┘           │    /api/verbosity                    │
                              │    /healthz                          │
                              │                                      │
                              │  SmallWebRTCTransport                │
                              │      │                               │
                              │      ▼                               │
                              │  Silero VAD + Whisper large-v3-turbo │
                              │      │                               │
                              │      ▼                               │
                              │  OpenAILLMService ───► vLLM :8100    │
                              │      │      (or LLM_URL gateway)      │
                              │      ▼                               │
                              │  FishAudioTTS ──► HTTP :8092 ──┐      │
                              │  OR LocalKokoroTTS (in-proc)   │      │
                              │      │                         │      │
                              └──────┼─────────────────────────┼──────┘
                                     │                         │
                                     │                         ▼
                                     │      ┌─────────────────────┐
                                     │      │  fish-speech (GPU 1) │
                                     │      │  tools.api_server    │
                                     │      │  --half --compile    │
                                     │      └─────────────────────┘
                                     ▼
                               (audio back to browser)

Why two containers? ​

Dependency isolation. Fish Audio ships its own .venv with pinned torch, VQ-VAE, llama decoder, and codec models. Jamming it into the same Python environment as vLLM creates dep conflicts that take a week to unwind. Separate containers keep the matrix small.

GPU separation. Fish S2-Pro at --compile wants ~22 GB + compile memory. Whisper (~6 GB) + vLLM routing (~15 GB) + Kokoro fallback (~2 GB) want ~23 GB. Two GPUs, one workload each.

Restartability. Fish's ~2-minute cold compile happens in its container. Restarting the voice agent doesn't retrigger it. Restarting Fish doesn't tear down the voice agent either.

Why FastAPI under Pipecat? ​

Pipecat's SmallWebRTCTransport doesn't ship a server; it's a library you mount on whatever HTTP framework you want. We use FastAPI so we can host, alongside the voice pipeline:

  • WebRTC signalling — POST /api/offer + PATCH /api/offer (trickle ICE).
  • Session control — POST /api/verbosity, POST /api/skills, POST /api/voice/clone, GET /healthz, GET /metrics.
  • Inbound A2A JSON-RPC — POST /a2a handles both message/send (sync) and message/stream (SSE) per spec; the text agent runs a bounded ReAct loop so external fleet agents can use our tool registry. See A2A Integration.
  • A2A push callbacks — POST /a2a/push (spec-conformant) and POST /a2a/callback (legacy permissive shape).
  • Agent card — GET /.well-known/agent.json for A2A discovery.
  • The static HTML client served from static/.

The pipeline itself runs inside a PipelineTask spawned per connected WebRTC peer; text-only A2A traffic bypasses the pipeline entirely and calls the text agent directly.

Network topology ​

Signalling (HTTPS for non-localhost clients) can go through any reverse proxy. Media (WebRTC UDP) must go directly browser ↔ server. Practical paths:

  • Same LAN — direct.
  • Tailnet — direct via 100.x addresses. Works across the internet because Tailscale does the NAT traversal for us.
  • Internet with TURN — not yet configured; planned for public deployment.

Signalling over HTTPS plus media over UDP is a hard split; Tailscale Funnel forwards HTTPS but does NOT relay arbitrary UDP back to the server, so Funnel is fine for signalling only. Tailscale Serve (tailnet-only HTTPS) works end-to-end because both peers sit on the tailnet.

Connection lifecycle ​

  1. Browser hits GET / → static HTML loads.
  2. User clicks Start → getUserMedia → new RTCPeerConnection with audio + video transceivers.
  3. Browser POSTs SDP offer → server creates a SmallWebRTCConnection, links it to a fresh PipelineTask with its own STT/LLM/TTS instances.
  4. Browser PATCHes ICE candidates as they trickle in.
  5. DTLS + SCTP + data channel open; RTP flows both ways.
  6. User speaks → VAD fires UserStartedSpeakingFrame → STT accumulates audio until stop → TranscriptionFrame → LLM.
  7. LLM streams LLMTextFrames → TTS aggregates into sentences → TTSAudioRawFrame → transport → browser.
  8. Browser disconnects → on_client_disconnected → task.cancel() → resources freed.

Multi-user state ​

Each browser connection gets its own PipelineTask with fresh service instances. Shared module-level state is limited to:

  • VERBOSITY (filler settings) — shared, will be per-session later
  • vLLM subprocess — shared, stateless per request
  • Fish sidecar connection — shared, stateless per request
  • Whisper / Kokoro HF models — shared, loaded once, stateless

No session state is persisted across browser reconnects yet. Memory and skill personas land in later milestones.

Part of the protoLabs autonomous development studio.