White paper ·
MAX3 · Technology Evaluation
Hardware & Software Tech-Stack Evaluation
Specifications, alternatives (OSS / free / paid), and recommendations
2026-06-14 RTX 3090 Ti · 64 GB · local-first
00
Headline
The stack is well-chosen. Keep ~90 %. Three changes are worth piloting (TTS naturalness, embeddings quality, object detector), one upgrade is worth planning (vector store), and a handful of tempting-but-wrong options are explicitly rejected because they break a hard constraint. Nothing here argues for a rewrite, and no new hardware is needed for the design as specified.
01
Hardware specifications (the box this runs on)
| Resource | This machine | What it means for choices |
|---|---|---|
| GPU | RTX 3090 Ti, 24 GB GDDR6X (Ampere, sm_86) | Fits a 14B LLM at Q4 comfortably, a 32B at Q4 tightly, a 70B only with slow CPU/RAM offload. No FP8/FP4 tensor cores (that's Ada/Blackwell). |
| CPU | 8 cores / 16 threads | Enough for Piper (CPU TTS), the bus, DuckDB, and background workers — the voice loop must not fight ingestion (Rule 6 enforces this). |
| RAM | 64 GB | Generous. DuckDB buffer pool tuned to 8 GB. Headroom for a second model resident in RAM. |
| Network | Local-first, behind OPNsense; LAN opt-in | Every cloud option is a fallback, never a dependency. Outbound is gated — this single fact rejects most paid managed services. |
THE FOUR BINDING CONSTRAINTS (PRE-FILTER EVERY OPTION)
■ Local-first. Routine use works with the network down; cloud is an upgrade path, not a default.
■ Open-source dependencies. MIT/BSD/Apache preferred; GPL/AGPL avoided; proprietary/telephone-home not acceptable — the one exception is cloud LLM APIs as a paid fallback.
■ Single core process, no microservices. The voice-turn latency budget (1500–2000 ms) will not survive an IPC hop per faculty.
■ No heavyweight agent frameworks. No LangChain / LangGraph / CrewAI / AutoGen; a custom thin orchestration layer we own.
02
Software scoreboard — the whole stack at a glance
Legend: Keep · Pilot a change · Plan an upgrade · Reject (tempting-but-breaks-a-rule).
| Layer | Current | Verdict | Reason |
|---|---|---|---|
| Backend lang | Python 3.12 | Keep | The ML ecosystem is Python; nothing else is close. |
| Web framework | FastAPI + uvicorn | Keep | Async, Pydantic-native, WebSocket-native. |
| LLM server | Ollama | Keep | Single-process, local; vLLM/TGI optimise concurrency we don't have. |
| LLM (capable) | Qwen 14B / 70B Q4 | Tune | 14B Q4 fully on-GPU is the sweet spot; 70B too slow locally. |
| Cloud LLM | Claude Sonnet 4.5 | Keep | Correct as the paid fallback; stays a fallback. |
| Graph DB | DuckDB | Keep | Just migrated; embedded, analytical, MIT. |
| Vector index | sqlite-vec | Plan | Fold into duckdb-vss when WAL recovery is proven. |
| Tabular | SQLite | Keep | Event log / audit / ingestion bus. Boring and correct. |
| STT | Distil-Whisper-large-v3 | Keep | Parakeet disqualified on cuDNN deploy cost (decided). |
| VAD | Silero VAD | Keep | MIT, tiny, accurate. |
| Speaker ID | ECAPA-TDNN | Keep | TitaNet blocked on same NeMo/cuDNN issue (decided). |
| TTS | Piper | Pilot | Kokoro-82M much more natural, still Apache + local. |
| Wake word | openWakeWord | Keep | Apache, in-app trainable. |
| Face | InsightFace (ArcFace) | Keep | MIT, SOTA embeddings, <50 ms here. |
| Object detect | DETR | Pilot | RT-DETR is real-time + Apache (avoids AGPL YOLO). |
| Embeddings | nomic-embed-text | Pilot | bge-m3 / mxbai stronger at similar cost. |
| Frontend | React 19 + Vite + TS | Keep | Modern, fast, boring-in-a-good-way. |
| Orchestration | Custom thin layer | Keep | Hard rule; frameworks add latency + opacity. |
03
LLM serving — the most-scrutinised choice
| Option | Bucket | Pros | Cons for this design |
|---|---|---|---|
| Ollama (current) | OSS (MIT) | One process, trivial pulls, great single-user latency | Hides some llama.cpp knobs; not for high concurrency |
| llama.cpp direct | OSS (MIT) | Every knob (KV-quant, speculative decode) | You rebuild the model-mgmt + API Ollama gives free |
| vLLM | OSS (Apache) | Best throughput under concurrency | Server use case; heavier; more VRAM overhead. We have ONE user. |
| HF TGI | OSS (Apache) | Production serving, tensor parallel | Same multi-user-server mismatch as vLLM |
| LM Studio | free, not OSS | Nice GUI | Closed-source desktop app — fails OSS rule as a dependency |
| NVIDIA NIM/Triton | free + paid | Peak NVIDIA perf | Container/registry/telephone-home — fails local-first |
| Cloud (Claude/GPT) | paid | Frontier quality | Already the fallback; never the default |
Verdict: keep Ollama. For a single-user, local-first resident, throughput engines optimise a dimension we don't have (concurrency) at the cost of dimensions we do (simplicity, single-process, VRAM headroom for Whisper + ArcFace + TTS to coexist). Drop to llama.cpp-direct only if you want speculative decoding.
Which models, given 24 GB
| Tier | Recommendation on a 3090 Ti | Why |
|---|---|---|
| Small (fast) | Qwen2.5-3B / Llama-3.2-3B; trial Qwen3-4B | Sub-second; intent/extraction/summary |
| Capable | Qwen2.5-14B Q4 daily; trial Qwen3-14B/-32B | ~9–18 GB → fully on-GPU → 30–60 tok/s |
| Avoid local | Llama-3.3-70B | Q4 ≈ 40 GB → offload → single-digit tok/s; use cloud fallback instead |
04
The three pilots worth doing
TTS — pilot Kokoro-82M (Apache)
Piper is fast, CPU-friendly, MIT, and tunable on six axes — all still valuable — but its ceiling is 'clear and robotic'. Kokoro-82M (Apache) punches far above its size, sounds dramatically more natural, and is small enough to stream on the 3090 Ti. Keep Piper as the default + low-latency path and the six-axis DSP persona chain; add Kokoro behind a setting and A/B the persona. Do NOT clone a voice or adopt cloud TTS — both are out of bounds.
Vision — pilot RT-DETR (Apache)
DETR was chosen to avoid Ultralytics YOLO's AGPL — correct and non-negotiable — but vanilla DETR is slow, which matters because vision shares the GPU with the voice critical path. RT-DETR is the same HF family and licence posture with real-time speed. Keep InsightFace for faces.
Embeddings — pilot bge-m3 / mxbai-embed-large
nomic-embed-text is solid and local, but retrieval quality is the lever under GraphRAG + curiosity + memory. bge-m3 (MIT, multilingual) and mxbai-embed-large (Apache, top English) are both Ollama-pullable; A/B against MAX's own recall set and switch only on a measurable win (cost: a background re-embed pass, scheduled off-peak).
05
Plan one consolidation — and what to reject
PLAN
■ Vector store — converge sqlite-vec → duckdb-vss when its WAL recovery is proven (two engines collapse into one). Hold Qdrant as the documented escape hatch only if embedded search becomes the measured bottleneck.
EXPLICITLY REJECTED (AND WHY)
■ vLLM / TGI / NIM — optimise multi-user concurrency we don't have, at the cost of the single-process VRAM headroom Whisper + ArcFace + TTS need to coexist.
■ Ultralytics YOLO — AGPL-3.0; forbidden by the OSS-deps rule.
■ XTTS / cloud TTS / cloud STT / cloud vector — send audio, frames, or embeddings off-box; break local-first.
■ LM Studio / NVIDIA NIM — closed-source / telephone-home posture.
■ Pinecone / Neo4j — cloud SaaS / licence + server shape that don't fit a single-box local-first resident.
06
Hardware — what would actually move the needle
| Rank | Lever | Recommendation |
|---|---|---|
| 1 | GPU VRAM (the real ceiling) | 24 GB is enough for the spec. The only gap is local interactive 70B — the cloud fallback already covers that. Buy GPU (48 GB) only if local frontier reasoning becomes a daily need. |
| 2 | RAM | Ample at 64 GB. The HardwareRecommender watches for sustained pressure. No action. |
| 3 | Disk | Prefer fast NVMe for the graph + model files — the cheapest cold-boot/model-load latency win, not more GPU. |
| 4 | Network | OPNsense + local-first + LAN opt-in is exactly right. Keep exposure opt-in. |
Bottom line on hardware: hold. 24 GB is enough for the design as specified; the cheapest real latency win is a fast NVMe; spend on the cloud fallback, not a card, for the rare hard turn.
07
Recommendation summary
| Action | Item | Effort | Payoff |
|---|---|---|---|
| Pilot | Kokoro-82M TTS (Apache) behind a setting, A/B vs Piper | Low | Much more natural persona; keeps Piper + DSP chain |
| Pilot | RT-DETR (Apache) replacing DETR | Low | Real-time vision that respects the shared-GPU budget |
| Pilot | bge-m3 / mxbai vs nomic embeddings | Low (one re-embed) | Better retrieval under GraphRAG + memory |
| Plan | duckdb-vss to retire the sqlite-vec split | Medium | One engine instead of two |
| Optional | llama.cpp speculative decoding | Medium | Lower latency on the capable tier |
| Keep | Ollama, DuckDB, FastAPI/React, Whisper, ECAPA, InsightFace, custom orchestration, single-process | — | All correct for local-first single-user |
| Reject | vLLM/TGI, LM Studio/NIM, YOLO (AGPL), XTTS/cloud TTS, cloud STT/vector, Neo4j | — | Each breaks a hard rule or solves a problem we don't have |
| Hardware | Hold. 24 GB is enough; NVMe is the cheapest latency win | — | Spend on the cloud fallback, not a card, for the rare hard turn |
The through-line: this stack was chosen by the same constraints that should govern any change — local-first, open-source, single-process, auditable. The alternatives that beat it on a benchmark almost always lose on one of those; the ones that don't (Kokoro, RT-DETR, bge-m3) are model swaps inside the existing architecture, not replacements of it. The design is ambitious; the stack is already the right shape to carry it.
Keep reading
More field notes
This piece is part of the MAX Research Collective library. Browse the rest, or connect on LinkedIn.