Ternary Bonsai 2 27B VRAM requirements

On essentially any of them, at full 32K context. The PTQ1_0 build totals 8.2 GiB (5.54 GiB weights + ≈0.5 GiB runtime overhead + the 2.1 GiB classic KV bound) — an RTX 3060 12GB runs it with 2.3 GiB to spare, which this site calls tight (under 2.5 GiB of headroom), and a 24GB card runs it with 14.3 GiB spare. At 8K context the load drops to 6.7 GiB and the 12GB card is comfortable; at 4K it lands at ≈6.4 GiB — 8GB-class territory. The catch is not the GPU, it is the runtime: the GGUF files load only in PrismML's llama.cpp fork (see the fork question below).

It is a true ternary quantization, not a rounding quant: every weight takes a value in {−1, 0, +1} with one shared FP16 scale factor per group of 128 weights — 1.585 bits of information per weight plus the amortized scale lands at a measured 1.72 bits per weight across the whole model, with no high-precision escape hatch (embeddings, attention and MLP projections, and the LM head are all ternary; only the 26.2M-parameter recurrent-state path and normalization weights stay higher, already counted in the 1.72). The weights are also stored in a Hadamard-rotated basis that the runtime must apply. PTQ1_0 packs the trits densely at 1.75 bits per weight — that is the 5.95 GB / 5.54 GiB file; PQ2_0 stores each trit in a 2-bit slot (2.13 bits per weight, 7.21 GB / 6.71 GiB), trading footprint for cheaper unpacking and faster prompt processing.

None of them, as of 2026-10-12: per the publisher's KNOWN_ISSUES, stock llama.cpp, Ollama and LM Studio (GGUF) cannot load the PTQ1_0 or PQ2_0 files at all — they are custom ternary GGUF types — and the F16 file loads in stock llama.cpp but produces garbled output, because it depends on rotation metadata only PrismML's build applies. The working paths are: PrismML's llama.cpp fork (CUDA and Metal) with the GGUF packs, or the MLX build with its bundled loader (mlx-vlm 0.7.1+ for image input). One more CPU caveat: PQ2_0 segfaults while loading on AVX-512 CPUs (AMD Zen 4/5, some server chips) even with all layers offloaded to a GPU — pick PTQ1_0 for CPU-centric boxes.

8.2 GiB all-in for PTQ1_0: 5.54 GiB weights + 0.5 GiB runtime overhead + 2.1 GiB KV bound. The KV figure is the classic stacked bound for the GDN hybrid architecture (16 full-attention layers grow linearly with context; the 48 Gated-DeltaNet linear layers hold a constant 0.148 GiB state) — config-derived, unmeasured, actual ≤ value. At 4K context the KV term drops to ≈0.4 GiB and the total lands at ≈6.4 GiB, which is why phones enter the picture.

That is its home turf, with the fork caveats above. Publisher-measured decode on Apple silicon runs 28.1 tok/s on an M5 Pro laptop at a 7.2 GB footprint (PQ2_0, Metal, quiet machine — the publisher notes a ±4% swing with background load), 47.0 tok/s on an M5 Max and 18.0 on an M4 Pro on an earlier pre-rotation build, reported pending re-measurement. For phones: the PTQ1_0 build at 4K context totals ≈6.4 GiB, so an 8GB-class phone runs it tight and a 12GB-class phone runs it with room — via the MLX path with its bundled loader. On x86 CPUs use PTQ1_0: PQ2_0 crashes on AVX-512 hardware during load (marked fixed in source in the pinned KNOWN_ISSUES, merged 2026-09-23).

Publisher-reported, but unusually well-controlled: EvalScope + vLLM on H100, 14 benchmarks in thinking mode, against the Qwen3.8-27B FP16 reference and conventional quants of the same base. The ternary build scores an 84.78 thinking average — 98.2% of FP16 (86.32), within 0.4 points of UD-Q4_K_XL (85.18), and twelve points above IQ2_XXS (72.59) at about 82% of its size. The gap vs conventional 2-bit is not uniform: IQ2_XXS collapses exactly where reasoning chains matter (AIME26 57.5, LiveCodeBench 56.4) while Bonsai 2 holds those benchmarks (95.83, 90.07); math is 96.57 vs the base's 97.06 and coding is level (89.42 vs 89.07). Treat these as the publisher's numbers — we link them, we did not re-run them.

Same publisher, different trade. The original Bonsai-27B is a binary (two-value) 1.125-bits-per-weight QAT of Qwen3.6-27B — a 3.5 GiB weights file whose quality lands between conventional Q2 and Q4_K_XL. Bonsai 2 moves to ternary values on the newer Qwen3.8-27B base: 1.72 bits per weight, a 5.54 GiB file, and a quality story that flips from 'smallest, with a quality cost' to 'within half a point of UD-Q4_K_XL at a third of its footprint'. The original stays the raw-size champion — nothing on this site beats a 3.5 GiB 27B-class model — while Bonsai 2 is the pick when the quality-per-GiB ratio matters more than the last two GiB.

Weights are measured file sizes at pinned revisions (sources below); totals add runtime overhead plus the KV cache at 32K context. Here KV@32K = 2.148 GiB: 65,536 B/token across the 16 full-attention layers plus the 0.148 GiB GDN constant state (config-derived). A build fits a card at 32K when the total clears usable VRAM (nominal minus reserve), and tight means under 2.5 GiB of headroom — the full formula lives in the method box under the quantization table.