Nemotron 3.5 Bolt 74B-A9B VRAM requirements
fits a 24GB card — 1.1 GiB spare (tight) · the official safetensors build: 22.9 GiB before KV, over the 22.5 GiB usable — no fit
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB is the native 2-bit member of the Nemotron 3.5 family — a 74B-parameter (9B active) Mamba-2×Transformer hybrid LatentMoE compressed from the Super VL 120B flagship and quantization-aware-distilled into mixed precision (IQ2_XXS experts, NVFP4 Mamba, Q8_0 projections). NVIDIA's claim is "24GB": this site prices usable VRAM instead — 22.5 GiB on a 24GB card. The official GGUF build measures 20.51 GiB and totals 21.4 GiB at 32K context with the KV bound counted: it fits, with 1.1 GiB to spare (tight by this site's 2.5 GiB threshold). The official safetensors build — the ModelOpt mixed-precision original — weighs 22.43 GiB, which is 0.07 GiB under usable before runtime overhead and 22.9 GiB with it: over budget before any KV cache, a paper door rather than a real one. Smallest tracked build: unsloth's UD-IQ1_S at 17.24 GiB weights, 18.1 GiB at 32K. Multimodal through a C-RADIO vision tower (no audio), 256K context, released 2026-10-15 under OpenMDW-1.1.
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB is the 24GB-native member of the Nemotron 3.5 family, released 2026-10-15 under the OpenMDW-1.1 license. Architecture: nemotron_h_omni — 88 heterogeneous blocks picked per token by a Star Elastic router (40 Mamba-2 state-space blocks + 40 LatentMoE blocks + 8 full-attention blocks at sparse positions), 512 experts per MoE block with top-9 routing plus 1 shared expert, 74B parameters total / 9B active, 256K maximum context. It is multimodal through a C-RADIO v4-H vision encoder; there is no audio tower. Lineage: the Nemotron 3.5 Super VL 120B-A12B flagship was compressed via Star Elastic plus knowledge distillation into a BF16 checkpoint, then quantization-aware distillation (QAD, NVIDIA ModelOpt) produced native mixed precision — IQ2_XXS at 2.0625 bits per element (group 256) on all routed and shared experts, NVFP4 on the Mamba linear layers, Q8_0 on embeddings, attention and LatentMoE projections. Publisher-reported benchmark and speed tables are linked in the pinned sources.
Data verified · sources pinned in Where every number comes from
How much VRAM at each quantization? (weights + overhead + 4K KV, batch 1)
| Quantization | VRAM needed @4K | KV @32K provisional i · beyond the @4K column | Smallest tracked fit | Note |
|---|---|---|---|---|
| UD_IQ1_S | 17.9 GiB | 0.41 GiB | RTX 3090 24GB | measured · at 32K: 18.1 GiB |
| IQ2_XXS | 21.2 GiB | 0.41 GiB | RTX 3090 24GB | measured · at 32K: 21.4 GiB |
| IQ2_MIXED | 23.1 GiB | 0.41 GiB | RTX 5090 32GB | measured · at 32K: 23.3 GiB |
| UD_Q2_K_XL | 27.5 GiB | 0.41 GiB | RTX 5090 32GB | measured · at 32K: 27.7 GiB |
| BF16 | 140.2 GiB | 0.41 GiB | — | measured · at 32K: 140.4 GiB |
GB vs GiB: card names are the manufacturer's nominal GB; totals here are GiB (230 bytes) — a "24GB" card holds 24 GiB, and the official GGUF quoted as a 22.0 GB download is the same 20.51 GiB file (decimal vs binary). NVIDIA's "24GB" claim is a nominal-card claim; this page prices the 22.5 GiB you can actually use. Caliber →
The 48GB-class doors, spelled out: a dual-GPU combo (2× 24GB; not one card doubled), a workstation card (RTX 6000 Ada, 48GB), or a Mac Studio (unified memory; runs, slower tok/s). For Nemotron 3.5 Bolt they open only the quality steps above the 24GB ceiling — UD-IQ2_M and up — while the BF16 reference build (140.4 GiB at 32K) misses them all. The 24GB door is the story of this page.
How this page computes the numbers (reproduce any cell)
- usable VRAM = nominal minus reserve, per card: 12GB → 10.5 GiB, 16GB → 14.5 GiB, 24GB → 22.5 GiB, 32GB → 30.0 GiB.
- total@32K = weights + overhead + KV@32K; KV@32K = 0.41 GiB (config-derived — 13,435 B/token effective at 32K; hybrid models carry a constant state on top, composition in sources).
- total@4K (main column) = total@32K − (KV@32K − 0.16) × 7/8 — the constant state (0.16 GiB on this hybrid) is not scaled down at shorter context; only the linear part divides by eight.
- The "≤~XK ctx" note = 8K + 32K × (usable − total@8K) ÷ KV@32K — the context length that exactly fills the card (KV grows linearly with context on this model).
- A build fits a card at 32K when total@32K ≤ usable; tight = under 2.5 GiB of headroom, one site-wide threshold.
Is your GPU enough? (find your card: one row tells you)
| Your GPU | VRAM | Biggest quant that fits (ceiling) | How it feels |
|---|---|---|---|
| RTX 3060 12GB | 12 GB | — | — |
| RTX 4060 Ti 16GB | 16 GB | — | — |
| RTX 3090 24GB | 24 GB | UD_IQ2_XXS (2-bit) · 21.4 GiB | 1.1 GiB spare at 32K (incl KV bound) |
| RTX 4090 24GB | 24 GB | UD_IQ2_XXS (2-bit) · 21.4 GiB | 1.1 GiB spare at 32K (incl KV bound) |
| RTX 5090 32GB | 32 GB | UD_Q2_K_XL (2-bit) · 27.7 GiB | 2.3 GiB spare at 32K (incl KV bound) |
Quants stack by size: each GPU has exactly one ceiling — a card that runs a bigger build runs every smaller one. Main-column totals price the 4K load (weights + overhead + the 4K KV term — the community runtime default); each row's Note prices the full 32K load this page promises (21.4 GiB at IQ2_XXS); and the CPU path trades tok/s for the same headroom on plain RAM.
What GPU should you buy for it? (the 24GB sweet spot logic)
| The logic | Nemotron 3.5 Bolt re-centers the site's 24GB advice on a first-party build. The official GGUF totals 21.4 GiB at 32K context against 22.5 GiB usable on a 24GB card — 1.1 GiB spare, "tight" by this site's 2.5 GiB threshold, so it runs with room to breathe at 16K context or below and holds the full 32K exactly. There is no 16GB play: the smallest tracked build (unsloth UD-IQ1_S) floors at 17.7 GiB. And there is no engine roulette: unlike wrapper-engine routes, this is a plain GGUF file that llama.cpp-class runtimes load directly. |
| Why it fits | The weights are the whole story: 20.51 GiB of IQ2_XXS for a 74B-parameter model, where the BF16 original is 139.5 GiB. The hybrid KV bill stays tiny — 0.41 GiB at 32K — because only 8 of 88 blocks hold attention KV; the 40 Mamba-2 blocks hold a constant 0.16 GiB state. Weights + overhead + that bound = 21.4 GiB all-in. |
⚠ Tight is a site-wide threshold: under 2.5 GiB of headroom — and on this page it decides between the two official builds. IQ2_XXS @32K leaves 1.1 GiB on a 24GB card (tight — fits); the official safetensors build is over usable VRAM before KV is even counted (22.9 GiB weights + overhead vs 22.5 GiB usable). UD-IQ1_S, if you accept its quality cut, clears everything with 4.4 GiB spare. All totals carry the hybrid KV bound, so these verdicts are stable, not flip-check rows.
Tracked builds (actual file sizes, not theory)
| Build | Publisher | File size | 24GB @32K | Source |
|---|---|---|---|---|
| UD_IQ1_S | unsloth | 17.2 GiB | OK · 18.1 | HF @34841e36 verified |
| IQ2_XXS | unsloth | 20.5 GiB | TIGHT · 21.4 | HF @34841e36 verified |
| IQ2_MIXED | nvidia | 22.4 GiB | NO · 23.3 | HF @c6cc2e8f verified |
| UD_Q2_K_XL | unsloth | 26.8 GiB | NO · 27.7 | HF @34841e36 verified |
| BF16 | nvidia | 139.5 GiB | NO · 140.4 | HF @56702103 verified |
One publisher, three official builds: the IQ2_XXS GGUF (20.51 GiB, single file, the build this page recommends), the ModelOpt mixed-precision safetensors (22.43 GiB across 90 shards — IQ2_XXS experts, NVFP4 Mamba linear, Q8_0 embeddings/attention/projections), and the BF16 reference (139.5 GiB, 63 shards) the 24GB build was distilled from. unsloth ships the community ladder (UD-IQ1_S 17.24 GiB up to UD-Q6_K_XL); bartowski and ggml-org have not published repacks — eight name variants probed, all 404. The vision tower is in none of these files: it ships as a separate mmproj pack, loaded only when an image arrives. The LiLiCorr speculative-decoding drafter (1.13B parameters, 0.9 GiB NVFP4) is the trick behind the 84 TPS figure — links in the pinned sources. The table's IQ2_XXS row links the unsloth repack — a byte-sibling of the official NVIDIA GGUF (same bytes, different wrapper; the official single-file build is pinned in sources).
Self-host or API?
API prices are sourced from OpenRouter (linked, not re-verified). We have not pinned a price for Nemotron 3.5 Bolt, so the breakeven cell stays empty rather than guessing. When a price is pinned, this block compares API cost per million tokens against self-hosting on your own card.
Where every number comes from (every figure carries its tag)
Totals: measured weights + KV as the hybrid stacked upper bound (8 full-attention blocks' linear growth + 40 Mamba-2 blocks' constant state, config-derived, actual ≤ value) + measured overhead. This page exists for one config: the native 2-bit 74B × 32K context on a real 24GB card.
| Tag | Slot | What it is | What the verdict can claim |
|---|---|---|---|
| reported | model | Hugging Face config.json (identity + quantization_config carrier; nemotron_h_omni, 88 blocks = 40 Mamba-2 + 40 MoE + 8 attention, 512 experts top-9+1 shared) | Tight, conditional — stands on the linked source, re-checkable by you. |
| reported | static | HF model page (nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB) | Tight, conditional — stands on the linked source, re-checkable by you. |
| provisional | kv | KV cache @ 32K — no published measurement in our sources; the page shows the hybrid stacked bound instead: 0.25 GiB growth (8 attention blocks totaling 8,192 B/token) + 0.16 GiB Mamba-2 SSM constant state, config-derived, actual ≤ value; NVIDIA's own recipe runs fp8 KV (halved, reported) | Very likely right, order of magnitude only — replaced the moment measured values land. |
| measured | overhead | Deployment overhead — llama.cpp FA-on compute buffer anchor (≈0.5 GiB; measured empirical; same constant as the other champion pages) | Determinate — "it runs" means it runs. |
| measured | static | HF repo tree (nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF) — official IQ2_XXS single-file size (mmproj vision pack excluded as auxiliary); pinned | Determinate — "it runs" means it runs. |
| measured | static | HF repo tree (nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB) — official ModelOpt mixed-precision safetensors, 90 shards summed (IQ2_XXS experts + NVFP4 Mamba linear + Q8_0 projections); pinned | Determinate — "it runs" means it runs. |
| measured | static | HF repo tree (nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16) — full-precision reference the 24GB build was distilled from, 63 shards summed; pinned | Determinate — "it runs" means it runs. |
| reported | static | NVIDIA model card (pinned README) — architecture lineage (Star Elastic compression of Super VL 120B-A12B + KD → BF16 → QAD), quantization recipe (IQ2_XXS 2.0625 bpw g256 experts / NVFP4 Mamba / Q8_0 rest), 13-row quality table, Test Hardware list; runtime list on the card itself: vLLM | Tight, conditional — stands on the linked source, re-checkable by you. |
| reported | static | NVIDIA Research tech blog (2026-10-15) — 80 TPS design target, 84 TPS AIPerf SPEED-Bench on RTX Spark, AIPerf comparison table (Bolt 83.99 vs Qwen3.6-35B 96.63 vs Gemma4-26B 100.56 TPS), LiLiCorr drafter (1.13B, 7 drafted tokens, accept length 4.68/4.25), 128K bs1 architecture budget; the five-runtime list (llama.cpp/Ollama/LM Studio/Unsloth/vLLM) is stated here | Tight, conditional — stands on the linked source, re-checkable by you. |
| reported | static | NVIDIA blog (2026-10-15) — family announcement: Super VL + Bolt 74B-A9B available now, Bolt E4B coming Oct 23; RTX Spark laptop availability Oct 16; the five-runtime list (llama.cpp/Ollama/LM Studio/Unsloth/vLLM) is stated here | Tight, conditional — stands on the linked source, re-checkable by you. |
| reported | static | OpenMDW License Agreement 1.1 — full text (HF license field reads 'other'; BF16 repo LICENSE is a zero-byte file, text pinned from LiLiCorr-GGUF LICENSE + OpenMDW GitHub) | Tight, conditional — stands on the linked source, re-checkable by you. |
| reported | static | LiLiCorr-GGUF repo LICENSE (full OpenMDW-1.1 text, commit-pinned; the main 24GB repo ships no LICENSE file — 28 non-weight files enumerated) | Tight, conditional — stands on the linked source, re-checkable by you. |
FAQ
Can Nemotron 3.5 Bolt 74B run on a 24GB GPU?
Yes — on the official GGUF build. IQ2_XXS totals 21.4 GiB at 32K context (20.51 GiB weights + ≈0.5 GiB runtime overhead + the 0.41 GiB hybrid KV bound), leaving 1.1 GiB of a 24GB card's 22.5 GiB usable — tight by this site's 2.5 GiB threshold, and comfortable at 16K context or below. The official safetensors build does not fit: 22.43 GiB of weights is 0.07 GiB under usable before overhead, 22.9 GiB with it — over budget before any KV is allocated. 16GB cards have no play at all (the smallest tracked build floors at 17.7 GiB); a 32GB card adds the UD-Q2_K_XL quality step.
What is the difference between the two official builds — safetensors vs GGUF?
Same mixed-precision weights, two containers. The safetensors repo (nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB) is the ModelOpt original: 90 shards summing to 22.43 GiB — IQ2_XXS experts at 2.0625 bits per element, NVFP4 Mamba linear layers, Q8_0 embeddings/attention/projections — plus the custom modeling code transformers needs to run it. The GGUF repo ships one 20.51 GiB IQ2_XXS file that llama.cpp-class runtimes load directly, with a separate mmproj pack for vision. On capacity: the safetensors build misses every consumer card (22.9 GiB floor vs 22.5 GiB usable); the GGUF build is the 24GB door. The BF16 reference (139.5 GiB) is the distillation source, not a deployment option.
What does the 2-bit quantization cost in quality?
Per NVIDIA's own table (pinned model card, reported — we link it, we did not re-run it), Bolt leads Terminal-Bench 2.1 (50.98), BrowseComp (66.75), GPQA Diamond (86.43), HLE without tools (25.30), SciCode (41.02), MCP-Atlas (52.20), WebArena with vision (50.86), GDPVal-AA-V2 (28.47), Tau3 Banking (17.01) and AA-LCR (64.06) against Qwen3.6-35B, Gemma4-26B and Nemotron 3.5 Lightning 30B; it trails on PinchBench (87.82 vs 88.07), AA-Omniscience accuracy (16.90 vs 22.17) and non-hallucination (61.27 vs 68.81). The tech blog states the narrower claim: #1 PinchBench, #2 WebArena, #2 BrowseComp among models with a similar memory footprint. On speed, NVIDIA's framing is tokens per GB, not absolute first place — see the speed question below.
How does Bolt relate to Super VL — and what is the E4B?
Bolt is the 24GB-native sibling of the Nemotron 3.5 Super VL 120B-A12B flagship: Super VL was compressed via Star Elastic plus knowledge distillation into the 74B BF16 checkpoint (139.5 GiB), then quantization-aware-distilled to the 24GB build. Super VL itself is cliff material — its BF16 build is 231.8 GiB (NVFP4: 69.9 GiB), past every consumer card this site tracks — and it has no page here yet; this page says so instead of pretending otherwise. The Bolt E4B is the family's other half: a 4GB-class model for voice assistants, interactive characters, and embedded interfaces (GeForce RTX / Jetson / RTX Spark per NVIDIA’s blog), which NVIDIA says releases Oct 23, 2026 — as of 2026-10-20 there is no E4B repo on Hugging Face (three name probes all 404, search returns zero), and this page carries the note until it lands.
How is the Nemotron route different from the Strata route to big models on small cards?
Two engineering answers to the same question. Strata is an inference-engine answer: it streams a model that does not fit — the 125B-class Qwen3.8-Flash-Next — across RAM, VRAM and SSD tiers, asking at least 32GB of combined RAM+VRAM and a single 12GB+ GPU. Nemotron Bolt is a weights answer: native 2-bit training compresses the 74B into 20.51 GiB that sits in VRAM outright, no streaming layer, at 21.4 GiB all-in for 32K context. Strata keeps you at full-precision quality with an engine dependency; Bolt bets that QAD-recovered 2-bit quality (the table above) is enough. Pick Strata for maximum capability per dollar of memory; pick Bolt if you want a plain GGUF file that llama.cpp-class runtimes load directly.
What license is Nemotron 3.5 Bolt released under?
The OpenMDW License Agreement 1.1 — NVIDIA's open-weights model license. The Hugging Face license field reads 'other', with license_name openmdw-1.1 in the card frontmatter. The BF16 repo's LICENSE file is a zero-byte placeholder, so the full text is pinned from the LiLiCorr-GGUF repo LICENSE and the OpenMDW GitHub (both in sources). Commercial use is governed by that text — read it before shipping a product on this model.
How does VRAM scale with the 256K context?
Only 8 of the 88 blocks carry attention KV, so the bill grows slowly: 8,192 B per token is ≈0.06 GiB per 8K of context on top of the 0.16 GiB Mamba-2 constant state — 0.41 GiB at 32K, ≈1.2 GiB at 128K, ≈2.2 GiB at 256K (all bf16, constant state included). NVIDIA's own 24GB budget is defined at batch size 1 with 128K context, fp8 KV cache and quantized runtime states — a reported caliber that halves the KV term. This page's fit verdicts price the conservative 32K bound; on a 24GB card the official GGUF at 128K bf16 KV would total ≈22.2 GiB — arithmetically inside, practically pinned to the metal, which is exactly why NVIDIA's recipe reaches for fp8.
How fast is it, really?
NVIDIA reports 84 TPS on the AIPerf SPEED-Bench benchmark running on an RTX Spark (llama.cpp, Windows 11 ARM x64), against a design target of at least 80 TPS — with the LiLiCorr drafter attached (1.13B parameters, 7 drafted tokens, 4.68 average accepted length at greedy under vLLM concurrency tests). The same table shows two 4-bit models decoding faster in absolute terms — Qwen3.6-35B at 96.63 TPS and Gemma4-26B at 100.56 TPS against Bolt's 83.99 — so the honest pitch is speed per GB of weights, not the tok/s crown. All speed figures are NVIDIA-reported on NVIDIA test hardware; we link them, we did not re-run them.