localvm-research

Research / research/notes/quantization.md

quantization

draft

research/notes/quantization · created Mon Aug 10 2026 20:00:00 GMT-0400 (heure avancée de l’Est) · Simon-Pierre Boucher

Quantization — deep literature note (charter §4.1)#

Scope: post-training quantization (PTQ), quantization-aware training (QAT), 8→1-bit and ternary regimes, mixed/per-layer/per-channel/dynamic/progressive precision, residual/additive/vector/lattice/trellis (codebook) quantization, weight-only vs weight+activation quantization, KV-cache quantization, and extreme low-bit inference — with explicit attention to (a) memory-capacity reduction vs memory-bandwidth reduction as distinct effects, and (b) which formats have efficient Metal/MLX kernels vs CUDA-only implementations. All sources listed in §Sources were accessed 2026-08-11.

1. Landscape overview#

The field has converged on a fairly stable picture as of mid-2026:

  1. 4-bit weight-only is a solved commodity. GPTQ (ICLR 2023), AWQ (MLSys 2024 best paper), and calibration-free HQQ all produce 4-bit models within a few percent (often <0.1–0.3 PPL) of FP16. Every serious Mac runtime (llama.cpp Metal, MLX) executes 4-bit weights natively at close to memory-bandwidth limits.
  2. 2–3-bit is where the frontier is. State of the art moved from scalar quantization (GPTQ) → incoherence processing + scalar (QuIP) → vector/lattice codebooks (QuIP#, AQLM, GPTVQ) → trellis-coded quantization (QTIP, EXL3), each step buying quality at the same bitrate. QuIP# was the first PTQ method where 3-bit scaled better than 4-bit. The catch for us: all SOTA sub-3-bit codebook/trellis kernels are CUDA-first; on Apple GPUs the analogous formats (llama.cpp i-quants) decode measurably slower than simple affine formats.
  3. Weight+activation quantization (W8A8, W4A4) is a compute/throughput play, not primarily a capacity play — relevant for prefill speed, mostly irrelevant to the decode-time bandwidth wall on a Mac at batch 1. Rotation methods (QuaRot, SpinQuant) made W4A4 viable by killing activation outliers.
  4. Sub-2-bit exists but changes character. BiLLM (PTQ, ~1.08 bit) is "viable" but degraded; BitNet-style ternary needs (re)training; ParetoQ shows a sharp representational transition between 2 and 3 bits — below it, weights leave the pretrained basin. For a system that must preserve the original model's behavior (charter §14 hard constraint), 2-bit is the practical floor for a base representation, not an endpoint.
  5. The newest and, for us, most important thread is multi-precision / nested / progressive representations: Any-Precision LLM (bitplanes, ICML 2024 oral), Matryoshka Quantization (MSB-nesting, ICLR 2025 oral), Drop-by-Drop additive codebooks with successive-refinement theory (2026), and Progressive Mixed-Precision Decoding (ICLR 2025). These decouple stored bits from read bits — exactly the total size ≠ bytes read per token decoupling in our core research question. None of them targets Apple Silicon, and none streams residual precision from SSD on demand.

Key systems distinction used throughout: at batch-1 decode, token latency ≈ (bytes of weights+KV read per token) / (achievable memory bandwidth). A format reduces memory if the resident representation is smaller; it reduces bandwidth per token only if the bytes actually traversed per token shrink and the dequant compute doesn't become the new bottleneck. These usually coincide for dense inference (every weight is read every token) but diverge for: codebook lookups (extra random reads), lossless coding (decode compute), bitplane/nested formats (read fewer planes than stored), and any offload scheme.

2. Technique families#

2.1 Round-to-nearest and second-order weight-only PTQ (GPTQ family)#

What it does. GPTQ ("Accurate Post-Training Quantization for Generative Pre-trained Transformers", Frantar et al., ICLR 2023) quantizes weights column-by-column, using a per-layer Hessian proxy (from ~128 calibration samples) to update remaining unquantized weights and cancel accumulated error. Quantized OPT-175B to 3–4 bits in ~4 GPU-hours.

  • Memory reduction: ~4× at 4-bit, ~5.3× at 3-bit vs FP16.
  • Bandwidth reduction: proportional to memory for dense decode (all weights read each token); reported 3.25× (A100) / 4.5× (A6000) generation speedups vs FP16.
  • Quality: near-lossless at 4-bit (e.g., OPT-1.3B: FP16 PPL 14.63; naive RTN 4-bit 48.24; GPTQ 4-bit 15.47). 3-bit usable; 2-bit/ternary "viable" only with tiny groups. AWQ's authors document GPTQ overfitting to the calibration distribution (2.3–4.9 PPL worse under calibration/eval distribution shift, vs 0.5–0.6 for AWQ).
  • Calibration/retraining: calibration data required; no retraining.
  • Apple Silicon status: the algorithm is platform-neutral; GPTQ-produced integer weights convert to GGUF/MLX layouts fine. The fast GPTQ inference kernels (Marlin, ExLlama) are CUDA-only. mlx-lm ships a GPTQ-style learned-quant recipe natively.
  • Main limitation: scalar, uniform grids; hits a wall below 3 bits.
  • Extension opportunity for us: GPTQ's Hessian machinery is reusable for allocating precision (which blocks deserve residuals) rather than just rounding.

AWQ (Lin et al., MLSys 2024 Best Paper, arXiv 2306.00978): observation that ~1% of weight channels are salient as measured by activation magnitude; protecting them via per-channel scaling (no mixed precision needed) preserves quality. No backprop; more robust to calibration shift than GPTQ. TinyChat gives >3× over HF FP16 on desktop/mobile GPUs. mlx-lm includes an AWQ-style learned quant. Same memory/bandwidth profile as GPTQ.

HQQ (Mobius Labs, blog Nov 2023): calibration-free — solves a per-group half-quadratic optimization on weights only. Llama-2-70B quantized in <5 min (>50× faster than GPTQ); their 2-bit 70B beats FP16 Llama-2-13B in PPL at comparable memory. Useful to us as the cheapest way to generate base representations at many bit-widths for experiments. Python/PyTorch; runs anywhere (incl. MPS backend) but fast fused kernels are CUDA.

SqueezeLLM (Kim et al., ICML 2024, arXiv 2306.07629): sensitivity-based non-uniform (k-means) codebooks per output channel + dense-and-sparse decomposition (0.05–0.45% of outlier/sensitive weights kept in FP16 sparse format). 3-bit LLaMA-7B: sensitivity-weighted clustering takes PPL from 18.08 (unweighted k-means) to 7.75. Explicitly frames single-batch LLM inference as memory-bandwidth bound, and shows LUT dequant can still hit ~2.3× GPU speedup. The dense+sparse split is a primitive "base + residual" — the sparse part is a tiny, importance-selected correction stream. CUDA kernels only.

2.2 Weight + activation quantization (SmoothQuant → rotations)#

SmoothQuant (Xiao et al., ICML 2023, arXiv 2211.10438): migrates activation-outlier difficulty into weights via per-channel equivalent scaling → W8A8 training-free, negligible loss on OPT/BLOOM/Llama; up to 1.56× speedup and 2× memory reduction; enabled serving 530B in one node.

QuaRot (Ashkboos et al., NeurIPS 2024): computational-invariance rotations (randomized Hadamard) applied to hidden state, FFN activations, attention, and KV cache remove outliers without changing model output, enabling end-to-end 4-bit (weights, activations, KV). Llama-2-70B W4A4: ≤0.47 WikiText-2 PPL loss, 99% zero-shot retention; lossless W6/W8 with plain RTN and no calibration data.

SpinQuant (Liu et al., ICLR 2025): same invariance idea but with learned rotations (Cayley-optimized R1/R2 absorbed offline, plus online Hadamard R3/R4), closing more of the W4A4KV4 gap than random rotations.

  • Memory/bandwidth: activation quantization does not shrink the checkpoint; its win is compute density (INT8/INT4 matmul) and KV size. On a Mac at batch 1 decode this matters mainly through the KV cache and prefill speed.
  • Apple Silicon status: no shipped Metal W4A4 path. MLX recently added quantize_input and mxfp8/nvfp4 modes (see §2.4), which is the beginning of an activation-quantization story; fast Hadamard transform kernels for Metal would need to be written. Note M5-class Neural Accelerators change the compute/bandwidth ratio in favor of such schemes.
  • Relevance: incoherence/rotation preprocessing is upstream-compatible with any base representation we choose — it makes weights more Gaussian, which is exactly what codebook, lattice, and progressive-residual encodings want. QuIP#/QTIP already rely on it.

2.3 GGUF block formats: k-quants, i-quants, ternary types (llama.cpp)#

What they are. llama.cpp's zoo: legacy Q4_0/Q4_1/Q5_x/Q8_0 (per-block scale ± min); k-quants (Q2_K…Q6_K) with two-level scale hierarchies and per-tensor mixes (the _S/_M/_L file types give sensitive tensors more bits); i-quants (IQ1_S…IQ4_XS) which are codebook/grid-based sub-4-bit formats requiring an importance matrix (imatrix, calibration-derived) for quality; ternary TQ1_0/TQ2_0; and MXFP4 (added Aug 2025 for gpt-oss, whose FFN weights ship natively in MXFP4).

  • Memory/bandwidth: proportional (dense decode). Q4_K_M ≈ 4.9 GB for an 8B model. Practical measurements put Q4_K_M ~+0.08 PPL over FP16 on Llama-3-8B class models.
  • Quality per bit: k-quants beat MLX affine at matched bpw — one practitioner measurement (Feldman, 2026): Q4_K_M at 4.88 bpw gives 0.0208 nats KL-to-FP16 vs MLX affine 4-bit (4.69 bpw) 0.0577 nats, ~2.8× better; similar at 5-bit. The reason is the two-level scale hierarchy and mixed per-tensor precision.
  • Apple Silicon / Metal status — the critical bandwidth-vs-compute datapoint: all GGUF types have hand-written Metal kernels, but i-quants pay a large decode penalty on Apple GPUs. ikawrakow (llama.cpp discussion #5617): 7B IQ-quant ~53.9 t/s vs Q4_0 63.1 t/s on M2 Max 30-core GPU, despite reading ~half the bytes; vs an RTX-4080 the gap for IQ2_XS is 3.5× (only 2× for Q4_0). Community measurements agree the codebook lookups make dequant compute-bound on Apple Silicon (and older CPUs). Lesson: on Metal, "fewer bits" only converts to "faster tokens" if the decode path stays trivially cheap (shift/mask + multiply-add), which is exactly what k-quants and MLX affine do and what LUT-heavy formats do not.
  • Limitation: static, uniform-per-tensor decisions; no notion of loading part of a weight's information.
  • Extension: the imatrix (importance) infrastructure is a ready-made per-block sensitivity signal usable for residual allocation.

2.4 MLX native quantization (our primary substrate)#

Formats (mlx.core.quantize / mlx.nn.quantize docs, MLX 0.32): default affine mode — bits ∈ {2,3,4,5,6,8}, group size ∈ {32,64,128} (default 4-bit/g64), per-group scale+bias, ŵ = round((w−β)/s); plus mxfp4 (E2M1, g32, E8M0 scale), mxfp8 (g32), nvfp4 (g16, E4M3 scale). quantize_input=True enables input/activation quantization for linear layers. All modes have first-class Metal kernels (fused dequant-matmul); quantized-KV attention supported via mlx_lm.generate --kv-bits {2,4,8} --kv-group-size.

Learned quants in mlx-lm (LEARNED_QUANTS.md): DWQ (distilled weight quantization — distills a 16/8-bit teacher into the quantization parameters (scales/biases) of a low-bit student; works best at 2–4 bit; community measurements ≈ +0.6 effective bits of quality); AWQ port; GPTQ-style; dynamic_quant (per-layer sensitivity-based bit allocation: layers that hurt most get more bits). This is the closest thing to a maintained, Apple-first learned-PTQ toolchain, and it's pure Python on top of MLX — directly hackable for our experiments.

  • Memory/bandwidth: proportional; MLX 4-bit ≈ 4.5 GB for 8B. MLX vs llama.cpp decode speed on identical Macs is within ±10–20% either way depending on version/model; MLX tends to win on 4-bit 7–30B decode, llama.cpp on prefill (mature simdgroup matmuls). Ollama switched its Apple Silicon backend to MLX (2026), and M5 "Neural Accelerators" give MLX further headroom.
  • Quality: plain affine 4-bit/g64 trails Q4_K_M slightly (see §2.3); group-size 32 and/or DWQ closes the gap.
  • Main limitation: uniform affine only — no non-uniform codebooks, no two-level scales, no sub-2-bit, no nested/progressive layout. Quantization is chosen once at convert time; the runtime has no concept of refining a weight after load.
  • Extension: MLX's mode plug-point plus custom Metal kernels is where a progressive/residual format would be implemented. The DWQ distillation loop is also the obvious way to calibrate a low-bit base to be maximally correctable by its residuals.

2.5 Vector / additive / lattice / trellis codebook quantization (the 2-bit frontier)#

QuIP (Chee et al., NeurIPS 2023): incoherence processing (random orthogonal pre/post rotations) + adaptive rounding; first theoretical analysis at LLM scale; first "viable" 2-bit.

QuIP# (Tseng et al., ICML 2024): randomized Hadamard incoherence + E8-lattice 8-dimensional codebook (E8P) + inter-layer fine-tuning. Higher bitrates built via residual vector quantization (RVQ — quantize, then quantize the residual with another codebook; e.g., 4-bit = 2+2). Llama-2-70B Wiki2 PPL (no FT): FP16 3.12 → E8P 2-bit 4.16 (vs 5.90 for QuIP scalar 2-bit). First PTQ where 3-bit scaled better than 4-bit. >3× faster than FP16 inference (CUDA); codebook decodable in <4 instructions/weight due to E8 symmetry.

AQLM (Egiazarian et al., ICML 2024, arXiv 2401.06118): additive quantization — each weight group is a sum of several codewords from learned 8-dimensional, 2^16-entry codebooks, trained by beam search + block-wise then end-to-end fine-tuning. First scheme Pareto-optimal below 3 bits; Pareto-optimal bitwidth ≈ 2.5 bpw. PV-Tuning (NeurIPS 2024) adds representation-agnostic fine-tuning of discrete+continuous params → first Pareto-optimal 2-bit Llama-2. Cost: ~720 GPU-hours for a 70B; 1 MiB codebooks blow L1 cache → AQLM decode is slow (20.6 tok/s vs QuIP# 106.3 on 2-7B, per QTIP measurements).

GPTVQ (van Baalen et al., Qualcomm, ICML 2024): GPTQ-style Hessian-interleaved updates extended to non-uniform VQ (1–4D); codebooks themselves compressed (int + SVD). 70B processed in 3–11 h. Notably demonstrated simultaneous DRAM-footprint and latency reduction on a mobile-class Arm CPU — one of the few codebook methods validated on unified-memory consumer silicon rather than discrete GPUs.

QTIP (Tseng et al., NeurIPS 2024): trellis-coded quantization — stateful codes over long sequences instead of fixed-dim VQ; with compute-based "bitshift trellis" codebooks there is no large LUT, decode is ~2 instructions/weight, and matvecs run at >80% of peak GPU memory bandwidth. Beats QuIP#/AQLM at all bitrates; QTIP-without-finetune ≈ QuIP#/AQLM-with-finetune. EXL3 (exllamav3) is a streamlined QTIP variant productized for consumer GPUs — Llama-3.1-70B coherent at 1.6 bpw, 70B in <16 GB VRAM. CUDA-only.

NestQuant (Savkin et al., ICML 2025, arXiv 2502.09720): self-similar nested lattices (Gosset/E8), information-theoretically near-optimal for low-precision matmul, covering weights and activations. Also NeurIPS 2025 work on learned grouped lattice VQ (learnable generator matrices, Babai rounding). Lattice methods are converging with VQ methods.

  • Memory reduction: the whole point — 2.0–2.5 bpw with usable quality (~8× vs FP16).
  • Bandwidth reduction: conditional. Weight bytes drop 8×, but decode cost decides whether that becomes tokens/sec: QTIP/QuIP# reach near-bandwidth-limit on NVIDIA; AQLM does not; and the Apple-GPU evidence from i-quants (§2.3) says LUT-based decode may leave Metal compute-bound. No published Metal implementation of QuIP#/AQLM/QTIP exists as of this writing.
  • Calibration: all need calibration; AQLM/QuIP#/QTIP quality depends significantly on (expensive) fine-tuning.
  • Main limitation for us: CUDA-first ecosystems; heavy encode cost; fixed bitrate at encode time.
  • Extension opportunity: RVQ/additive structure is inherently progressive — codeword sums can be truncated (see Drop-by-Drop, §2.7). A Metal "bitshift-trellis" decoder (no LUT) is plausibly the right way to get QTIP-class quality on Apple GPUs; nobody has published one.

2.6 Extreme low-bit: ternary, 1-bit, and low-bit QAT#

BitNet b1.58 (Ma et al., Microsoft, arXiv 2402.17764): ternary {−1,0,+1} weights, trained from scratch; claims parity with FP16 at same params/tokens, with large latency/memory/energy wins. Not a post-training transform → excluded as a solution by charter §14, but relevant as an existence proof of ~1.58-bit information sufficiency and for its inference stack. bitnet.cpp (arXiv 2410.16144; ACL 2025): CPU LUT kernels — I2_S (lossless MAD-based), TL1 (ARM), TL2 (x86); 1.37–5.07× speedups on ARM; runs a demo on Apple M2, and TL2_0 reaches 7.45 tok/s for a 100B ternary model on an M2 Ultra — an interesting datapoint that sub-2-bit makes 100B-class models CPU-feasible on Macs. Note: ternary kernels are CPU-only (NEON/AVX LUTs), no Metal GPU path; llama.cpp's TQ1_0/TQ2_0 exist but were measured badly inaccurate for BitNet models by the bitnet.cpp authors.

BiLLM (Huang et al., ICML 2024): first 1-bit PTQ — Hessian-selected salient weights get binary residual approximation (a second binary pass over the residual: again base+residual!), non-salient bell-shaped weights get optimal-split binarization. ~1.08–1.11 bpw; LLaMA2-70B PPL 8.41; 7B binarized in 0.5 h on one GPU. Quality is far from FP16 (usable-ish, clearly degraded); ARB-LLM (2024) refines it.

ParetoQ (Liu et al., Meta, NeurIPS 2025, arXiv 2502.02631): unified QAT-finetune framework across 1/1.58/2/3/4-bit. Key findings: ~10% of training budget on QAT finetuning suffices; sharp representational transition between 2 and 3 bits — at ≥3 bits finetuned models stay close to the pretrained distribution ("compensation"); at ≤2 bits representations change drastically ("reconstruction"). Ternary/2-bit/3-bit beat 4-bit on size-accuracy Pareto in their runs.

EfficientQAT (Chen et al., ACL 2025, arXiv 2407.11062): block-wise QAT (Block-AP) + end-to-end training of quant params only (E2E-QP). 2-bit Llama-2-70B on a single A100 in 41 h, −3% accuracy vs FP16 (69.48 vs 72.41); w2g64 Llama-2-7B PPL 6.86 vs 5.47 FP16. This is the realistic quality ceiling for "2-bit that still behaves like the original model" with modest compute.

  • Implication for localvm-research: ParetoQ's 2↔3-bit transition + EfficientQAT numbers suggest a 2-bit base is recoverable with light training but is near the edge; a 2.5–3-bit-effective base (or 2-bit + streamed residual) is the safer floor if the base must preserve routing/decision structure without full QAT.

2.7 Multi-precision, nested, progressive, and residual representations ⭐ (most relevant family)#

Any-Precision LLM (Park et al., ICML 2024 oral, arXiv 2402.10517; code SNU-ARC/any-precision-llm): store ONE n-bit "parent" model (non-uniform, incremental-upscaling from a 3-bit seed) in bitplane layout; any child bit-width 3…n is obtained by reading only the top-k bitplanes. Memory: supporting {3,4,5,6,7,8} bits costs 8.4 GB instead of 29.9 GB for separate models (3.56× saving). Crucially, "any runtime request of reduced bit-width directly translates into proportional speedup, as we can simply load the specified number of bits" — i.e., bytes-read-per-token scales with chosen precision, not stored precision. Engine has custom CUDA kernels (bitplane layout, bit-transpose, merged table lookups). No Metal port exists.

Matryoshka Quantization / MatQuant (Nair et al., Google DeepMind, ICLR 2025 oral, arXiv 2502.06786): exploit MSB-nesting of integers — co-optimize (via QAT or OmniQuant-style PTQ) a single int8 weight tensor so that slicing its top 4 or 2 bits yields good int4/int2 models; int2 extracted this way is up to 10% more accurate than dedicated int2 QAT; an int2-FFN Gemma-2 9B beats an int8-FFN Gemma-2 2B. Interpolative bit-widths (int3/int6) come for free.

Drop-by-Drop / multi-bitwidth additive codebooks (arXiv 2606.12876, Feb 2026): grounds multi-precision PTQ in information-theoretic successive refinement; trains additive (AQLM-style) codebooks with Matryoshka supervision so that ordered subsets of codebooks give accurate partial reconstructions — progressive compression by literally dropping codebooks at inference. Explicitly motivates "run-time hardware-aware dynamic loading". This is the closest published object to a "progressive residual base representation," and it is brand new — no systems implementation, no SSD tier, no Apple port.

Progressive Mixed-Precision Decoding (PMPD) (Chen et al., Samsung AI, ICLR 2025, arXiv 2410.13461): phase-aware precision — higher-precision weights for prefill, lower for decode, and progressively lower precision as generation deepens (later tokens tolerate more error), with task-/prompt-adaptive schedulers. 3.8–8.0× decode throughput on an NPU. Validates, at small scale, the premise that required weight precision is token-position-dependent and can be scheduled at runtime.

Related: AnyBCQ (2025) — binary-coded multi-precision with hardware-efficient bit-plane access; Squeeze10-LLM (2025) staged sub-2-bit PTQ; the mixed-precision survey arXiv 2510.16805 (Oct 2025) taxonomizes this whole space (§3.1 covers Any-Precision, PMPD, M2Cache).

  • Memory: one artifact serves all precisions (≈ cost of the highest).
  • Bandwidth: the defining feature — bytes/token = f(precision requested now), decoupled from storage. This is breakthrough criterion B in embryo.
  • Quality: MatQuant int2-slice ≈ or better than dedicated int2; Any-Precision children match dedicated SqueezeLLM-class models at each width.
  • Apple status: none. All engines are CUDA (or NPU simulators). Bitplane decode is bit-manipulation-heavy — Apple GPU/AMX behavior unknown; a Metal bitplane-matvec kernel is an obvious micro-experiment (ties to charter Exp. D/E).
  • Limitation: all of these still load the full chosen precision for every weight every token — precision is scheduled globally (per phase/token), not per weight-block; and the lowest usable slice is ~2–3 bits.

2.8 Random-basis and lossless recompression (orthogonal tricks)#

SeedLM (Shafipour et al., Apple + Meta, ICLR 2025, arXiv 2410.10714): compress each weight block into a seed + coefficients of an LFSR-generated pseudo-random basis — at inference, regenerate the basis from the seed and reconstruct the block. Data-free, 3–4-bit effective, ~zero-shot parity with calibration methods at 4-bit; trades memory bandwidth for free compute (PRNG regeneration), which is exactly the right trade on bandwidth-bound hardware — FPGA demo ~4× speedup at 70B. Apple authored this: conceptually adjacent to "weights need not be stored, only recoverable."

DFloat11 (Zhang et al., NeurIPS 2025, arXiv 2504.11651): lossless Huffman coding of BF16 exponents → ~30% size cut (≈11 bpw), bit-identical outputs, GPU online-decompression kernels. Cautionary datapoint: even with careful kernels, decode costs ~2–3× tokens/s vs uncompressed-in-VRAM on 8–32B models (it only wins when the alternative is offload). Lossless coding shrinks capacity, not effective bandwidth — entropy decode sits on the critical path. CUDA-only.

2.9 KV-cache quantization#

  • KIVI (Liu et al., ICML 2024, arXiv 2402.02750): tuning-free 2-bit KV — keys per-channel, values per-token (asymmetric, matches outlier structure), small FP16 sliding window; 2.6× peak-memory cut, 2.35–3.47× throughput (batch effect). Inspired HF Transformers' KV quantization.
  • KVQuant (Hooper et al., NeurIPS 2024, arXiv 2401.18079): per-channel pre-RoPE key quant + non-uniform sensitivity-weighted datatypes + per-vector dense-and-sparse outliers → 3-bit KV with <0.1 PPL degradation; enables 1M-token context for LLaMA-7B on one A100.
  • Coupled Quantization (NeurIPS 2024): exploits inter-channel dependence to reach ~1 bit/channel-equivalent KV.
  • Apple Silicon status — good: llama.cpp --cache-type-k/v {q8_0,q5_0,q4_0,iq4_nl} with flash attention on Metal (q8_0 ≈ half KV memory, negligible loss; q4_0 noticeable on long reasoning); mlx-lm --kv-bits {2,4,8} (group 64 default). Community work (KVSplit) confirms asymmetric K/V precision pays on Metal. Google's 2026 sub-3-bit KV result (picked up in llama.cpp discussion #20969, "TurboQuant", with working Metal kernels at 3.25/4.25 bits) is being adopted.
  • Relevance: KV bytes/token grow with context and can rival weight bytes at long context on 48 GB; any working-set argument must model both. KV quantization is the already-solved part of the working-set problem on Macs.

2.10 Precision granularity: per-layer / per-channel / per-token / dynamic#

Cross-cutting rather than a single method: per-channel scales are standard (AWQ/GPTQ groups); per-layer bit allocation is in mlx-lm dynamic_quant and GGUF _M mixes; per-token/phase precision is PMPD (§2.7); dynamic runtime precision selection appears in Any-Precision serving and M2Cache (below). The mixed-precision survey (arXiv 2510.16805) is the map of this space. MoE note: for MoE models, per-expert precision (hot experts high-bit in RAM, cold experts low-bit near SSD) is an obvious composite nobody ships on Mac; but the energy study arXiv 2508.06978 warns that naive SSD expert-offload raises per-token energy up to ~12× vs HBM — prefetch hides latency, not energy or bandwidth.

2.11 Quantization + offload hybrids (closest existing systems to our target)#

M2Cache (arXiv 2410.14740): neuron-level modularization + importance ranking + dynamic sparse mixed-precision quantization + three-level cache (GPU HBM ← DRAM ← SSD). Highest-importance neurons stay high-precision and cached; low-importance ones live at low precision on lower tiers. Up to 14× throughput vs baseline offload on RTX 3090-class hardware, 70B on 24 GB VRAM. This is the closest published architecture to charter §13 — but: Linux/CUDA, discrete-GPU tiering assumptions (PCIe copy costs that don't exist on unified memory), static importance ranking, and no progressive-precision refinement (a neuron is fetched at one precision, not refined).

3. Relevance to localvm-research#

3.1 What quantization gives us as a base representation#

The working-set-decoupling question needs a representation where bytes read per token is a runtime variable. The literature supplies four composable building blocks:

  1. A 2–4-bit affine base with native Metal speed (MLX affine / k-quants) — the only formats today where fewer bits reliably equal proportionally fewer nanoseconds on Apple GPUs. A 2-bit/g32 (+DWQ-calibrated) MLX base is implementable now; expected quality per EfficientQAT/ParetoQ: degraded but structurally faithful (PPL +~1.4 on 7B-class at w2g64 with training; worse pure-PTQ).
  2. Nested/bitplane layouts (Any-Precision, MatQuant) — store 6–8 bits, read 2–4. MSB-slicing means the "residual" is literally the next bitplane: refinement = read more planes of the same tensor, perfectly sequential, prefetchable, and idempotent. This is the natural on-SSD layout for progressive weight materialization (charter Exp. D).
  3. Additive/RVQ codebooks with successive-refinement training (QuIP#'s RVQ, Drop-by-Drop) — higher quality per bit than bitplanes at the same budget, refinement = add codewords; but decode-cost risk on Metal (i-quant lesson) unless a QTIP-style computed (LUT-free) codebook is used.
  4. Error-side instruments: SqueezeLLM/BiLLM's dense-and-sparse and KVQuant's outlier streams show that a tiny, importance-ranked sparse residual captures outsized quality — a cheap, cache-friendly correction channel that can be paged independently of the dense base.

3.2 The specific opening (what is NOT in the literature)#

Verified against everything above, the following combination does not exist:

  • Progressive precision as a memory hierarchy. Any-Precision/MatQuant/Drop-by-Drop keep all bitplanes/codebooks resident and slice for latency; PMPD schedules precision globally per token phase. Nobody stores the low-bit base resident in unified memory and treats higher-order residual planes as a demand-paged tier on NVMe, fetched per-block, conditioned on (layer sensitivity × current-token need × cache state). That system would make RAM bound the base size (e.g., 2 bits/param ≈ 17.5 GB for a 70B) while total quality lives on disk — precisely resident size ≠ total size ≠ bytes/token.
  • Decision-stability-driven refinement. No published method decides how many residual planes to fetch based on whether the token decision (top-1 margin / top-k set) is already stable — PMPD's schedulers are position-based, not confidence-based; Drop-by-Drop's profiles are static. This connects quantization directly to charter §4.10 / Exp. G: fetch residuals only when the low-bit forward pass is uncertain.
  • Apple Silicon kernels for any of it. No Metal implementations exist for: bitplane matvec (Any-Precision), MSB-sliced decode (MatQuant), trellis decode (QTIP/EXL3), E8/additive codebooks (QuIP#/AQLM). Unified memory actually helps here versus CUDA: a refined block is written once and visible to the GPU without PCIe traffic, and the SSD→RAM→GPU path has no copy step. The i-quant Metal evidence defines the design constraint: decode must be shift/mask-cheap (bitplanes and bitshift-trellises qualify; big LUTs do not).
  • Residual-aware calibration. DWQ/EfficientQAT optimize a single bit-width; MatQuant co-optimizes slices; but nobody calibrates a base jointly with a sparse importance-ranked residual stream under a bytes-per-token budget ("make the 2-bit base maximally correctable by its cheapest residuals"). mlx-lm's DWQ loop is the natural place to prototype this on-device.

3.3 Falsifiable premises to test first (feeds Exp. D/E/H)#

  1. Metal bitplane decode cost: does a 2-of-8-bitplane matvec on M-series GPU run at ≥70% of the speed of a dense 2-bit affine matvec? (If not, MSB-sliced packed MatQuant layout instead of bitplanes.)
  2. Quality-vs-planes curve: on a 7–8B model, measure PPL/KL/greedy-token-agreement at 2, 2+1, 2+2 … planes (charter Exp. D). MatQuant/Any-Precision predict smooth improvement; the open question is how few blocks need refinement to recover most of it.
  3. Residual locality: are the blocks whose refinement matters stable across tokens/domains (Exp. B/C)? If yes, an SSD residual tier with an LRU of refined blocks beats static mixed precision; if no, bandwidth math likely kills it (cf. arXiv 2508.06978's energy warning).
  4. Bandwidth accounting: a 48 GB M5 Max has O(500+) GB/s unified memory vs O(6–8) GB/s NVMe — residual fetches must therefore be ≤~1–2% of weight bytes per token, or fully overlapped/amortized across tokens. This ratio, not quality, is the most likely failure mode; measure before building (Exp. H).

Sources#