localvm-research

Research / research/research_gaps.md

research gaps

draft

research/research_gaps · created Tue Aug 11 2026 20:00:00 GMT-0400 (heure avancée de l’Est) · Simon-Pierre Boucher

Research Gaps: Candidate Approaches for Over-Budget LLM Execution on Apple Silicon#

Phase 3 deliverable (charter §6). Each gap below is a substantially different candidate mechanism, generated by crossing the four decoupling axes identified in research/state_of_the_art.md §9 (precision, sparsity, sharing/delta, speculation) with the four systems levers (residency policy, layout, prefetch, verification), then pruning every cell already occupied per SOTA §7.1. Every gap cites the specific systems whose limitation it exploits, and every falsifying experiment is runnable on the target machine (Apple M5 Max, 48 GB unified memory, AP2048Z NVMe, MLX 0.32 / Metal, no CUDA).

Binding substrate numbers that every gap must respect (expH, this repo, 2026-08-12):

SSD random read:   ~13 GB/s at 1 MiB QD8;  67 MB/s at 4 KiB QD1  (200× cliff)
Access contract:   16 KB-aligned blocks, ≥256 KiB effective reads at QD≥4–8
Per-token budget:  ≤~650 MB SSD reads/token for ≥5 tok/s (50% I/O share);
                   ≤~1.3 GB/token theoretical limit
Overlap:           SSD reads under full Metal matmul load lose <5% — prefetch is free
RAM:SSD ratio:     ~35:1 best case → non-overlapped residual traffic must stay
                   ≲1–2% of weight bytes/token, or be amortized across tokens
Decode rule:       Metal kernels must be shift/mask-cheap (i-quant LUT penalty)
Agreement priors:  ~91% same-top-token at 4-bit; 69–75% acceptance at 25–40% depth;
                   >90% QuantSpec precision-draft acceptance; 5–15% knife-edge tokens

Gaps are grouped by mechanism class (A–H). Groups do not imply combinations; each gap stands or falls on its own falsification criterion. Contrarian gaps (§H) are designed to kill the project's premises cheaply — per charter §17, they are as valuable as the rest.

A. Progressive representations and refinement#

G01 — ResidualPager: progressive precision as a demand-paged memory hierarchy#

  1. What existing systems do. Any-Precision LLM (ICML 2024), MatQuant (ICLR 2025), BitStack (ICLR 2025), RRQ, and Drop-by-Drop store nested/progressive weight encodings whose quality scales with bits consumed; each slice matches dedicated models at that bitrate. M2Cache tiers neurons across HBM/DRAM/SSD at mixed precision.
  2. What they do not do. All §5 systems keep every plane resident and pick the operating point statically per deployment; none pages residuals from storage; none reports bytes-read-per-token (SOTA §5 common ledger). M2Cache never upgrades a fetched neuron's precision (no refinement) and assumes PCIe/CUDA.
  3. Why it might matter. A 2–3-bit base of a 70B model is ~17.5–26 GB — resident on a 48 GB Mac — while full quality lives on NVMe. RAM would bound base size, not model size: exactly the charter §2 decoupling (resident size ≠ total size ≠ bytes/token).
  4. Plausible reason it could work. Successive-refinement theory (Drop-by-Drop's grounding) guarantees monotone quality-in-bytes; BitDelta shows corrections quantize to ~1 bit near-losslessly; expH shows the SSD can deliver ~650 MB/token in ≥256 KiB blocks with free compute overlap; unified memory makes a fetched residual GPU-visible with zero copy — the mechanic discrete-GPU prior work lacked.
  5. Biggest reason it might fail. The 35:1 RAM:SSD ratio: if quality recovery needs residuals for a large, unstable fraction of blocks each token, traffic blows the 1–2% budget and cannot be hidden. BitStack already reports reconstruction overhead slowing inference even with everything resident.
  6. Smallest falsifying experiment. expD + expB. On a 7–8B model, measure KL/agreement at 2, 2+1, 2+2… planes with per-block refinement ranked by sensitivity; then Jaccard(t, t+1) of the refined-block set. Kill: reaching KL ≤ 0.05 to FP16 requires touching >20% of residual bytes per token AND temporal reuse of the refined set is <50% — i.e., steady-state fetch traffic >650 MB/token for a 70B-class model.

G02 — MarginGate: decision-uncertainty-gated residual escalation with a correctness story#

  1. What existing systems do. QSpec/QuantSpec/ML-SpecQD draft with a low-precision cast of the same weights and verify exactly — but both operand sets stay resident and every token is verified. BiLD escalates on low confidence (no guarantee, two separate resident models). CALM gives distribution-free calibrated early exit — for the depth axis only. PMPD/QuickSilver schedule precision per token position, feed-forward, with no verification.
  2. What they do not do. Nobody decides how many residual planes to fetch from storage based on whether the token decision is already stable (top-2 margin, entropy, or a learned error predictor), and nobody attaches either rejection-sampling exactness or CALM-style calibrated risk to that decision, and no §5 or §6 paper optimizes or even reports bytes/token (SOTA §7.3.2 — the verified open intersection).
  3. Why it might matter. If ~91% of tokens at 4-bit (and possibly 70–85% at 2–3-bit) are decided identically to FP16, then escalation — the only expensive event — happens on 5–15% of tokens, and residual streaming cost is paid only there: bytes/token collapses to base-read + (escalation rate × residual fetch).
  4. Plausible reason it could work. The correctness machinery already exists and is proven: speculative rejection sampling is exact; CALM calibration is distribution-free. Margins demonstrably carry signal (the entire early-exit line works). EMNLP-2025 residual-magnitude features predict quantization breakage at ρ = 0.82, giving a cheap learned signal beyond raw margin.
  5. Biggest reason it might fail. The dangerous quadrant: tokens where the base is confidently wrong (high margin, wrong token). If disagreements are not concentrated at low margin, gating misses exactly the errors that matter — and the 4-bit KLD tail (max 22–26 nats) plus the "flips are load-bearing tokens" evidence says this quadrant is not empty. Also: escalation may exceed 40% on reasoning/code (Sirius's finding that sparsity specifically damages reasoning is a warning for any cheap-first scheme).
  6. Smallest falsifying experiment. expG. On 1B→8B models, 2-bit/3-bit/4-bit base vs FP16 reference, log per token: (base top-2 margin, base entropy, agreement with reference, reference token rank under base). Domains: code, math, chat, FR, EN, reasoning. Kill: AUROC of margin (or margin+learned features) for predicting disagreement <0.8, or required escalation rate at 99% effective agreement >40% on any major domain. (This one measurement also arbitrates G17, G23, G24.)

G03 — DeltaKernel: incremental refinement ΔY = (ΔW)·X instead of recompute#

  1. What existing systems do. Every verification/refinement scheme in the literature (LayerSkip, QSpec, QuantSpec, Sirius dense correction) recomputes the full layer with the better operand. GMRES-IR (Carson–Higham) refines a cheap factorization with residual-driven corrections and convergence guarantees — in numerical linear algebra, never written down for transformer inference (SOTA §6.6, §7.3.3).
  2. What they do not do. No published primitive computes only the correction term ΔY = (ΔW)X against cached activations, where ΔW is the streamed residual plane (sparse, low-rank, or bitplane), and adds it to the cached base output.
  3. Why it might matter. Refinement cost becomes proportional to residual size, not layer size: a 10%-density residual costs ~10% of the base matmul in both bytes and FLOPs. Without this, every escalation (G02) or block verification (G17) doubles compute, halving the whole family's economics.
  4. Plausible reason it could work. Linearity: for linear layers Y = WX, Y_full = Y_base + (ΔW)X exactly. Nonlinearities (softmax, SwiGLU, norms) only need recomputation downstream of corrected layers, and expF maps which layers tolerate deferred correction. Activation caching for a deferred block costs ~hidden_dim × layers × block_len × 2 bytes ≈ tens of MB — trivial next to weights.
  5. Biggest reason it might fail. Attention breaks the linear shortcut (Q/K/V corrections propagate through softmax non-linearly), so exactness holds per-linear-op, not per-layer; and a sparse/irregular ΔW matvec on Metal may be gather-bound and slower than a dense recompute (the Endor/i-quant lesson transposed to residuals).
  6. Smallest falsifying experiment. New micro (expE extension, expE_delta). Implement in MLX/Metal: (a) dense base matvec, (b) fused ΔY with 5/10/25%-density block-sparse ΔW, (c) full recompute at base+residual precision. Kill: the ΔY kernel at 10% density is not ≥2× faster than full recompute, or numerical error of base+ΔY vs direct full compute exceeds quantization noise (>1e-3 relative on layer outputs).

G04 — RDLayout: rate-distortion-optimal embedded weight bitstreams, compiled to the SSD contract#

  1. What existing systems do. EZW/SPIHT/EBCOT (JPEG2000) achieve per-block R-D-optimal truncation of significance-ordered bitstreams — 30-year-old, mature. BitStack unwittingly rediscovers embedded coding with SVD residual blocks but sorts by a global significance heuristic. GGUF i-quants use imatrix calibration to set static per-tensor precision. LLM-in-a-flash and Ripple optimize flash layout for co-activation, not for truncation.
  2. What they do not do. Nobody compiles weights into per-block embedded bitstreams with truncation points optimized against calibration-set logit KL (not MSE), aligned to the measured storage contract (16 KB pages, ≥256 KiB fetch bundles), so that "bytes loaded per matrix" is a continuous, precomputed quality knob (SOTA §5.2, notes/decomposition §4).
  3. Why it might matter. It converts layer sensitivity (expF) into layout, at compile time, with no runtime policy needed to get the first win: a static-but- nonuniform operating point strictly better than uniform bpw at the same bytes, and the substrate every dynamic gap (G01, G02, G17) would page from.
  4. Plausible reason it could work. Sensitivity-weighted allocation already beats uniform (SqueezeLLM, imatrix, LASER's "some matrices improve when truncated"); EBCOT's Lagrangian truncation is exactly the right optimizer once distortion = KL; the charter's compile stage (§14) explicitly budgets unlimited offline time.
  5. Biggest reason it might fail. Diminishing returns: imatrix/k-quant mixing may already capture most of the nonuniform-allocation gain, leaving <0.01 KL at matched bytes — real but not research. Also per-block KL attribution is approximate (block interactions), so the "optimal" truncation may be optimal only for the calibration distribution.
  6. Smallest falsifying experiment. expD + expF. On an 8B model, compare at matched total bytes: uniform 3-bit; imatrix-style per-tensor mixing; per-block KL-optimized truncation of a 2-bit+residual embedded code. Kill: per-block R-D allocation improves KL-to-FP16 by <15% relative over per-tensor imatrix mixing at the same byte budget.

G05 — MetalTrellis: LUT-free sub-3-bit decode kernels for Apple GPUs#

  1. What existing systems do. The 2–2.5 bpw frontier (QuIP#, AQLM, QTIP, EXL3, Any-Precision bitplane engines) is entirely CUDA; QTIP reaches >80% of peak GPU bandwidth using a bitshift trellis (~2 instructions/weight). llama.cpp's i-quants prove LUT-heavy decode loses on Apple GPUs (53.9 vs 63.1 tok/s despite half the bytes).
  2. What they do not do. Zero Metal implementations exist of bitplane matvec, MSB-sliced decode, or trellis decode (SOTA §7.3.5). Nobody knows whether the i-quant penalty applies to shift/mask-only progressive layouts on M-series GPUs.
  3. Why it might matter. Every progressive gap (G01–G04) presumes the base and its residual planes decode at near-dense speed on Metal. If only affine group-quant decodes fast, the design space collapses to MLX's existing formats and residuals must be affine-quantized deltas — a materially different (and weaker) encoding.
  4. Plausible reason it could work. Bitplane and bitshift-trellis decode are shift/mask-cheap by construction (satisfying the §1 decode rule); decode-bound kernels benefit from Apple's high ALU:BW ratio at batch 1 (dequant is free when memory-bound); MLX exposes custom-kernel plug points (mode).
  5. Biggest reason it might fail. Bitplane matvec needs bit-transpose gathers with poor memory coalescing on Apple's GPU cache hierarchy; trellis decode is sequential within a block, fighting SIMD-group width. The failure would be quantitative and final: the format tax eats the byte savings.
  6. Smallest falsifying experiment. expE variant (expE_decode). Implement three Metal matvecs at matched effective bpw (~2.5): MLX affine 2-bit+scales baseline; 2-of-8 bitplane; small bitshift trellis. Measure tok/s-equivalent throughput on 4096×4096. Kill: neither progressive variant reaches ≥70% of the affine baseline's throughput.

B. Sparsity as a paging policy (SwiGLU, post-training)#

G06 — TealPager: thresholded-SwiGLU sparsity as bytes/token reduction on Metal#

  1. What existing systems do. TEAL achieves 40–50% model-wide training-free activation sparsity on stock SwiGLU models (1.53–1.8× decode, Triton/CUDA); CATS 50% FFN-only; GRIFFIN selects 50% of FF params once per sequence; DIP picks masks given cache contents. DejaVu/LLM-in-a-flash proved sparsity-driven paging — for ReLU models only (0.2 GB/token vs 13.4 GB).
  2. What they do not do. No Metal kernel exists for any of them (SOTA §3.3); nobody has published bytes/token for thresholded-SwiGLU paging; all working-set statistics (Jaccard, reuse distance) in print are ReLU-era.
  3. Why it might matter. 40–50% is only ~2× — but it is the only training-free selectivity mechanism for dense stock models (Llama-3, Qwen3-dense), and 2× on the FFN dominates once attention/KV are handled; combined with a 20–30% RAM deficit it is the difference between thrashing and fitting.
  4. Plausible reason it could work. TEAL's thresholds are calibration-only; GRIFFIN's per-sequence "flocking" granularity is inherently prefetch-friendly (choose once, fetch 256 KiB bundles, reuse all sequence); expH says the fetch side is comfortable if masks are stable.
  5. Biggest reason it might fail. Two independent cliffs: (a) Sirius showed contextual sparsity specifically damages reasoning/GSM8K — the 99%-retention claims are perplexity mirages; (b) a 50%-sparse gather matvec on Metal may run slower than dense (no sparse tensor cores; Endor's indexing-overhead point), saving bytes on paper and losing tokens/s in practice.
  6. Smallest falsifying experiment. expA + expB (SwiGLU masks), then expE. Measure on Llama-3-8B: model-wide safe sparsity at ≤0.05 KL and GSM8K delta ≤1 pt; mask Jaccard(t, t+1) and per-sequence union growth. Kill: safe sparsity <30% model-wide, or Jaccard(t, t+1) <0.6 (mask churn makes paging traffic exceed the dense-read baseline), or (expE) the Metal gather kernel at 50% sparsity is slower than the dense matvec.

G07 — WandaTiers: importance scores as residency policy, with dense correction#

  1. What existing systems do. Wanda/SparseGPT compute cheap importance scores (|W|·‖x‖, Hessian) and delete weights — costing 27% PPL at 7B for 50% unstructured. Endor packs pruned weights into a dense-readable bitmap format (CUDA). Sirius restores reasoning quality lost to sparsity by dense-verifying ~11% of tokens at 78% of the efficiency gain. M2Cache ranks neurons statically for tier placement.
  2. What they do not do. Nobody uses one-shot importance scores as a reversible tier assignment — top-p% resident, bottom demoted to NVMe in a dense-readable format and re-materialized on demand — with a Sirius-style correction loop restoring the dense ceiling (SOTA §7.3.6). Pruning answers "total size"; this answers "residency".
  3. Why it might matter. It is the cheapest possible route to a working over-budget dense runtime: no re-encoding, no predictors, no new format — just placement. If cold-weight touch rates are low, it alone runs a 1.3–1.5× over-RAM model interactively.
  4. Plausible reason it could work. The demoted weights are, by construction, the ones whose absence changes outputs least (that's what the scores measure); misses can be served by zero (pruned behavior) immediately and corrected asynchronously; Sirius quantifies the correction cost at ~11% of tokens.
  5. Biggest reason it might fail. Wanda-style scores are global averages; the tokens that need demoted weights may be exactly the load-bearing ones ("Accuracy is Not All You Need": 25%+ flips for pruning-type interventions at matched accuracy). If per-token demand for cold weights is diffuse rather than rare, traffic is random, small, and constant — the 67 MB/s regime.
  6. Smallest falsifying experiment. expA (blocked variant). Demote the bottom-{20,35,50}% of FFN blocks by Wanda score; per token, measure (a) flip rate / KL vs dense with misses-as-zero, (b) how many demoted blocks would actually be needed to stay within KL ≤ 0.05. Kill: at 35% demotion, cold-block demand exceeds 5% of demoted bytes per token (≈ diffuse demand), or flip rate with Sirius-style 11% dense correction still exceeds 5%.

C. Cache, replacement, and residency policy#

G08 — MRURing: DBMIN-style per-tensor-class buffer pools with an MRU ring for cyclic scans#

  1. What existing systems do. llama.cpp mmap and the MLX mmap prototype delegate replacement to the kernel's approximate LRU; all published expert caches (Eliseev–Mazur, PowerInfer-2) use LRU/LFU variants. Databases solved this in 1985: DBMIN assigns a per-file policy from the known access pattern, and for looping sequential scans MRU is optimal while LRU is the worst case.
  2. What they do not do. No LLM runtime applies MRU to the dense-decode weight scan (which is a perfect cyclic scan, same order every token), or partitions the budget into per-tensor-class pools (pin embeddings; MRU ring for dense; ARC for experts; sliding window for KV) — SOTA §4.7 OS imports, apparently unpublished anywhere.
  3. Why it might matter. For a dense model overflowing RAM by X GB, MRU keeps a stable (model−X) resident set and re-reads exactly X GB/token — the theoretical floor — where LRU re-reads everything (the measured 0.025 tok/s MLX failure). At X = 3 GB, the floor is ~0.23 s/token of I/O on our SSD: a 1.1× over-budget model becomes usable by policy alone.
  4. Plausible reason it could work. The access pattern is fully known (static layer order — the "query plan"); the math is 40 years old; expH shows the re-read stream (large sequential slices at QD≥4) hits ~13 GB/s and overlaps freely with compute.
  5. Biggest reason it might fail. The window of usefulness is narrow: the floor itself caps overflow at ~650 MB (5 tok/s) to ~6.5 GB (0.5 tok/s) — MRU rescues slightly over-budget models only, and macOS gives no direct control over UBC eviction, so the ring must be implemented with explicitly managed wired buffers under the iogpu wired limit (engineering risk: the panic cliff).
  6. Smallest falsifying experiment. New micro (expI_cachesim) + expH numbers. Trace-driven simulation (traces are free: layer order is static) of LRU vs MRU vs OPT at 5–30% overflow, then a minimal MLX prototype streaming the overflow slice. Kill: measured end-to-end gain of the MRU prototype over kernel-LRU mmap <1.5× at 10–20% overflow, or wired-arena management overhead eats the gain.

G09 — ArcExperts: scan-resistant expert cache + explicit slice reads for MoE on macOS#

  1. What existing systems do. llama.cpp runs MoE overflow by mmap demand paging (kernel LRU); discussion #18758 measured that layout-aware explicit slice reads cut cold-decode reads/token 1418→370 (2.23×, +13–14% end-to-end); a two-tier expert cache is an open, unfilled feature request (#20757). ARC/2Q/LIRS ghost-list caches are standard in storage systems; nobody has published one for experts (SOTA §4.7).
  2. What they do not do. No shipping runtime combines: compiled 16 KB-aligned expert bundles, an ARC cache with ghost lists auto-partitioning recency vs frequency, QD≥8 explicit reads, and purgeable-warm demotion — on any OS, least of all macOS.
  3. Why it might matter. MoE is where selectivity is free (the router already picks ~2/8–8/128 experts); the entire gap between llama.cpp's current behavior and the achievable floor is cache/IO policy. Qwen3-30B-A3B/80B-class models on 16–48 GB Macs are the most immediately deployable win of the whole project.
  4. Plausible reason it could work. Every ingredient has isolated evidence (2.23× from layout alone; MoE temporal locality confirmed; ARC's superiority on mixed workloads is textbook); expert granularity (MBs) naturally satisfies the ≥256 KiB fetch contract.
  5. Biggest reason it might fail. Modern MoE routing may be so uniform (load-balanced by training) that no cache policy beats LRU by more than a few points — the gain would then be layout+QD only, which llama.cpp could (and may) merge upstream first, making this engineering, not research.
  6. Smallest falsifying experiment. expB (MoE-trace variant). Log real routing traces (llama.cpp/mlx-lm on Qwen3-30B-A3B) across domains; replay through LRU, MRU, ARC, LIRS, OPT simulators at cache = 25/50/75% of expert pool. Kill: ARC beats LRU by <5 points of hit rate at every cache size (policy headroom ≈ 0; only layout matters).

G10 — PurgeableWarm: kernel-cooperative weight caching via purgeable MTLHeaps#

  1. What existing systems do. Inference runtimes either wire weights (mlx wired limit; panic at the cliff, mlx-lm #883) or leave them to the page cache (opaque eviction). macOS uniquely exposes purgeable/volatile memory with reclaim notification (setPurgeableState), used by graphics apps since iOS, never by an LLM runtime (SOTA §4.7 import #9).
  2. What they do not do. No runtime maintains a warm weight tier the OS may reclaim under pressure — converting "jetsam/kernel-panic risk" into "graceful quality/latency degradation," and no runtime distinguishes wired-hot / purgeable-warm / SSD-cold as three managed tiers.
  3. Why it might matter. Every other gap that caches anything (G01, G06–G09, G13) needs a warm tier that coexists with the user's other apps on a 48 GB machine; without cooperative purgeability, the practical resident budget is far below nominal RAM and failure is catastrophic rather than gradual.
  4. Plausible reason it could work. The API exists and ships; reclaim is block-granular (per-resource/heap); a purged block is exactly a cache miss, which the G08/G09/G11 machinery already handles; re-fetch at 13 GB/s makes wrong purges cheap.
  5. Biggest reason it might fail. Per-use volatile→nonvolatile state transitions on the render path may cost too much (an API round-trip per block per token), and the OS's purge order is not model-aware — it may purge the blocks we need most, making quality under pressure worse than a self-managed fixed budget.
  6. Smallest falsifying experiment. New micro (expJ_purgeable). Fill a purgeable MTLHeap with weight blocks; run a matvec loop touching them; apply memory pressure (memory_pressure -l critical); measure transition overhead per access, purge granularity, and recovery latency. Kill: state-transition overhead >10 µs/block per token, or purges are so coarse/unpredictable that post-pressure recovery costs more than a fixed self-managed budget with SSD misses.

G11 — NeverBlock: anti-caching semantics — compute never stalls on a weight miss#

  1. What existing systems do. All paging runtimes (llama.cpp mmap, LLM-in-a-flash, PowerInfer-2) block on a miss (fault or explicit read stall); MoBiLE substitutes a smaller expert on miss (MoE-only, with training); anti-caching databases (H-Store, VLDB 2013) abort-and-restart transactions so no thread ever waits on disk.
  2. What they do not do. No LLM runtime has the invariant "a miss is served immediately by the resident approximation (base-precision block, zero, or substitute expert), the true bytes are fetched asynchronously, and the computation is optionally repaired later" — as an architectural rule rather than an ad-hoc trick (SOTA §4.7 import #5).
  3. Why it might matter. Tail latency: expH's averages hide P99 fetch latency (PowerInfer-2 reported +40.9% P99); a never-block runtime converts I/O jitter into bounded, measurable quality jitter — which G02's verification machinery can then repair. It decouples token cadence from storage latency entirely.
  4. Plausible reason it could work. With a G01-style representation, a "miss" has a natural graceful value: the resident low-bit base — an approximation with known, calibrated error (expF), not garbage. The repair path is G03's ΔY kernel applied one-to-a-few tokens late.
  5. Biggest reason it might fail. Error compounding across layers (QEP: near-exponential with depth): a miss at layer 10 contaminates all downstream layers for that token, so "repair later" may require replaying the whole suffix — at which point never-block is just deferred blocking with extra bookkeeping.
  6. Smallest falsifying experiment. expF. Inject base-precision substitution at single layers and at realistic miss patterns (Poisson over layers at 5/10/20% miss rate) on an 8B model; measure flip rate and KL with and without one-token-deferred ΔY repair. Kill: at a 10% miss rate, deferred repair fails to keep flips <2% (vs the blocking baseline's 0%) — i.e., misses must be repaired synchronously anyway.

D. Prefetch and prediction#

G12 — HiddenPrefetch: learned hidden-state → future-weight-block prediction#

  1. What existing systems do. Eliseev–Mazur apply layer k+1's router gate to layer k's hidden state (a static, 1-layer-lookahead heuristic) → 2–3 tok/s Mixtral on 11–16 GB; DejaVu/ShadowLLM exploit "slowly changing hidden states" to predict sparsity for FLOP-skipping (ShadowLLM: one early-layer predictor shadows all layers); Pythia (MICRO 2021) shows online-RL hardware prefetchers that optimize hit-rate-minus-wasted-bandwidth.
  2. What they do not do. Nobody trains a small predictor from hidden-state features to SSD prefetch decisions N layers ahead (N sized so lead time ≥ fetch latency), for dense-sparse blocks or residual planes rather than MoE experts, with wasted-bytes penalized (SOTA §3.1 extension; §4.7 import #6).
  3. Why it might matter. The paging gaps (G01, G06, G07) live or die on hiding fetch latency; expH says overlap is free if the runtime knows what to fetch ~1–5 ms early. Hidden states are richer context than any hardware prefetcher has ever had.
  4. Plausible reason it could work. ShadowLLM proved one early probe predicts all-layer needs with >15% accuracy gain over per-layer criteria; predictors cost <2.4% overhead in LLM-in-a-flash; a logistic probe per layer is microseconds on AMX.
  5. Biggest reason it might fail. Prediction quality may collapse exactly where it matters: rare blocks (the ones not already cached) are by definition the tail of the distribution, and PowerInfer-2's predictors cost 2.6 GB of DRAM at 47B — the apparatus can eat the budget it saves.
  6. Smallest falsifying experiment. expB + a probe. From expB traces, train a logistic probe: hidden state at layer k → block set needed at layer k+4. Kill: recall <70% at ≤2× overfetch for the *non-resident* block subset (measured against a realistic cache), or probe+metadata footprint >5% of the RAM budget.

G13 — DomainCache: temporary model specialization via semantic working sets#

  1. What existing systems do. PowerInfer places hot neurons statically from global power-law statistics; GRIFFIN picks FF blocks per sequence; prompt/prefix caches specialize KV, never weights. No system precomputes or persists per-domain weight working sets ("coding session" vs "French chat") and swaps them at session granularity.
  2. What they do not do. Nobody has even measured whether same-domain prompts reuse the same weight regions substantially more than cross-domain prompts on modern SwiGLU models (all locality numbers are ReLU-era) — charter expC exists precisely because this is unmeasured.
  3. Why it might matter. Session granularity changes the economics completely: fetches amortize over thousands of tokens, the 35:1 ratio becomes irrelevant, and the mechanism composes with everything (warm the G08/G09 caches from a domain profile at session start — seconds of sequential 13 GB/s reads).
  4. Plausible reason it could work. MoE routers demonstrably develop domain-correlated expert usage; LoRA's success proves task-relevant weight subspaces are small; if activation masks have any cross-prompt structure, a domain profile captures it offline for free.
  5. Biggest reason it might fail. Transformer weight usage may be dominated by token-level (syntactic) rather than domain-level (semantic) factors — every prompt touching the same high-frequency circuitry plus a thin idiosyncratic tail — giving within-domain overlap barely above cross-domain overlap, i.e., nothing to cache at session granularity.
  6. Smallest falsifying experiment. expC. Important-block sets (expA criterion) for 50 prompts × 6 domains on an 8B model; compute within-domain vs cross-domain Jaccard of the union working sets. Kill: mean within-domain minus cross-domain Jaccard <0.10 absolute, or the domain-specific tail is <10% of the working set (everything important is universal → a single static hot set suffices, use G07).

E. Decomposition and sharing as memory-hierarchy assignments#

G14 — TrunkDelta: shared cross-layer trunk resident, per-layer deltas paged#

  1. What existing systems do. Basis Sharing, DeltaLLM, and Relaxed Recursive Transformers show layers can be re-expressed as shared components + per-layer corrections (12–25% compression, light calibration); BitDelta shows finetune deltas quantize to ~1 bit near-losslessly, and iterating it yields a monotone stack of 1-bit masks. All are framed purely as compression ratios.
  2. What they do not do. Nobody assigns the decomposition to the memory hierarchy: shared trunk (touched by every layer, maximal reuse) wired in RAM; per-layer 1–2-bit delta masks streamed sequentially from SSD in layer order (SOTA §2.10 extension, §7.3 open intersection).
  3. Why it might matter. The trunk is read L times per token but stored once — its effective bytes/token amortize across layers, a reuse structure no flat encoding has. Delta streaming is perfectly sequential (layer order is static): the one access pattern our SSD serves at full 13 GB/s with trivial prefetch.
  4. Plausible reason it could work. BitDelta's ~1 bit/param deltas are real, measured structure; "Secretly Linear" reports ~0.99 Procrustes linearity between consecutive layers (adjacent layers nearly share a basis); sequential delta prefetch needs zero prediction machinery.
  5. Biggest reason it might fail. All published sharing needs light retraining (DeltaLLM: 30–40M tokens; RRT: distillation); zero-shot/calibration-only trunk extraction may degrade so much that deltas must carry most information — then delta bytes/layer ≈ layer bytes and nothing is saved. BitDelta's evidence is for finetune-vs-base pairs, not layer-vs-layer within one model — the transfer is unproven.
  6. Smallest falsifying experiment. New micro (expK_trunk, expD-adjacent). On an 8B model, fit a shared basis across layer groups by calibration-only alternating least squares (charter permits calibration passes); measure PPL/KL with trunk-only vs trunk+quantized-delta at delta budgets {0.5, 1, 2} bits/param. Kill: at trunk = 30% of total bytes, KL ≤ 0.1 requires deltas ≥3 bits/param (delta traffic ≈ flat-model traffic; no amortization win).

G15 — SpectralSplit: low-rank hot path resident + quantized sparse residual cold path#

  1. What existing systems do. SVD-LLM/SliceGPT keep the low-rank part and discard the residual (quality collapses past ~25–40% compression because the residual carries most spectral energy); CALDERA encodes W ≈ Q + LR jointly as a static 2–2.5 bpw format (CUDA, both terms always read); LASER shows selected late-MLP matrices improve when their high-order components are dropped.
  2. What they do not do. Nobody stores the SVD residual at all (SVD family) or treats the two terms as different tiers (CALDERA): low-rank factors resident (dense small GEMMs — the one decomposition Metal already runs fast), residual on SSD, fetched per-block only where LASER-style analysis says the residual matters (SOTA §2.8 extension).
  3. Why it might matter. Unlike bitplanes (G01), both components use existing, fast MLX kernels today — no custom Metal work gates a prototype; and LASER implies the residual tier can be skipped entirely for some matrices (negative bytes).
  4. Plausible reason it could work. Rank-r hot path costs r(m+n)/mn of dense bytes (~12% at r=256 for 4096²); per-matrix residual need is measurable offline; the split is exact by construction (base + residual = W), so quality is a pure function of which residual blocks are fetched — a clean knob for G02's gating.
  5. Biggest reason it might fail. Weights are not globally low-rank (SOTA §2.8's headline): the low-rank base alone may be so poor (unlike a 2-bit quantized base, which is uniformly mediocre) that residual demand is near-universal per token — worse-behaved than G01's precision split, with double the kernel launches.
  6. Smallest falsifying experiment. expD (SVD variant). On an 8B model, base = rank-{128,256} per matrix + 3-bit quantized residual in importance-ranked blocks; measure KL vs fraction of residual blocks applied, per matrix class. Kill: base-alone KL >1.0 nats and reaching KL ≤ 0.05 needs >50% of residual bytes for most matrices — i.e., strictly dominated by the G01 bitplane split at equal bytes.

G16 — SeedSynth: regenerate weights from seeds — trade bandwidth for idle ALUs#

  1. What existing systems do. SeedLM (Apple) replaces weight blocks with LFSR seeds
    • coefficients, regenerating pseudo-random bases at inference: 4-bit parity, data-free, ~4× speedup on an FPGA at 70B — explicitly trading memory bandwidth for free compute. No Metal port exists; DFloat11 shows the wrong version of this trade (entropy decode on the critical path → 2–3× slower).
  2. What they do not do. Nobody has tested generative reconstruction on Apple GPUs, where batch-1 decode is bandwidth-bound and 40 GPU cores sit partially idle — precisely the regime the trade targets; and nobody composes it with paging (seeds are ~100× smaller than blocks: a "prefetch" of seeds is nearly free).
  3. Why it might matter. If it works, it attacks bytes/token below the information-theoretic floor of stored-bits formats: resident bytes and streamed bytes both shrink to seeds+coefficients, and SSD traffic per block drops ~10×, trivially satisfying every budget in this document.
  4. Plausible reason it could work. LFSR generation is shift/XOR-only (passes the §1 decode rule, unlike DFloat11's Huffman); Apple authored SeedLM with edge inference as the stated motivation; the ALU:BW imbalance at batch 1 is real and measured.
  5. Biggest reason it might fail. Regeneration cost per weight might exceed the ~2 instructions/weight ceiling that QTIP shows is needed to stay bandwidth-limited; quality is 4-bit-parity at best (no path to 16-bit exactness — it cannot serve as a verification tier, only as a base), and per-block coefficient search at compile time is expensive for 70B-class models.
  6. Smallest falsifying experiment. expE variant (expE_seed). Metal kernel: LFSR-basis generation fused into matvec vs MLX affine 4-bit matvec at matched quality (SeedLM's published 4-bit parity settings on one 8B layer). Kill: fused generate+matvec throughput <80% of the 4-bit dense baseline (the compute trade loses on this GPU), or per-layer PPL parity claims fail to reproduce on a SwiGLU model.

F. Scheduling, speculation, and amortization#

G17 — BlockVerify: token-batch amortization of residual fetches (deferred verification)#

  1. What existing systems do. LayerSkip/Draft&Verify verify a drafted token block with one full-model pass (depth axis, all weights resident and read every verification); SpecExec shows ~20 accepted tokens per full offloaded-weight sweep turns 4.5 s/token into 4–6 tok/s; Kangaroo stops drafting on low confidence; QuantSpec verifies with resident 16-bit weights every block.
  2. What they do not do. Nobody drafts N tokens with a resident low-bit base and then performs one residual-augmented verification sweep whose streamed residual bytes amortize over the accepted block — i.e., SpecExec's economics applied to residual planes instead of full weights, with rollback on rejection (SOTA §8 open: "block-deferred verification policy").
  3. Why it might matter. It divides residual bytes/token by the acceptance length: if base-vs-refined agreement resembles QuantSpec's (>90%), expected block length ≥7–10 and the 1–2% traffic ceiling relaxes to 10–20% of weight bytes per sweep — a qualitatively easier target than G02's per-token gating.
  4. Plausible reason it could work. The rejection-sampling correctness proof carries over unchanged (draft distribution = base model, target = refined model, same support); acceptance priors are measured (91% at 4-bit); verification sweeps read residuals sequentially by layer — the SSD-friendly pattern; MLX already ships speculative verification kernels on Metal (ReDrafter).
  5. Biggest reason it might fail. Acceptance at a 2–3-bit base (required for real oversubscription) is unmeasured and the ParetoQ transition suggests it may crater; and rollback economics: one early rejection wastes the whole drafted suffix and the residual sweep — at low acceptance the scheme is strictly worse than reading residuals per token.
  6. Smallest falsifying experiment. expG (acceptance-length variant). Simulate exactly: base = 2/3-bit MLX quant drafting blocks of 8–16, verify with 4/8-bit refined weights, standard rejection rule; measure expected accepted length per domain. Kill: mean accepted block length <4 at a 3-bit base on chat/code (amortization factor too small: sweep bytes ÷ 4 still exceeds the per-token budget for 1.5× over-RAM models).

G18 — SpecSweep: dense over-RAM models via tree drafting + amortized model sweeps (no residuals)#

  1. What existing systems do. SpecExec achieves Llama-2-70B (4-bit) at 4–6 tok/s on consumer GPUs by verifying huge draft trees in one pass over RAM-offloaded weights (~20 accepted tokens/sweep, 10.6–18.7× over sequential offloaded decoding) — CUDA, PCIe staging. llama.cpp ships --model-draft on Metal but never composes it with an overflowed target; MLX mmap overflow decodes at 0.025 tok/s.
  2. What they do not do. Nobody has built the SpecExec economics on unified memory
    • NVMe (no PCIe staging, zero-copy Metal buffers, and a 13 GB/s sweep of overflow bytes) — the pure port is unoccupied and needs no new representation at all (SOTA §6.4: "concepts portable").
  3. Why it might matter. It is the only gap that helps dense stock models beyond RAM today with zero model transformation: overflow of 20 GB sweeps in ~1.5 s; at 20 accepted tokens/sweep that is ~13 tok/s of I/O ceiling — interactive. It also provides the honest baseline every fancier gap (G01, G17) must beat.
  4. Plausible reason it could work. The arithmetic above uses only measured numbers (expH ceiling, SpecExec acceptance); MRU (G08) makes the swept overflow set stable; the draft model (1B, 4-bit, ~0.7 GB) is noise in the RAM budget.
  5. Biggest reason it might fail. SpecExec's ~20-token acceptance relied on very large draft trees and a strong draft model; with a small MLX draft and greedy-ish chat workloads acceptance may be 4–6, and tree verification batch width on Metal at batch≈64–256 may be compute-bound (unified-memory bandwidth shared with the concurrent sweep — the one contention expH did not measure: RAM-vs-RAM, not SSD-vs-RAM).
  6. Smallest falsifying experiment. New micro (expL_specsweep). Measure (a) acceptance length of Llama-3.2-1B drafting for Llama-3.1-70B-Q4 (both runnable; target via llama.cpp Metal, RAM-resident for the measurement) with tree width 16–64; (b) Metal batch-verification throughput at batch 64 concurrent with a 13 GB/s read stream. Kill: accepted length <5, or verification throughput under concurrent streaming drops the compound I/O+compute ceiling below 2 tok/s at 1.4× over-RAM.

G19 — RouterBias: cache-conditional expert routing on stock MoE, zero-shot#

  1. What existing systems do. Cache-conditional experts (trained) bias routing toward resident experts; MoBiLE substitutes smaller experts on miss (trained); llama.cpp serves expert misses by blocking page faults. All model-side co-design in print requires training the router.
  2. What they do not do. Nobody has measured how much zero-shot router-logit biasing (a runtime knob: add λ·resident_mask to gate logits, renormalize) buys in hit rate on a stock MoE, at what quality cost — despite the router's output being a distribution over near-substitutes by construction.
  3. Why it might matter. For MoE on Macs (the nearest-term deployment target, cf. G09), even a 10–15 point hit-rate lift at negligible quality cost multiplies directly into tokens/s; it needs ~5 lines in llama.cpp/mlx-lm — the highest value-per-effort entry in this document if it works.
  4. Plausible reason it could work. Top-2 gate margins are often small (experts are partially redundant by load-balancing training pressure); swapping expert #2 for resident expert #3 when gates are near-tied is a bounded perturbation, measurable per token; the bias λ can be gated by the gate margin itself (only re-route near-ties).
  5. Biggest reason it might fail. The trained systems exist because zero-shot biasing failed to preserve quality: gate margins may be decisive exactly on the tokens that matter, and systematically starving rarely-resident experts could compound into distribution drift over long generations (a failure aggregate PPL won't show — flips will).
  6. Smallest falsifying experiment. expF (MoE variant). On Qwen3-30B-A3B in mlx-lm, apply margin-gated resident bias at λ sweeping 0→strong with simulated 50%-resident caches; measure hit-rate lift vs flips/KL/GSM8K. Kill: every λ that lifts hit rate ≥10 points costs >2% flips or >1 pt GSM8K.

G20 — AmxSidecar: CPU (AMX/Accelerate) as a free co-executor on unified memory#

  1. What existing systems do. PowerInfer splits hot/cold across GPU/CPU — meaningful only with separate memories (its macOS port is CPU-only with "limited" gains); llama.cpp offloads whole layers to CPU as a capacity fallback; no runtime treats Apple's AMX/Accelerate as a concurrent engine for the irregular fraction of work (sparse gathers, residual GEMVs, predictor probes) while the GPU runs the dense base — on the same bytes, no copies (SOTA §4.3: "GPU/CPU split is meaningless on unified memory" — as a capacity split; as a kernel-affinity split it is unexplored).
  2. What they do not do. Nobody measures whether M-series CPU matvec bandwidth (~100+ GB/s via AMX) survives concurrent GPU saturation, or schedules irregular-access kernels (where GPUs are weakest: G03's ΔY, G06's gathers) on the CPU specifically to dodge the Metal gather penalty.
  3. Why it might matter. Several gaps (G03, G06, G15) may die on Metal gather-kernel cost alone (item 6 kills above); a CPU sidecar is the escape hatch — the P-cores are idle during decode, and unified memory means the split costs zero copies.
  4. Plausible reason it could work. expH already showed the memory controller tolerates concurrent SSD+GPU load (<5%); Accelerate's sgemv on P-cores is latency-optimized and cache-friendly for irregular rows; the sidecar's work share (5–15% of bytes) matches the CPU:GPU bandwidth ratio.
  5. Biggest reason it might fail. RAM-bandwidth contention (unmeasured in expH: that tested SSD-vs-GPU, not CPU-vs-GPU on RAM): if concurrent CPU streaming steals

    15% of GPU bandwidth, batch-1 decode slows more than the sidecar contributes; and CPU↔GPU synchronization per layer (~µs via shared events) × 2 ops × 80 layers may dominate at 10 ms/token.

  6. Smallest falsifying experiment. New micro (expM_contention). Concurrent: GPU dense matvec loop (MLX) + CPU Accelerate GEMV loop on separate buffers; measure both throughputs vs isolation, plus shared-event sync latency. Kill: CPU sustains <50 GB/s effective while the GPU is saturated, or GPU loses >15%, or per-layer sync >50 µs.

G. Attention-side working set#

G21 — KVBudget: joint weight+KV working-set accounting, SparQ-class KV fetch on Metal#

  1. What existing systems do. KV quantization on Metal is solved-enough (llama.cpp --cache-type-k/v, mlx-lm --kv-bits, KIVI/KVQuant algorithms); SparQ fetches only the KV rows whose keys matter (8× attention-transfer cut, CUDA); SolidAttention pages KV to SSD with medium-chosen units (FAST 2026, Linux). Every weight-side system in this document ignores KV growth.
  2. What they do not do. No system co-budgets weights and KV as one working set on unified memory (SOTA §2.7: "any weight working-set argument must co-model KV"), and SparQ-style selective KV fetch has no Metal implementation — at 128k context the KV cache of an 8B model is multiple GB and grows, silently eating the residency that G01/G07/G09 assume is theirs.
  3. Why it might matter. If KV bytes/token dominate past ~32k context, every weight-side gap's benefit saturates: the project would be optimizing the minority term. Conversely, SparQ + 2-bit KV on Metal could free gigabytes of residency for weight caches — a direct multiplier on every other gap.
  4. Plausible reason it could work. All the algorithms are training-free and published; the fetch pattern (top-r key rows) is a gather over a RAM-resident cache — no SSD in the loop, so the 35:1 ratio does not apply; mlx has quantized-KV plumbing to extend.
  5. Biggest reason it might fail. As standalone research it may be moot: at the target interactive contexts (4–32k), KV even at 4-bit may be <10% of per-token bytes for the models we care about — engineering worth doing, but not a gap that changes any conclusion; and the Metal gather cost question (same as G06 item 5b) applies to key-row fetching too.
  6. Smallest falsifying experiment. New micro (expN_kvshare, one afternoon). Instrument mlx-lm on 8B/30B models: measure per-token bytes read from weights vs KV at contexts {4k, 32k, 128k} with KV at {16, 4, 2} bits. Kill (as a priority gap): KV share of per-token bytes <10% at ≤32k context for 8B–30B models — then defer to Phase 10 engineering and strike it from the candidate list.

H. Contrarian gaps — cheap ways the premise dies#

G22 — WarmEnough: maybe the kernel page cache already solves residency once selectivity exists#

  1. What existing systems do. llama.cpp mmap + kernel LRU beat its own direct-I/O experiment because the UBC retained the hot expert working set (#18758); the page cache is shared, persistent across runs, and warm-start economics on macOS are excellent (SOTA notes §5). All of Group C (G08–G11) presumes custom residency management beats the kernel.
  2. What they do not do. Nobody has published steady-state warm paging overhead for a selective workload (MoE/sparse) on macOS — every dramatic number (0.025 tok/s, 2.23×) is a cold-start or dense-cyclic measurement. The custom-policy case rests on worst cases that warm selective workloads may never hit.
  3. Why it might matter (i.e., why this must be tested). If warm steady-state fault overhead is <10% of token time, Group C is ~40% of this document's engineering for ~nothing — the correct architecture would be "compile a good layout (G04), mmap it, let the kernel work, spend all effort on representation (Group A) and speculation (Group F)."
  4. Plausible reason it could be true. LRU's pathology is cyclic dense scans; selective workloads (experts, gated blocks) have genuine recency structure — the regime where LRU is fine; 16 KB faults with 64-page clustered readahead may amortize adequately; and the UBC uses all free RAM opportunistically, which no self-managed wired budget can (it must leave headroom for the OS).
  5. Biggest reason it might be false. Fault-path costs are per-16 KB-page Mach overhead (the #18758 result: explicit 1 MiB slice reads beat faulting by 2.23× cold) and the UBC cannot exploit model-aware prefetch (router lookahead) — warm hit rates may be fine while miss latency still gates P99 token time.
  6. Smallest falsifying experiment (of the contrarian claim — and, if it survives, of G08–G11). New micro (expO_warmsteady, needs only llama.cpp + dtrace). Run Qwen3-30B-A3B GGUF mmap'd with RAM constrained (wired ballast) to 50/75/100% of model size; after 2k warm-up tokens, measure faults/token, fault-time share, and tok/s vs the fully-resident baseline. Decision: warm fault-time share <10% of token time at 75% residency ⇒ G08–G11 are demoted to engineering polish;

    25% ⇒ Group C stands.

G23 — ConfidentlyWrong: maybe uncertainty signals cannot find the tokens that matter#

  1. What existing systems do. The entire escalation family (BiLD, CALM, cascades, Kangaroo, and this document's G02/G17) assumes cheap-model confidence correlates with cheap-vs-full disagreement. Published evidence is indirect: acceptance rates and margins work on average; the joint distribution (margin × agreement) has never been published (SOTA §6.7).
  2. What they do not do. No paper reports P(disagree | margin) for quantized bases, nor what fraction of disagreements occur at high margin — the confidently-wrong mass. Thinking-Machines' knife-edge result and the 22–26-nat KLD outliers hint the tail is adversarial, and "Accuracy is Not All You Need" shows flips concentrate on load-bearing answers.
  3. Why it must be tested. If ≥30% of disagreements sit above any usable margin threshold, gating (G02) silently ships exactly the errors users notice, and calibrated-risk variants must set thresholds so conservative that escalation approaches 100% — the charter §17 "prediction overhead exceeds savings" failure, in its sharpest form.
  4. Plausible reason the contrarian claim could be true. Quantization error is not random noise w.r.t. the model's confidence: it is systematic (per-block rounding biases), so it can shift logits coherently — producing high-confidence flips rather than low-margin dithering; MoE-router-protection evidence (rankings change with which weights get precision) shows exactly such systematic sensitivity.
  5. Biggest reason it might be false. The early-exit literature works in practice across many models and tasks; QuantSpec's >90% acceptance with naive verification implies gross miscalibration is not the norm at 4-bit; and learned error features (ρ = 0.82) can catch systematic error that raw margin misses.
  6. Smallest falsifying experiment. expG (the same run as G02 — one experiment, two hypotheses). From the per-token logs, compute P(disagree | margin decile) and the fraction of disagreements above the 50th-percentile margin. Decision: disagreement-detection AUROC <0.65 for margin+entropy+learned features at a 3-bit base ⇒ G02 and G17's gated forms die (block-level rejection sampling in G17/G18, which never trusts the base, survives); AUROC >0.85 ⇒ the escalation family is the project's center.

G24 — BasinCollapse: maybe the low-bit base is not a usable draft at all#

  1. What existing systems do. Every progressive/escalation gap here (G01–G03, G11, G15, G17) presumes a 2–3-bit-effective resident base whose behavior is "the same model, slightly noisy." ParetoQ reports a sharp representational transition between 2 and 3 bits (≤2-bit leaves the pretrained basin); EfficientQAT needs QAT to make 2-bit behave; llama.cpp blind tests found IQ2/IQ1 plainly distinguishable; BiLLM's 1.08-bpw PPL of 8.41 is "clearly degraded" (SOTA §2.6).
  2. What they do not do. Published numbers are PPL/benchmarks per bitrate; nobody reports the quantities the paging premise needs: agreement with the full model and smoothness of recovery per residual byte at 2–3-bit PTQ bases — the actual gates for G01/G17.
  3. Why it must be tested. RAM arithmetic forces the question: a 70B model on 48 GB (minus KV, OS, apparatus) needs a ≤3-bit base; a 100–250 GB stretch-target checkpoint (charter §15) needs ~2-bit-effective. If agreement craters there, the honest conclusion is that this project's ceiling is ~1.5× oversubscription of 4-bit models — a finding worth publishing and pivoting on (charter §17).
  4. Plausible reason the contrarian claim could be true. The ParetoQ transition is corroborated independently (EfficientQAT's QAT requirement, the IQ blind tests); error compounding is near-exponential in depth (QEP), and 70–80-layer models give 2-bit noise dozens of chances to leave the basin; PTQ at 2-bit has no mechanism to protect routing/decision structure (the MoE-router-protection lesson).
  5. Biggest reason it might be false. The transition literature measures standalone quality, not draft quality: a base can be individually mediocre yet agree with its own refined version on easy tokens (which are the majority), and G04's R-D allocation + residual-aware calibration (DWQ loop) may buy back ~0.5 effective bits exactly where the basin risk concentrates.
  6. Smallest falsifying experiment. expD + expG jointly (same models, one run matrix). MLX-quantize an 8B model at {2, 2.5(mixed), 3, 4} bits; measure (a) same-top-token vs the 8-bit reference per domain, (b) KL-vs-residual-bytes recovery smoothness. Kill (for the progressive family at stretch scale): 3-bit agreement <75% on chat/code or recovery is non-smooth (KL plateaus until >50% of residual bytes applied). 2-bit failing alone only caps oversubscription at ~1.6× — record the measured ceiling either way.

Clusters and dependencies#

Shared infrastructure clusters (build once, serve many):

  • Trace/importance pipeline (per-token block-importance and mask logging on MLX models): required by G01, G06, G07, G08, G09, G12, G13, G24 — this is the expA/expB/expC instrumentation and must be built first; it is the single largest shared dependency.
  • Quantize/refine ladder (MLX models at 2/2.5/3/4/8 bits + per-block residual application): required by G01, G02, G03, G04, G11, G15, G17, G24.
  • Custom Metal microbench rig (matvec kernel harness with bytes/throughput accounting): required by G03, G05, G06 (gather), G16, G20, G21 — all expE variants.
  • Cache/paging simulator (trace replay through LRU/MRU/ARC/OPT with expH cost model): required by G08, G09, G12, G22 — pure Python, cheap, decouples policy questions from runtime engineering.
  • llama.cpp/mlx-lm instrumentation (fault/read/tok-s logging on real runs): required by G09, G18, G19, G21, G22.

Falsification fan-in — experiments that arbitrate several gaps at once, in order:

  1. expG (joint margin × agreement, multi-bitrate, multi-domain) — one run matrix decides G02 (gating viability), G17 (acceptance lengths), G23 (confidently-wrong mass), and half of G24 (agreement at 2–3 bits). Highest information per compute-hour in the project; no custom kernels needed (MLX quantized casts + logging). Run first.
  2. expA/expB/expC trace campaign (SwiGLU importance + stability + domain locality) — one logging campaign over an 8B model across six domains feeds G01 (residual-set stability), G06 (mask sparsity/stability), G07 (cold-block demand), G12 (prefetch predictability), G13 (domain structure), and supplies the traces for the G08/G09/G22 cache simulator. Run second, concurrently with 3 where possible.
  3. expD/expE ladder (recovery curves + Metal decode/gather costs) — decides G01 (bytes-to-recover), G04 (R-D gain), G05 (format tax), G15 (spectral vs precision split), G16 (bandwidth-for-compute trade), the other half of G24, and gates G03's kernel. This is where custom Metal work concentrates; do the pure-MLX parts (expD) before writing any kernel.

Standalone/cheap probes that can run anytime (fit in idle time, each ≤1 day): expN_kvshare (G21 — may strike a gap from the list), expO_warmsteady (G22 — may strike four), expM_contention (G20), expJ_purgeable (G10), G19's router-bias sweep.

Dependency edges that order Phase 4 ranking: G22 gates the priority (not the design) of G08–G11; G23/G24 gate the entire escalation/progressive families (G01–G03, G17) — which is why expG runs first; G05 gates whether G01/G04 use bitplanes or must fall back to affine-delta residuals; G20 is the contingency for G03/G06/G15 gather-kernel failures; G18 is the baseline that every representation-heavy gap (G01, G15, G17) must beat at equal oversubscription before being declared interesting. Contrarian outcomes are publishable results per charter §16–17 either way: if G22–G24 all "win," the honest deliverable is a measured map of why the regime does not exist on this hardware — with G18 + G09 + G04 as the salvage architecture (speculative sweeps over a well-laid-out, kernel-cached, R-D-compiled model), which is itself unoccupied territory.