localvm-research

Research / research/novelty_check.md

novelty check

draft

research/novelty_check · created Tue Aug 11 2026 20:00:00 GMT-0400 (heure avancée de l’Est) · Simon-Pierre Boucher

Phase 11 — Novelty verification of candidate_01 (margin-gated deferred refinement with a streamed verifier)#

Method. Per charter §19, the working assumption was NOT NOVEL until evidence suggests otherwise. The search was adversarial: for each component of the architecture, and for the composition, I searched for conceptual equivalents (not just identical terminology) across arXiv (with emphasis on 2025–2026, i.e., after the Phase 1 sweep of 2026-08-11), ACL/EMNLP, MLSys/OSDI/ASPLOS-adjacent venues, OpenReview, and the llama.cpp / MLX / mlx-lm GitHub ecosystems. All sources are listed in §5 with access date 2026-08-12. The two closest prior systems (Lever, CATS — both May 2026, both absent from the Phase 1 notes) were read from their full PDFs, not abstracts.

1. The architecture under check (précis)#

A single-machine runtime for unified-memory consumer hardware (Apple Silicon, MLX/Metal):

  • A q4 quantization of the model is resident and decodes greedily at full speed, recording the top-1 logit margin of every token decision; low-margin decisions are deferred (margin mode) rather than escalated immediately.
  • Every W tokens (W=32 measured), a q8 variant of the same model — which does not fit in memory beside the base — is streamed layer-by-layer from NVMe (materialize → compute → re-lazify; one strictly sequential pass over the checkpoint per sweep at the SSD device ceiling, ~11.6 GB/s) and teacher-forces the window in one pass against a persistent incremental q8 KV cache (O(window), not O(context)).
  • On the first flip (checked at deferred positions only in margin mode; at all positions in verify-all mode) the runtime rolls back via KV-cache trim, accepts the q8 token, and resumes. Acceptance is deliberately approximate in margin mode (unchecked high-margin positions are trusted); verify-all is greedy-exact up to Metal numerics.
  • Objective and headline metric: bytes streamed per generated token, decoupled from checkpoint size (9.4× under the q8 checkpoint at W=32 on Qwen3-32B); the margin threshold τ is a runtime knob trading bytes for quality (47%/83%/~100% of the q4→q8 quality gap at 150/211/237 MB/token on 1.7B; +0.28 nats over pure-q4 at 32B where q8 cannot run at all).

2. Per-component prior-art tables#

Verdict scale: ANTICIPATED (the component exists in print, differences are cosmetic) · PARTIALLY ANTICIPATED (the idea exists but in a materially different setting or without a load-bearing property) · NOT FOUND (no conceptual equivalent located).

2.1 Precision-tiered self-speculation (q4 draft / q8 verify, shared weights)#

Closest prior art What it does What it does NOT do Verdict
QSpec (EMNLP 2025, arXiv 2410.11305) Same W4 weights; W4A4 draft, W4A16 verify; near-zero switching cost; lossless; 1.64× Both operand sets resident; verifies every token; objective is arithmetic speed, not capacity or bytes/token ANTICIPATED (core mechanism)
QuantSpec (Apple+Berkeley, ICML 2025) Self-speculation with 4-bit weights + 4-bit hierarchical KV draft, full-precision verify; >90% acceptance; edge-motivated Full-precision verifier fully resident and fully read; lossless only; latency objective ANTICIPATED (core mechanism)
ML-SpecQD (arXiv 2503.13565) MXFP4 direct cast of the target as turnkey draft (no training), 16-bit verify, >2× Same: verifier resident; verify-all; latency framing ANTICIPATED (core mechanism)
Sub-variant (a) verifier kept OUT of memory / streamed Lever and CATS stream their verifier from flash (see §2.3/2.4) — but their drafts are a separate small model (Lever) or a distilled shallow sub-network (CATS), never a precision cast of the same weights No published system pairs a low-bit resident cast with a streamed higher-precision variant of the same checkpoint NOT FOUND (within the precision-tier family)
Sub-variant (b) verify only uncertainty-flagged positions Exists outside this family (U-HLM, UGSD, SPRINTER — §2.2) No precision-tiered system gates which positions are verified NOT FOUND (within the precision-tier family)
Sub-variant (c) deliberately approximate acceptance Rich lossy-verification literature (§2.2): rank/top-R acceptance, judge heads, relaxed acceptance in TensorRT-LLM/SGLang; BiLD thresholds; taxonomy in arXiv 2607.26627 Not combined with precision tiers or a storage tier ANTICIPATED (in other settings)
Sub-variant (d) bytes-per-token objective SpecExec's economics imply bytes/token ÷ acceptance length; CATS names flash↔DRAM traffic as the bottleneck Nobody states or reports bytes-streamed-per-token vs checkpoint size as the optimization target and headline result NOT FOUND

2.2 Confidence/margin-gated selective verification or escalation#

Closest prior art What it does What it does NOT do Verdict
U-HLM (arXiv 2412.12687, IEEE 2025) — the sharpest hit On-device SLM drafts; tokens whose on-device uncertainty ≤ threshold skip uplink transmission and LLM verification entirely (74.8% of uplinks skipped); rejection-behavior predicted from draft-side uncertainty Uncertainty via temperature-perturbation consistency, not logit margin; expensive tier is a remote different model, cost is radio bytes, not weight-streaming I/O; no rollback of committed tokens; no storage tier PARTIALLY ANTICIPATED
UGSD (arXiv 2603.11397, Mar 2026) Edge draft; entropy-gated escalation of only high-uncertainty token blocks to a cloud verifier; top-R rank acceptance (lossy); suffix discarded on correction (rollback); adaptive block length; only 18.2% of tokens escalated Cloud tier (communication cost), different model as verifier, domain-specific (speech emotion captioning); no weight streaming; no bytes/token framing PARTIALLY ANTICIPATED
SPRINTER (arXiv 2502.04557) Tiny learned verifier predicts whether the target would accept; target LLM invoked only when a token is deemed unacceptable (approximate, not exact) Verifier is a learned classifier, not the draft's own margin; both models resident; no storage tier PARTIALLY ANTICIPATED
BiLD (NeurIPS 2023) Small model decodes; confidence-threshold fallback to large model + distance-threshold rollback of recent tokens Two separate resident models; no storage tier; no exactness mode; latency objective ANTICIPATED (gating + rollback pattern)
CALM (NeurIPS 2022) / cascades (FrugalGPT, RouteLLM) Margin/confidence-calibrated early exit with distribution-free risk control; request-level routing to expensive models Depth axis / request granularity; nothing about weight loading ANTICIPATED (signal + calibration recipe)
Token-level margin gating of a weight-loading (SSD-streaming) decision — Searched: "uncertainty triggered precision escalation", "confidence-gated weight loading", "margin-gated verification bytes", dynamic-precision line (PMPD, HOBBIT, DP-LLM — all feed-forward, no verification), lossy-verification taxonomy (2607.26627 — no memory-constrained verifier) NOT FOUND

2.3 Layer-by-layer weight streaming from storage during a forward pass (unified memory)#

Closest prior art What it does What it does NOT do Verdict
AirLLM (lyogavin/airllm) / ZeRO-Inference Layer-by-layer load→compute→free; capacity decoupling only Streams the whole model for every token; no draft; no selectivity ANTICIPATED (streaming primitive)
mlx-flash (GitHub, matt-k-wong) Exactly materialize→compute→release per layer in MLX on Apple Silicon (StreamingProxy wrapping, predictive I/O scheduler prefetching layer N+1 during layer N); "30B on 16 GB, 70B+ on 32 GB+" Streams on every token; no draft/verify role for the streamed model; no periodicity, no quality tiering ANTICIPATED (the MLX materialize→compute→re-lazify scheduling itself is not ours)
mlx-moe, vMLX, MLX issue #2878, MLX discussion #615 SSD expert streaming for MoE in MLX; open feature request for out-of-core streaming (Dec 2025); maintainer position that weights must be materialized/wired for GPU use Nothing combines a streamed model with a resident draft; #2878 does not mention speculation supports NOT FOUND for the verifier-role variant
Lever (arXiv 2605.16786, May 2026) DRAM-resident draft, flash-resident target streamed at each verification invocation on smartphones; "the SSM can remain resident in DRAM … the LLM can be stored in flash and invoked only intermittently"; I/O is 78–93% of verification latency (their Table 1) Separate small draft model (not same weights); CPU/NPU + UFS, not unified-memory Metal; verification invoked every speculative cycle (no margin-gated skipping); lossless greedy acceptance; latency objective ANTICIPATED (streamed-verifier-while-resident-draft as a category)
CATS (arXiv 2605.11186, May 2026) Self-speculative on edge: shallow sub-network drafts under the DRAM budget; intermediate layers "streamed from flash once per cycle" for shallow verification; remaining layers "offloaded from the flash chunk by chunk" for target verify; peak memory = target model alone; 5.08× Draft axis is depth (+ Reduced-KL distillation training), not a training-free precision cast; verification every cycle, not margin-deferred; no bytes/token metric; GPU/edge-Linux, not MLX/Metal ANTICIPATED (periodic flash-streamed verifier, window-amortized)
llama.cpp discussion #10466 (Nov 2024) Public proposal: fully-resident speculator + target layers ping-ponged into VRAM from CPU RAM for verification Never implemented; CPU-RAM/PCIe tier, not SSD; verify-all, exact corroborates ANTICIPATED (idea circulating)

2.4 Windowed/batched deferred verification with rollback, where verifier cost is I/O#

Closest prior art What it does What it does NOT do Verdict
SpecExec (NeurIPS 2024) Huge draft trees verified by a RAM-offloaded 70B in one pass; ~20 accepted tokens per full-model weight sweep; 10.6–18.7× over sequential offloaded decoding — the amortization economics of one weight sweep per window Separate 7B draft; CUDA/PCIe; exact sampling; never states bytes/token as metric ANTICIPATED (window-amortized I/O-bound verification)
Lever / CATS (above) Explicitly amortize flash reads of the verifier over the accepted window ("the cost of loading and executing the LLM is amortized across several output tokens" — Lever §1) See §2.3 ANTICIPATED
SpecOffload (arXiv 2505.10259) / SpecMoEOff (arXiv 2508.21706) / SpecInfer (ASPLOS 2024) / TriForce (COLM 2024) Speculation embedded into offloading pipelines (weights or KV) to hide transfer latency; TriForce: resident sparse-KV draft of the same weights, offloaded full KV verify Throughput/batch regimes; PCIe tiers; verify-all; exact ANTICIPATED
Rollback via KV-cache trim on first flip Standard speculative-decoding machinery; BiLD rollback policy; UGSD suffix discard — ANTICIPATED (not claimable at all)
Window-amortized verification whose cost is I/O and whose frequency/coverage is margin-gated — Lever/CATS/SpecExec verify every window; U-HLM/UGSD gate but have no weight I/O NOT FOUND

2.5 The composition as a whole#

Search phrasings tried (plus synonyms): "resident draft streamed verifier", "quantized draft full model verification SSD", "deferred verification language model rollback", "amortized weight streaming verification", "two-tier quantization runtime offload verify", "speculative decoding out-of-core verifier", "verifier offloading speculative decoding", "uncertainty triggered precision escalation", "self-speculative decoding offloading edge", "flash-backed target model speculative decoding", plus GitHub sweeps of llama.cpp / ik_llama.cpp / mlx / mlx-lm issues and community MLX streaming projects.

Candidate Overlap with our composition Decisive differences
CATS (May 2026) — nearest overall Resident cheap variant of the model drafts; verifier layers streamed from flash once per window; rollback; single memory-limited device; peak memory = one model Cheap variant is a depth truncation requiring distillation training, not a training-free precision cast; every cycle verified (no margin deferral; no τ knob; no approximate mode); metric is wall-clock speedup, not bytes/token; not unified-memory/Metal
Lever (May 2026) DRAM-resident draft + flash-streamed target verifier, invocation cost amortized over the accepted window; I/O identified as the dominant verification cost Separate draft model (no shared weights / no precision pair); verification never skipped by draft confidence; lossless; smartphone NPU/UFS
U-HLM + UGSD (Dec 2024 / Mar 2026) Draft-side uncertainty gates whether the expensive tier is invoked at all; UGSD adds block granularity, lossy top-R acceptance and suffix rollback Expensive tier is a remote, different model; the gated cost is communication, not weight-streaming bytes; nothing resides on or streams from storage
QSpec / QuantSpec / ML-SpecQD The q4-drafts / q8-verifies-with-shared-weights pair, proven acceptance >90% Verifier resident; verify-all; arithmetic-latency objective

Composition verdict: NOT FOUND as assembled — but only by the conjunction, and the margin is thin. Every load-bearing part exists separately, three of them in systems published in 2025–2026 that the Phase 1 sweep missed (Lever, CATS, UGSD). Prior-art density around this exact point is increasing quarter over quarter.

3. Honest bottom line — what survives, what must be attributed#

Must be attributed (do not claim):

  1. Precision-tiered draft/verify on shared weights — QSpec, QuantSpec, ML-SpecQD own it.
  2. Resident draft + storage-streamed verifier, amortized over a token window — Lever and CATS own the category (with SpecExec owning the amortization economics and llama.cpp #10466 as a public 2024 proposal). Our verify-all mode is, at the category level, "Lever/CATS with a precision-cast draft on Apple Silicon."
  3. Uncertainty-gated skipping of verification / escalation to an expensive tier — U-HLM, UGSD, SPRINTER, with the older lineage BiLD / CALM / cascades.
  4. Deliberately lossy acceptance — the relaxed/lossy-verification literature (top-R rank acceptance in UGSD; judge/relaxed acceptance lines; taxonomy in arXiv 2607.26627).
  5. Rollback via KV trim; teacher-forcing a window in one pass — standard speculative decoding machinery.
  6. Layer-by-layer materialize→compute→release streaming in MLX on Apple Silicon — mlx-flash (and AirLLM/ZeRO-Inference generically). We did not invent the scheduling pattern; we gave the streamed pass a verifier role.

Surviving as (narrowly) novel contributions:

  • N1 — Margin-gated deferral of verification bytes. Using the resident draft's top-1 logit margin to decide whether higher-precision weights are streamed from local storage at all (which positions a sweep must check, and hence how much I/O the quality repair costs). Every located gating system gates compute or communication to a different, remote or resident model; none gates the weight-streaming I/O of the same model's higher-precision representation. NOT FOUND after targeted search.
  • N2 — Bytes-streamed-per-token as the explicit objective, reported against checkpoint size (9.4× under at W=32), with the τ threshold exposed as a runtime bytes↔quality knob (47%→83%→~100% of the q4→q8 gap at 150→211→237 MB/token). Dynamic-precision systems (PMPD, HOBBIT, DP-LLM) have precision knobs without verification; lossy verification has quality knobs without a storage tier; no system prices quality in bytes-of-weights-streamed. NOT FOUND as a stated objective/metric.
  • N3 — The training-free same-checkpoint precision pair as the draft/verify split of an out-of-core runtime (q4 resident / q8 streamed, no distillation, no auxiliary model), making "quality level that does not fit in RAM" usable on unified memory — demonstrated where the alternative (resident q8) is physically impossible (32B on 48 GB), judged +0.28 nats over the only configuration that fits. The conjunction is unoccupied; each half is not.
  • N4 (engineering, weaker) — The unified-memory instantiation: strictly sequential sweep at SSD device ceiling, persistent incremental verify-KV-cache (O(window) sweeps) with trim-rollback, measured on Metal with the prefill/decode numerics ceiling identified. Claimable only as a measured system contribution on this platform, not as a concept.

Claims that are now weaker than when candidate_01 was designed: "first resident-draft/streamed-verifier runtime" (false — Lever, CATS), "first selective verification by draft uncertainty" (false — U-HLM, SPRINTER, UGSD), "first precision-pair self-speculation" (false — QSpec family), "first out-of-core amortized verification" (false — SpecExec, Lever, CATS).

What is new here is the conjunction, and two specific properties of it. Precision-pair self-speculation (QSpec, QuantSpec, ML-SpecQD), storage-streamed verifiers behind a resident draft (Lever, CATS), window-amortized verification of an offloaded target (SpecExec), and uncertainty-gated escalation (BiLD, CALM, U-HLM, UGSD) all exist separately. To our knowledge, no prior system (i) uses the draft's own top-1 logit margin to gate how many bytes of higher-precision weights are streamed from local NVMe to repair quality — making the margin threshold τ a runtime knob that prices output quality in bytes-per-token — and (ii) does so with a training-free precision pair of the same checkpoint (4-bit resident, 8-bit streamed) on unified-memory hardware, so that a quality level whose weights cannot reside in RAM remains usable, with bytes streamed per token decoupled from checkpoint size (measured 9.4× under the streamed checkpoint at W=32). We explicitly do not claim novelty for precision-tiered drafting, streamed verification, windowed rollback, or confidence-gated escalation taken individually; the attribution table in research/novelty_check.md maps each component to its prior art.

Avoid in all public text: "first", "novel runtime" unqualified, "no one has streamed a verifier", "quality becomes a runtime variable" without the qualifier via verification bytes (dynamic-precision papers already make quality a runtime variable without verification).

5. Sources#

All accessed 2026-08-12.