Hypothesis — expG_decision_stability#

The Gate-Zero experiment (candidate_ranking.md C4): measures the joint distribution of (cheap-pass top-1 margin × agreement with the full-precision model) that the literature does not report, and which decides whether margin-gated escalation (G02 MarginGate, and with it candidate C1) is viable.

Hypothesis
  For a small pretrained LLM, the top-1 logit margin of a low-bit quantized
  variant is a strong predictor of whether its greedy token decision matches
  the full-precision model: disagreements concentrate at small margins.
  Concretely, margin achieves AUROC ≥ 0.8 as a disagreement detector at
  3–4 bits, so that escalating only the low-margin fraction of tokens
  (expected 10–30%) recovers ≥98–99% greedy agreement.

Falsification criterion
  If AUROC < 0.65 at every bit-width, or if reaching 99% agreement requires
  escalating > 50% of tokens at 4-bit (i.e., disagreements are NOT
  concentrated at low margins), then decision-uncertainty is not a usable
  gating signal: G02/G23 die and candidate C1 must be redesigned around
  amortized verification (C2) instead.

Method
  Model: Qwen3-1.7B (bf16 reference; representative modern architecture,
  runs fully in memory). Variants: MLX affine quantization at 2, 3, 4, 8
  bits (group size 64), quantizing all linear layers.
  For each of 48 prompts across 6 domains (code, math, chat, FR, EN,
  reasoning; benchmarks/datasets/eval_prompts.json): generate 128 tokens
  greedily with the bf16 reference, then teacher-force every variant over
  the same token sequence and record, per position: variant top-1 margin
  (logit gap between its top-2 candidates), variant argmax, reference
  argmax, agreement bit, and KL(ref||variant).
  Analysis: per bit-width and per domain — agreement rate; margin
  distributions conditioned on agree/disagree; AUROC of margin as a
  disagreement detector; escalation curve (fraction of tokens with margin
  below τ vs residual disagreement rate after escalating those tokens);
  the operating point (escalation fraction) that reaches 99% agreement.
  Seed fixed (greedy decode; deterministic). Hardware manifest embedded.

Baseline
  The 8-bit variant (known near-lossless: expected ≥99% agreement without
  any gating) anchors the top of the scale; published aggregate agreement
  rates for 4-bit (~90–91%, llama.cpp 'same top p' data) sanity-check our
  pipeline. No straw men: the comparison is against unconditional
  escalation (loading full weights for all tokens).

Result
  CONFIRMED (kill criteria not triggered). AUROC: 0.852 (3-bit), 0.898
  (4-bit), 0.975 (8-bit); esc@99%: 60.2% / 36.6% / 5.1%. Naive affine
  2-bit collapsed entirely (3% agreement, AUROC 0.605) — the ParetoQ
  cliff, measured. Full numbers: results/expG_decision_stability/
  20260812T042744Z/ and analysis.md.

Interpretation
  Margin is a strong, free, domain-uniform gating signal → G02 promoted.
  4-bit escalation need (36.6%) exceeds the hoped 10–30% band, so C1's
  economics now hinge on bytes-per-escalation (expD). Resident base must
  be ≥3-bit affine or codebook 2-bit (G05).

Next experiment
  expD — bytes of residual needed per escalated token; then expG rerun at
  3B–8B scale.

Analysis — expG_decision_stability#

Run: results/expG_decision_stability/20260812T042744Z/ · code committed before run. Model: Qwen3-1.7B (mlx-community bf16 reference), MLX affine quantization (group 64, all divisible Linear layers) at 2/3/4/8 bits. 48 greedy reference trajectories (8 per domain × 6 domains), 128 tokens each → 6 144 scored positions per bit-width. Hardware: M5 Max / 48 GB / macOS 27 (manifest embedded).

Hypothesis
  Margin of the cheap pass predicts disagreement with the full model:
  AUROC ≥ 0.8 at 3–4 bits; escalating 10–30% of tokens recovers ≥98–99%.

Falsification criterion
  AUROC < 0.65 at every bit-width, or >50% escalation needed at 4-bit.
  → NOT triggered.

Result (n = 6 144 positions/bit-width)
  bits  agree    mean KL   AUROC   esc@99%   median margin agree/disagree
   2    0.0296   11.89     0.605   99.0%     1.50 / 1.00
   3    0.7310    0.78     0.852   60.2%     5.25 / 1.12
   4    0.8740    0.199    0.898   36.6%     6.50 / 0.75
   8    0.9803    0.0039   0.975    5.1%     6.25 / 0.25
  Per-domain AUROC is uniform (4-bit: 0.880–0.921; math best, chat worst);
  no domain inverts the signal. Sanity: 4-bit agreement (87.4%) sits near
  published ~90–91% 'same top token' rates for 4-bit k-quants on larger
  models — plausible for a 1.7B model with plain affine quantization.

Interpretation
  1. PROMOTE G02 (MarginGate): AUROC 0.852–0.975 clears the 0.8 promote
     threshold at 3, 4, and 8 bits. Disagreements concentrate sharply at
     small margins (median disagree-margin ≤ 1.1 vs agree-margin ≥ 5.25).
     The margin signal is nearly free (it falls out of the cheap forward
     pass), and it generalizes across all six domains including reasoning.
  2. G23 (ConfidentlyWrong, contrarian) is weakened but not fully dead:
     at 4-bit, reaching 99% agreement still requires escalating 36.6% of
     tokens — above the hypothesized 10–30% band, though far below the 50%
     kill line. The gate works; whether it is *cheap enough* now depends on
     the per-escalation byte cost (expD's question, not this experiment's).
  3. NAIVE 2-BIT AFFINE IS DEAD as a resident base: 3.0% agreement,
     KL 11.9, AUROC 0.605 — the model leaves the pretrained basin entirely,
     confirming the ParetoQ 2–3-bit cliff on modern SwiGLU models with
     amplitude that perplexity tables understate. Candidate C1's resident
     base must therefore be ≥3-bit affine, or a codebook/trellis 2-bit
     format (which requires the G05 Metal kernel work). This partially
     answers G24 (BasinCollapse) for the affine family.
  4. The 8-bit row shows the ceiling: 98% agreement, 5.1% escalation for
     99% — an 8-bit-base system would stream almost nothing but also saves
     only 2× memory. The interesting operating region is a 4-bit-class base
     (0.5 byte/param resident) escalating ~1/3 of tokens, or a 3-bit base
     escalating more; which wins depends on residual fetch granularity.
  5. Note the escalation curves in results.json give the full frontier
     (escalated fraction vs residual disagreement) for policy design.

Next experiment
  expD (progressive reconstruction): measure how many *bytes* of residual
  (4→8-bit planes, per-layer or per-block) an escalated token actually
  needs to flip its decision to the reference — the product
  (escalation rate × bytes/escalation) against the expH budget
  (~650 MB/token) decides candidate C1's viability. Secondary: rerun expG
  at 3B–8B scale to check the expected agreement improvement with size.

Addendum — scale check at 8B (2026-08-12)#

Run: results/expG_decision_stability/20260812T051129Z/ — same protocol, Qwen3-8B bf16 reference, bits {3,4,8}.

bits   agree(1.7B → 8B)      AUROC(1.7B → 8B)    esc@99%(1.7B → 8B)
 3     0.731 → 0.853         0.852 → 0.883       60.2% → 41.0%
 4     0.874 → 0.921         0.898 → 0.920       36.6% → 22.6%
 8     0.980 → 0.987         0.975 → 0.981        5.1% →  1.6%

Every metric improves with scale: bigger models are more robust to quantization (consistent with published trends), margins are MORE informative, and the escalation budget SHRINKS — at 8B a 4-bit base needs q8-level correction on only ~23% of tokens for 99% agreement. The margin-gated architecture's economics improve exactly where the project needs it (the 30B+ regime where checkpoints stop fitting in RAM).

expG_decision_stability#

Decision stability: how many token decisions are stable before full precision is available

Status: scaffolded 2026-08-11, not yet run.