Hypothesis — expA_weight_concentration#

After expF killed layer-granularity escalation, this measures the next grain down: are FFN weight blocks (bundled neurons) unequally important per token? Feeds G01/G06/G07 and expE (partial GEMM); shares its trace with expB.

Hypothesis
  Per-token FFN intermediate-activation energy is concentrated: on a modern
  SwiGLU model, the top 20% of 64-neuron blocks capture ≥60% of the energy,
  and ≤50% of blocks suffice for 95% of the energy (per token, averaged
  across positions and domains). Per-neuron concentration is substantially
  stronger than block-64 concentration (bundling cost is real but moderate).

Falsification criterion
  If capturing 95% of per-token energy requires >70% of 64-neuron blocks
  (near-uniform importance), then block-level weight selection cannot cut
  bytes materially on this architecture and G06-style paging must rely on
  thresholded sparsity of individual neurons or die; C1 escalation-byte
  reduction via block selection (route c from expF) is dead too.

Method
  Qwen3-1.7B bf16. Wrap every layer's mlp.down_proj with a recorder; its
  input IS the SwiGLU intermediate activation h = silu(gate(x))·up(x),
  whose per-neuron magnitude determines the contribution of up/gate rows
  and down columns (the Gate-Up-Down bundle of the paging literature).
  Forward the 48 reference trajectories (same protocol as expG/D/F,
  greedy 128-token continuations, teacher-forced positions only).
  Record per predicted position: block energy (sum of h² over 64-neuron
  blocks; 96 blocks × 28 layers), stored float16 npz for expB reuse; plus
  streaming per-neuron stats (fraction of neurons for 90/95/99% energy).
  Report: energy captured by top {10,20,40,60}% blocks; blocks needed for
  {90,95,99}% energy; per-layer, per-domain aggregates; neuron-vs-block
  comparison.

Baseline
  Uniform importance (top k% of blocks capture exactly k% of energy) —
  the null hypothesis; and per-neuron granularity as the upper bound on
  achievable concentration.

Result
  KILL TRIGGERED at 64-neuron granularity: 95% energy needs 77% of blocks (>70% line). Neuron-level real (20% for 95%) but scattered — bundling destroys it. Depth gradient: late layers concentrated, early diffuse. Domain-independent.

Interpretation
  SwiGLU energy has no exploitable block structure; SSD fetch contract (≥256 KiB) and neuron-scale sparsity are mutually exclusive. Energy ≠ decision importance (cf. expF).

Next experiment
  expB on the same trace; then pivot decision.

Analysis — expA_weight_concentration#

Run: results/expA_weight_concentration/20260812T044829Z/ · Qwen3-1.7B bf16, 48 trajectories × 128 tokens (6 144 positions), SwiGLU intermediate energy per 16-neuron block (384 blocks × 28 layers), 64-neuron granularity derived.

Hypothesis / Falsification
  Hoped: top 20% of 64-neuron blocks ≥60% energy; ≤50% of blocks for 95%.
  Kill: >70% of 64-neuron blocks needed for 95% energy.

Result — KILL CRITERION TRIGGERED at 64-neuron granularity
  Granularity   top10%   top20%   needed for 90% / 95% / 99%
  neuron        0.88     0.94     0.13 / 0.20 / 0.38
  block-16      0.58     0.71     0.48 / 0.61 / 0.81
  block-64      0.44     0.57     0.65 / 0.77 / 0.92   ← 0.77 > 0.70 kill line
  Domain-independent (95% needs 61–62% of block-16 across all six domains).
  Strong depth gradient: late layers concentrate (L27 needs 11% of blocks
  for 95%; L20–26: 30–47%) while early/mid layers are diffuse (64–81%).

Interpretation
  1. NEGATIVE at pageable granularity: real per-token concentration exists
     at neuron level (20% for 95% — consistent with TEAL-class ~40–50%
     approximate sparsity claims), but bundling to SSD-friendly blocks
     destroys it: at 64 neurons (≈ 256 KiB rows bundle at 1.7B dims) the
     important set is 77% of the layer — no meaningful byte savings.
     Important neurons are SCATTERED, not clustered: block energy ≈
     uniform mixing. This is the quantitative reason LLM-in-a-flash used
     ReLU models — SwiGLU energy has no exploitable block structure.
  2. The expH fetch contract (≥256 KiB) and neuron-level concentration
     (4 KiB-scale rows) are mutually exclusive on this architecture:
     the SSD wants big blocks, the sparsity lives in small ones. A
     permutation/clustering pass (grouping co-active neurons) is the one
     remaining idea for this route — but expB (below) must first show the
     sets are stable enough to be worth clustering.
  3. The depth gradient is scientifically interesting: late layers are
     energy-concentrated but (expF) decision-insensitive; early layers are
     decision-relevant but energy-diffuse. Energy is not the right
     importance signal for escalation — margins are (expG).

Next experiment
  expB on the same trace (temporal stability) — run before drawing final
  conclusions on route (c); if sets churn too, the sparsity-paging family
  (G06/G07/G12/G13) dies for dense SwiGLU models at this scale.

expA_weight_concentration#

Weight contribution concentration: can a subset of weight blocks reproduce most of each layer's output?

Status: scaffolded 2026-08-11, not yet run.