localvm-research

Research / research/notes/out_of_core_memory_systems.md

out of core memory systems

draft

research/notes/out_of_core_memory_systems · created Mon Aug 10 2026 20:00:00 GMT-0400 (heure avancée de l’Est) · Simon-Pierre Boucher

Out-of-core LLM inference and memory systems (charter §4.4 + §4.8)#

Research notes on out-of-core inference systems, macOS/Apple Silicon substrate facts, and memory-systems ideas from OS/architecture/databases that have not yet been translated to neural-weight execution. Target hardware framing throughout: Apple M5 Max, 48 GB unified memory, macOS 27, internal Apple NVMe AP2048Z (2 TB), Metal/MLX.

All sources listed in §6 were located and, where marked, fetched on 2026-08-11.

1. Landscape#

Out-of-core inference means executing a model whose weights (and/or KV cache) exceed fast memory, by staging data across a hierarchy: GPU HBM → CPU DRAM → NVMe → (network). Every published system is a point in a small design space defined by four questions:

  1. What is the unit of paging? Whole layers (AirLLM, ZeRO-Inference), tensors/blocks (FlexGen, llama.cpp pages), neurons/neuron clusters (PowerInfer-1/2, LLM-in-a-flash, M2Cache), experts (MoE offloading), or KV blocks (SolidAttention, InstInfer, Swarm).
  2. What decides residency? Static placement (FlexGen's linear program, PowerInfer's offline hot/cold profiling), OS demand paging (llama.cpp mmap), explicit LRU/segmented caches (Eliseev–Mazur, PowerInfer-2, M2Cache), or predictors (LLM-in-a-flash sparsity predictors, speculative expert prefetch).
  3. What hides transfer latency? Batch-level pipelining (FlexGen, ZeRO-Inference), compute/I-O overlap at neuron-cluster granularity (PowerInfer-2), speculative prefetch (Eliseev–Mazur, FlexInfer), or nothing (AirLLM).
  4. What is the true bottleneck? Almost never capacity. For dense models streamed per token it is storage bandwidth; for sparse/selective access it is small-random-read latency and read amplification; for discrete-GPU systems it is PCIe bandwidth and CPU↔GPU synchronization; for batch-throughput systems the bottleneck is deliberately traded against latency.

A key structural observation for this project: most of the literature assumes Linux + discrete GPU, where there are two memory tiers above storage (VRAM and DRAM) joined by PCIe (16–64 GB/s). On Apple Silicon the VRAM/DRAM distinction vanishes (unified memory), so the hierarchy collapses to exactly two tiers: **unified RAM (460–614 GB/s on M5 Max) and internal NVMe (~5–7 GB/s sequential, far less for small random reads)**. That is a ~100:1 bandwidth cliff with no intermediate tier — but also no PCIe copy, no pinned-memory staging, and a legitimate zero-copy path from mmap'd file pages into GPU-visible Metal buffers. Several published designs (FlexGen's DRAM tier, PowerInfer's GPU/CPU split) are meaningless on this substrate; others (LLM-in-a-flash, PowerInfer-2) map almost directly onto it.

The single most important accounting identity (charter §2/§8.3): out-of-core systems decouple capacity (checkpoint can exceed RAM) but only the sparsity/selectivity systems decouple bytes-read-per-token from checkpoint size. AirLLM proves capacity decoupling alone is nearly worthless (full checkpoint read per token → minutes/token); LLM-in-a-flash and MoE-offloading prove bytes/token decoupling is where all the leverage is.

2. Systems#

2.1 FlexGen (ICML 2023)#

  • Mechanism: Offloading for throughput-oriented (latency-insensitive) generation on one commodity GPU. Formulates tensor placement/compute scheduling across GPU/CPU/disk as a linear program over a "zig-zag" block schedule; compresses weights and KV cache to 4 bit. OPT-175B on a single 16 GB T4-class GPU at ~1 token/s effective throughput with very large batches.
  • Resident vs streamed: weights partitioned across GPU/CPU/NVMe by the LP; entire active layer streamed per use; KV cache also tiered.
  • Bottleneck: PCIe bandwidth + disk bandwidth, amortized over huge batches; per-request latency is minutes. Bytes/token for the batch ≈ full model per layer-pass, divided by batch size per request.
  • Hardware assumptions: Linux, discrete GPU, separate CPU DRAM tier, pinned-memory DMA. None of this maps to Apple Silicon; the LP-placement idea does (placement between RAM and SSD), the batching trick does not help interactive local use.
  • macOS arm64: not supported (CUDA-only).
  • Extension opportunity: the formalization — solve residency as an optimization problem given measured tier bandwidths — is reusable for a RAM/SSD split on Mac.

2.2 DeepSpeed-Inference / ZeRO-Inference / DeepNVMe#

  • Mechanism: ZeRO-Inference streams model weights layer-by-layer from CPU DRAM or NVMe into GPU, overlapping fetch of layer k+1 with compute of layer k; recent versions add weight quantization and KV-cache offload ("20× throughput" claims). DeepNVMe adds io_uring/NVIDIA-GDS-based direct NVMe↔GPU transfer paths. Reported: OPT-175B full-offload ~43 tokens/s (CPU-offload) and ~30 tokens/s (NVMe) aggregate with large batches; prefetching improves NVMe offload 1.13–1.21×.
  • Resident vs streamed: essentially nothing resident; entire model read per forward pass — bytes/token = checkpoint size unless batched.
  • Bottleneck: PCIe/NVMe bandwidth (sequential; layer-granular reads are large, so latency and random I/O are non-issues). Throughput scales with batch size only.
  • Hardware assumptions: Linux (io_uring, O_DIRECT, GDS), discrete GPU. io_uring and O_DIRECT do not exist on macOS.
  • Extension opportunity: demonstrates the hard ceiling of dense layer-streaming: tokens/s ≤ storage_bandwidth / bytes_not_resident. On our SSD (~5–7 GB/s), a 70 GB non-resident model can never exceed ~0.1 tok/s dense, no matter the engineering.

2.3 "LLM in a flash" (Apple, arXiv 2312.11514) — the closest prior art to this project#

  • Mechanism: keep dense attention weights + embeddings in DRAM (~50 % of model), keep FFN weights on flash; a low-rank predictor (r=128 early layers → r=1024 late layers, <2.4 % overhead) predicts which ReLU-sparse FFN neurons will activate; only those rows are read. Windowing (reuse neurons active in the last k=4 tokens; only load the delta) and row–column bundling (store up-proj column + down-proj row of the same neuron contiguously, doubling read chunk size) reduce and enlarge I/O respectively.
  • What's resident vs streamed: ~52 % of model in DRAM (OPT-6.7B: 52.1 %, Falcon-7B: 52.9 %); per token only 2.4 % (OPT) / 3.1 % (Falcon) of FFN neurons are loaded.
  • Measured (on macOS!): Apple M1 Max, 1 TB SSD and M2 Ultra 2 TB: >6 GiB/s for a 1 GiB linear read, but random-read throughput increases with chunk size and thread count; effective ~2.25 GB/s after bundling. I/O latency per token dropped from 2196 ms (naive half-model reload) to 105 ms on M1 Max. 4–5× CPU and 20–25× GPU speedup vs naive loading; runs models ~2× DRAM size.
  • Bottleneck: flash random-read latency / read amplification — attacked by making reads bigger (bundling) and fewer (windowing, predictor).
  • Limitations: depends on ReLU-level activation sparsity (OPT, ReLU-fied Falcon); modern SwiGLU models (Llama-3, Qwen) have far weaker natural sparsity. No public code. Only ~2× DRAM demonstrated.
  • Extension opportunity: it validates the whole premise of localvm-research on our exact platform, and its flash-throughput-vs-chunk-size measurements are directly reusable priors for expH. Open gap: doing this for SwiGLU/MoE models, and integrating with Metal GPU compute rather than CPU.

2.4 PowerInfer (SOSP 2024)#

  • Mechanism: neuron activations follow a power law: a small set of "hot" neurons fire for most inputs. Hot neurons are preloaded into limited GPU VRAM; cold neurons are computed on CPU from DRAM (avoiding PCIe transfer); adaptive per-layer activation predictors + neuron-aware sparse kernels. Up to 11.69× over llama.cpp on a 4090; runs OPT-175B on one consumer GPU.
  • Resident vs streamed: hot neurons resident in VRAM, cold in DRAM; nothing streamed from disk in the base design (model ≤ DRAM assumed).
  • Bottleneck addressed: PCIe transfer + VRAM capacity; assumes model fits in DRAM — it is a VRAM/DRAM tiering system, not a true out-of-core system.
  • macOS mapping: the GPU/CPU split is meaningless under unified memory; the analog is hot neurons wired in RAM / cold neurons on SSD, i.e., exactly the LLM-in-a-flash regime. The transferable asset is the power-law hotness statistics and the offline profiling methodology (feeds our expA/expC).

2.5 PowerInfer-2 (arXiv 2406.06282)#

  • Mechanism: smartphone (24 GB Snapdragon, UFS 4.0 flash) inference of models beyond DRAM. Decomposes matmuls into neuron clusters; NPU handles dense prefill with large clusters, CPU handles sparse decode with small clusters; segmented neuron cache with per-segment policies; fine-grained I/O/compute pipelining at cluster granularity; I/O reads sized/aligned to flash characteristics.
  • Reported: first to run a 47B model on a phone; up to 11.68 tokens/s for TurboSparse- Mixtral-47B; ~22× faster than llama.cpp-class baselines when the model doesn't fit.
  • Bottleneck: UFS random-read latency + bandwidth, hidden with cluster-level pipelining; synchronization overhead of fine-grained pipelining explicitly engineered around.
  • Hardware assumptions: Android/Linux, heterogeneous NPU/CPU, UFS (much slower than Apple NVMe). This is the closest architectural template for a Mac: single shared memory pool + flash, no discrete GPU. Its flash is ~4× slower than ours; unified-memory Metal compute is far stronger than a phone CPU — the design should transfer favorably.
  • Limitation: requires activation-sparse ("TurboSparse") fine-tuned model variants — violates our "original model is the source model" constraint unless sparsity is achieved post-hoc.

2.6 M2Cache (arXiv 2410.14740)#

  • Mechanism: neuron-level mixed-precision + three-tier cache: neuron-level mixed-precision LRU cache in GPU HBM → layer-aware DRAM cache → full model on SSD. A predictor scores neuron activity; important neurons run FP16, less active ones are quantized more aggressively or demoted.
  • Relevance: first system I found that combines precision and tier as one axis — i.e., the residency decision and the precision decision are unified. Directly relevant to charter expD (progressive reconstruction): the "compressed tier" idea (§4.8 zswap analogy) already has one instantiation.
  • Bottleneck: SSD bandwidth; hidden with precision reduction (fewer bytes) rather than only prediction.
  • macOS: CUDA; concepts portable.

2.7 SolidAttention (FAST 2026)#

  • Mechanism: KV-cache (not weights) on SSD for memory-constrained PCs; identifies the conflict between dynamic sparse attention (wants small random reads) and SSD characteristics (want large sequential reads); consolidates KV pairs into blocks as the transfer unit, transforming irregular access into coarse-grained sequential access. Up to 3.1× faster inference, 98 % KV memory reduction at 128k context, ≤11 % throughput degradation vs in-memory.
  • Lesson: identical shape to LLM-in-a-flash's bundling lesson, proven for KV instead of weights: on flash, the paging unit must be chosen by the storage medium, not by the model's natural granularity. Any localvm design must co-design block layout with the ~16–256 KB sweet spot of Apple NVMe.

2.8 InstInfer / computational storage (arXiv 2409.04992)#

  • Mechanism: offloads attention computation into CSD (computational storage drives), so KV never crosses the host bus. Near-storage inference.
  • macOS relevance: none directly (Apple SSD controllers are closed), but conceptually: Apple's SSD controller already does inline encryption/compression; "compute where data lives" on a Mac translates to decompression/dequantization on GPU at load time, which MLX quantized kernels already do.

2.9 llama.cpp (mmap + Metal) — most important existing macOS evidence#

  • Mechanism: since PR #613/issue #91 (2023), model files are mmap'd; loading is lazy via page faults and the OS page cache (100× faster warm loads, half the memory — weights live once in the unified buffer cache, shared across processes). On Apple Silicon, the Metal backend accesses the mmap'd weights zero-copy via MTLResourceStorageModeShared (ggerganov: Metal "looks directly at the memory mapped buffers"; CUDA by contrast copies into device buffers). GGUF tensor data is alignment- padded, which is what makes wrapping file pages in Metal buffers possible.
  • Out-of-core behavior: when a model (esp. MoE) exceeds RAM, execution works by OS demand paging: hot expert pages stay in the UBC, cold ones fault in from SSD. Discussion #18758 (Dec 2025–2026) measured this concretely, including on an M5 Pro (Apple SSD AP1024Z) with Qwen3-Next-80B-A3B: expert weights are 95.4 % of file bytes; replacing fault-based streaming with explicit layout-aware slice reads gave +13–14 % end-to-end and reduced cold-decode reads from 1418 to 370 per token (2.23× faster cold-decode I/O); mmap beat direct I/O because the page cache retains the hot expert working set across tokens/runs.
  • Bottleneck: SSD random-read amplification (16 KB page faults scattered across expert tensors) + page-cache eviction unpredictability. Layout (contiguous per-expert placement) is as important as caching policy.
  • Limitations: replacement policy is the kernel's (approximate LRU, opaque, scan-vulnerable); no model-aware prefetch (a router decision is known before the expert FFN runs, but nothing uses it); feature request #20757 (two-tier GPU+RAM expert cache with pluggable eviction) is open — i.e., the gap is acknowledged and unfilled.
  • Runs on macOS arm64: yes, first-class.

2.10 MLX (Apple)#

  • Mechanism: arrays live in unified memory; device chosen per operation (stream=mx.cpu/mx.gpu) with automatic cross-stream dependencies; lazy evaluation builds a graph and materializes arrays only when needed; mx.load on safetensors/GGUF is lazy — weights are read from file when first evaluated, not all at load.
  • Out-of-core status: none. Maintainer (awni, discussion #615) states mmap wouldn't solve the problem since weights must still be materialized in (wired) memory for GPU use; lazy loading is the offered substitute. A community mmap prototype (antbob) found the practical blockers: safetensors tensor offsets are not page-aligned (Metal bytesNoCopy needs page alignment), and once the model exceeds RAM, uncontrolled page- cache eviction collapsed a 70 GB model on 64 GB hardware to 0.025 tokens/s (vs 6 tok/s quantized-fits-in-RAM). This is the negative result that defines our problem: naive mmap + kernel LRU is catastrophically bad for cyclic dense weight access.
  • Memory control: mx.set_wired_limit / Metal residency sets pin working memory; macOS iogpu.wired_limit_mb caps total GPU-wired memory (default ~66–75 % of RAM). mlx-lm issue #883 documents the failure mode when wiring is unbounded: IOGPUMemory kernel panic — wired memory is invisible to compressor/jetsam.
  • Extension opportunity: MLX is the natural host for our runtime (custom Metal kernels, lazy graph, quantized matmuls), but every paging/caching mechanism must be built by us — MLX offers none.

2.11 MoE offloading line: Eliseev & Mazur; caching/prefetch analyses; MoBiLE; cache-conditional experts#

  • Eliseev & Mazur (arXiv 2312.17238): Mixtral-8x7B on 11–16 GB consumer GPUs. Exploits (a) temporal locality of expert choice between adjacent tokens → LRU expert cache; (b) hidden state of layer k already predicts layer k+1's router choice → speculative expert prefetch (apply next layer's gate to current hidden state). 2–3 tokens/s on T4/RTX 3060-class GPUs with mixed quantization. This is "branch prediction for weights" in embryonic form.
  • In-depth caching/prefetching analysis (arXiv 2511.05814): measures expert reuse and prefetch accuracy across MoE models — confirms LRU-friendly temporal locality and cross-layer predictability are general, not Mixtral quirks.
  • Mixture of cache-conditional experts (arXiv 2412.00099): inverts the problem — biases the router toward experts already in cache (cache-aware routing), trading a little quality for dramatically fewer misses. Notable: the model adapts to the memory system rather than vice versa.
  • MoBiLE (arXiv 2510.12357): "big/little" experts on consumer GPUs — fallback to smaller substitutes for missing experts rather than blocking on I/O. A quality-for-latency miss handler — architecturally interesting for us: a miss need not stall if a low-precision resident approximation exists (connects to expD).

2.12 Petals (BitTorrent-style distributed inference)#

  • Mechanism: transformer blocks sharded across volunteer consumer GPUs; activations forwarded peer-to-peer (DHT discovery); Llama-3.1-405B / BLOOM-176B at ~1 step/s — claimed up to 10× faster than local disk offloading.
  • Relevance: replaces the SSD tier with a network tier (both ~GB/s, both high-latency) — confirms that activations are the cheap thing to move; weights are the expensive thing. For localvm the analogous observation: moving hidden states between compute contexts is ~MB/token; moving weights is ~GB/token. Any decomposition should ship activations, not weights. (Also relevant to the MacLustr cluster as a side path, though out of scope for the single-Mac charter.)

2.13 AirLLM (layer-by-layer streaming)#

  • Mechanism: load layer → compute → free → next layer; peak memory ~4 GB for a 70B model. Bytes/token = entire checkpoint (every layer re-read per token unless cached); a single response takes 15–30 minutes.
  • Value: the perfect straw-man baseline for our harness — pure capacity decoupling with zero bytes/token decoupling. Its existence proves the charter's core inequality (total size ≠ resident size ≠ bytes/token) is the entire game.

2.14 FlexInfer (arXiv 2503.03777) and Glinthawk (arXiv 2501.11779)#

  • FlexInfer: on-device offloading with asynchronous prefetching, balanced memory locking (explicitly budgeting pinned vs pageable memory), and flexible tensor preservation; up to 12.5× over existing offloading under tight memory. The "balanced memory locking" idea maps directly onto the macOS wired-limit / residency-set tension documented in §3.
  • Glinthawk: two-tier architecture for offline batch inference (throughput regime, like FlexGen) — noted for completeness; wrong latency regime for us.

3. macOS / Apple Silicon substrate (API-level facts)#

3.1 Files, page cache, and the absence of direct I/O#

  • No O_DIRECT, no posix_fadvise. The macOS substitute is fcntl(fd, F_NOCACHE, 1): it hints that pages should not be cached going forward, but (a) it does not purge already-cached pages — subsequent reads still hit them; (b) other processes can keep re-populating the cache; (c) it is per-fd advisory, not a DMA path (Apple dev forums #25464; fio issue #48). Community direct-I/O libraries document that with F_NOCACHE, unaligned reads are still buffered; to actually bypass the cache, offset/length should be 4096-byte (better: 16 KB page) aligned (ronomon/direct-io).
  • Unified Buffer Cache (UBC): since Mac OS X, the buffer cache and VM page cache are one; mmap'd file pages are the file cache pages. Consequences: warm model loads are ~free (llama.cpp's 100× warm-load speedup); memory shows as "cached files," is reclaimable, and is shared across processes mapping the same model file.
  • Eviction is opaque: the kernel's replacement is approximate LRU over the whole system; there is no fadvise(DONTNEED/WILLNEED); madvise exists but with weaker semantics. purge(8) clears the disk cache for cold-start experiments; vm_stat exposes pageins/pageouts/compressor counters for instrumentation (expH must use both).
  • Writeback of mmap(MAP_SHARED) dirty pages is at the OS's discretion until msync (Apple dev forums #763058) — matters if we ever write compiled-model caches through mmap.
  • APFS: copy-on-write, 4 KB blocks; clones and sparse files are free — useful for storing multiple weight layouts of the same checkpoint without duplicating cold data. (Performance note: APFS metadata ops are slow relative to data reads; large flat blob files with internal indexing beat many-small-files layouts.)

3.2 Memory: compression, wiring, jetsam#

  • Compressor: since OS X 10.9, LRU-cold anonymous pages are compressed (WKdm-family algorithm, tiny 16-entry dictionary, ~2:1 on pointer/integer-rich data) before any swap. Quantized/fp16 weights are near-incompressible entropy, so the compressor gives ~0 benefit on weight pages while burning CPU — weight overflow should go to file-backed (evict-don't-compress) memory, never anonymous memory. This asymmetry (file-backed pages get dropped, anonymous pages get compressed/swapped) is a design lever.
  • Wired memory: GPU-active buffers must be wired (non-pageable, non-compressible). Cap is iogpu.wired_limit_mb — default ≈ 66–75 % of RAM (≈ 32–36 GB on our 48 GB M5 Max), adjustable via sudo sysctl iogpu.wired_limit_mb=N (resets on reboot; unsupported by Apple; leave 8–16 GB headroom). MLX exposes set_wired_limit / residency sets. Over-wiring does not trigger graceful jetsam — mlx-lm #883 shows it can end in an IOGPUMemory kernel panic; macOS prefers compression over jetsam-style killing, but wired memory is exempt from both, hence the panic path.
  • Page size is 16 KB on Apple Silicon — the natural minimum paging unit for any weight-block store (also the fault granularity that produced the 1418 reads/token in llama.cpp #18758).

3.3 Metal: zero-copy, heaps, purgeability#

  • newBufferWithBytesNoCopy / makeBuffer(bytesNoCopy:) wraps existing memory as a MTLBuffer with no copy, but the pointer must be page-aligned and the length a multiple of page size, and the memory must come from mmap/vm_allocate (not malloc) (Apple docs; dev forums #8011). This is exactly how llama.cpp gets the GPU to read weights straight out of the page cache. Design consequence: our compiled weight format must place every independently-pageable block on a 16 KB boundary (GGUF does alignment padding; safetensors does not — the root cause of MLX's mmap dead-end).
  • MTLStorageModeShared: CPU and GPU share the allocation in system memory — the default and correct mode on Apple Silicon (no managed/private copies needed).
  • MTLHeap: suballocate many buffers from one allocation; resources can alias; setPurgeableState on a heap makes its whole backing memory volatile — the OS may reclaim it under pressure and tells you on reacquire whether contents survived. A purgeable MTLHeap is a kernel-cooperative weight cache: warm blocks live there, the OS reclaims them instead of paging/killing, and we re-fault from SSD on loss. No LLM runtime uses this today (see §4).
  • Residency sets / wired limit: the modern mechanism to guarantee the hot tier stays resident during command-buffer execution (MLX's wired-memory doc).

3.4 Apple NVMe (AP-class) measured behavior#

  • Sequential: recent MacBook Pro internal SSDs (AP2048/AP4096-class) measure ~5.4–7.3 GB/s reads (Blackmagic/AmorphousDiskMark reports for M4 Max 2–4 TB; M5-generation press claims ~2× M4 SSD speed). "LLM in a flash" measured >6 GiB/s for a 1 GiB linear read on an M1 Max 1 TB.
  • Small random reads are the cliff: community AmorphousDiskMark results consistently show a 4K QD1 dip on Apple Silicon — on the order of tens of MB/s (one documented M1 Pro result: 32 MB/s = ~8 K IOPS = ~120 µs effective latency), i.e. **200× below sequential**. Throughput recovers with (a) larger blocks and (b) concurrency: Apple's paper reports random-read throughput "increases with the size of sequential chunks and the number of threads," reaching ~2.25 GB/s effective with 32 KB-bundled multi-threaded reads. Rule of thumb for design: ≥256 KB blocks at QD≥8, or don't bother; expH must measure our exact AP2048Z across 16 KB–4 MB, QD1–32, cold (purge) vs warm, with and without F_NOCACHE, and while Metal compute runs (shared memory-controller contention is unmeasured in the literature).
  • Bandwidth ratio on target: M5 Max unified memory = 460–614 GB/s (per Apple specs; config-dependent) vs 6 GB/s SSD sequential → **80–100:1**; vs realistic mixed random 2 GB/s → **250:1**. Every design decision follows from this ratio.

4. OS/architecture/database ideas not yet translated to neural-weight execution#

  1. Working-set theory (Denning 1968). No LLM runtime measures a τ-window weight working set W(τ), yet the concept transfers exactly: the set of weight blocks touched in the last τ tokens. Denning's thrashing criterion (if RAM < W(τ), throughput collapses) gives a principled admission test: measure W(τ) per model/workload offline and predict — before running — whether a config will thrash (this is what MLX's 0.025 tok/s mmap failure was: W(τ) = whole model for dense access). Working-set analytics (Denning's later surveys) also give the math for choosing cache sizes from reuse- distance histograms — directly applicable to expB.
  2. Scan-resistant / adaptive replacement (ARC, 2Q, LIRS). Kernel page cache and all published expert caches use (approximate) LRU. But transformer weight access is a mixed workload: dense layers are pure cyclic scans (LRU's pathological worst case — it evicts every block just before reuse), while MoE experts have recency+frequency structure. ARC's ghost lists would auto-partition between the two. Nobody has published an ARC/LIRS weight cache. Even better than ARC:
  3. MRU for cyclic scans (DBMIN's insight). Databases learned in 1985 that for a looping sequential scan over N pages with a buffer of B<N, MRU is optimal and LRU is worst-case. Dense-decode weight access is a looping sequential scan (same order every token). A dense model overflowing RAM by X GB under MRU keeps a stable (model−X) resident set and re-reads exactly X GB/token — the theoretical floor — whereas LRU re-reads everything. Trivial to implement, apparently never applied.
  4. Query-informed buffer management (DBMIN / QLSM). DBMIN allocates a separate buffer pool per file instance with a policy chosen from the known access pattern of the query plan. A transformer's "query plan" is fully known: layer order is static; MoE router output is known milliseconds before the expert is needed; attention head usage is measurable. Per-tensor-class pools (embeddings: pin; dense scan: MRU ring; experts: ARC; KV: sliding window) with plan-derived sizes is a direct, unexplored translation.
  5. Anti-caching (H-Store/VLDB 2013). Inverts caching: memory is primary, cold data is evicted to disk with tombstones; a transaction touching evicted data aborts, the data is fetched asynchronously, and the transaction restarts — no thread ever blocks on disk. Translation: a token step that needs a non-resident expert could proceed speculatively with a resident low-precision substitute (MoBiLE-style) or abort-and- replay that layer after async fetch, keeping the GPU busy. "Never block compute on a miss" as an architectural invariant has no LLM instantiation.
  6. Prefetching as branch prediction (Pythia, MICRO 2021). Hardware prefetchers use program context + online RL to issue accurate, bandwidth-aware prefetches. The LLM analog has richer context than any CPU: hidden states. Eliseev–Mazur's one-layer-ahead gate trick is a static 1-bit predictor by comparison; a small online-learned prefetcher consuming hidden-state features and issuing SSD reads N layers ahead (with reward = hit-rate minus wasted bandwidth, Pythia-style) is unexplored.
  7. Tiered-memory page placement (TPP/ASPLOS'23, Pond). Hot/cold page promotion and demotion between DRAM and CXL based on lightweight access sampling. Translation: background promotion/demotion of weight blocks between wired-RAM / purgeable-RAM / SSD tiers driven by per-block access counters — no runtime does tier migration of weights during inference; placements are static after profiling (PowerInfer) or purely reactive (page faults).
  8. Compressed memory tier (zswap / macOS compressor). The OS compressor is useless on weight entropy (§3.2), but the architecture — a middle tier holding a cheaper representation, faulting to the full representation — translates as: low-bit weights resident in RAM as the "compressed tier," full-precision residuals on SSD, fetched only when needed (error-driven). This is M2Cache's direction and our expD; the OS analogy suggests the policy structure (compress on demotion, decompress on promotion, track compression benefit per page).
  9. Purgeable/volatile memory (macOS-specific, §3.3). Kernel-cooperative caches (volatile heaps the OS may reclaim, with reclaim notification) have existed since iOS's NSPurgeableData era. No inference runtime marks its warm weight cache purgeable; doing so converts "jetsam/panic risk" into "graceful quality/latency degradation under pressure."
  10. Economic residency rules (five-minute-rule style). Databases decide RAM residency by break-even between storage cost and access frequency. Per-block: wire it if (expected accesses/s × fetch cost) exceeds its RAM rent. Gives a closed-form split of a 48 GB budget across embeddings/dense/experts/KV rather than ad-hoc tuning. (See the buffer-management evolution survey for the modern framing incl. learned policies.)
  11. TLB/superpage thinking. 16 KB pages mean a 100 GB mapping has ~6.5 M PTEs; fault storms are Mach-message-expensive. Batching faults via explicit large-block reads into pre-mapped wired arenas (as llama.cpp #18758's slice-read experiment did: 1418→370 reads/token) is the superpage lesson in disguise; nobody states it as policy.

5. Relevance to localvm-research (M5 Max, 48 GB, AP2048Z)#

Where the bottleneck actually is on our target. With 460–614 GB/s RAM and ~6 GB/s (sequential) / ~2 GB/s (practical random) SSD:

  • If the active working set fits in RAM, decode is RAM-bandwidth-bound: ceiling ≈ 460 GB/s ÷ resident-active-bytes. A fully-resident 40 GB (Q4 70B) model → ~11 tok/s ceiling. This is the regime llama.cpp/MLX already serve.
  • If a dense model overflows RAM by X GB, the SSD must supply ≥X GB/token (MRU floor): X = 10 GB → ≥1.7–5 s/token. Dense overflow is irrecoverable by systems engineering alone — confirmed by ZeRO-Inference math, AirLLM, and the MLX mmap prototype (0.025 tok/s). Therefore bytes/token must be decoupled from checkpoint size before paging can help: activation sparsity (LLM-in-a-flash: 2–3 % of FFN/token), MoE routing (only active experts), or progressive precision (low-bit resident core + on-demand residuals).
  • Once access is selective, the bottleneck flips from bandwidth to random-read latency and read amplification — the fight moves to layout (16 KB-aligned, co-located bundles; SolidAttention/row-column-bundling lesson), replacement policy (ARC/MRU vs kernel LRU), and prefetch (router/hidden-state-driven, Pythia-style), with QD≥8 large-block reads overlapped with Metal compute.

What is uniquely favorable on macOS. (1) Zero-copy SSD→page-cache→GPU: file pages can be wrapped in Metal buffers with no copy (llama.cpp proves it in production) — discrete-GPU systems can't do this, so most published overheads (PCIe staging, pinned pools) simply vanish. (2) Purgeable heaps + residency sets give a three-tier RAM hierarchy (wired-hot / purgeable-warm / evictable page cache) that no other OS exposes as cleanly. (3) The page cache is shared and persistent across processes/runs — warm-start economics are excellent. What is uniquely unfavorable: no O_DIRECT/io_uring (F_NOCACHE is a weak hint; async I/O = thread pools or POSIX AIO), opaque eviction, 16 KB fault granularity, and the wired-limit/panic cliff.

Three most promising openings (ranked):

  1. Model-aware weight pager for MoE/sparse models on Metal — a compiled, 16 KB-aligned block store (APFS-friendly single blob) + ARC/ghost-list expert cache in a purgeable MTLHeap + router-lookahead prefetch (gate of layer k+1 applied at layer k, à la Eliseev–Mazur, generalized to N-layer Pythia-style learned prefetch) + explicit QD≥8 large-slice reads instead of fault streaming. Every ingredient has isolated evidence (llama.cpp #18758: 2.23× cold I/O from layout alone; #20757 open feature request; MoE temporal locality confirmed); the composition doesn't exist anywhere, least of all on macOS.
  2. DBMIN-style per-tensor-class buffer management with working-set admission control — per-class policies (pin embeddings; MRU ring for dense cyclic scans; ARC for experts; economic wiring of the hot tier under the iogpu wired limit), driven by offline W(τ) profiles (feeds directly on expA/expB/expC outputs). Cheap to build, high novelty (the MRU-for-cyclic-weight-scans observation appears to be unpublished), and it defines the measurement framework we need anyway.
  3. Precision-tiered residency ("compressed tier for weights") — low-bit core resident and wired; residuals on SSD fetched on demand (error- or importance-driven), misses served by the resident approximation so compute never blocks (anti-caching invariant, MoBiLE fallback). This is the systems-side realization of expD and the only opening that helps dense models, where pure paging provably cannot.

Immediate experimental consequences: expH (SSD feasibility) should replicate Apple's chunk-size×threads throughput surface on the AP2048Z including concurrent-Metal-compute contention; expB (token stability) should be scored as reuse-distance histograms so working-set/ARC math applies directly; and the baseline harness must include llama.cpp mmap-overflow and AirLLM-style layer streaming as the two "pure capacity decoupling" controls.

Sources#