Bibliography#

Every consulted source, with URL and access date. Grouped by theme (mirrors research/notes/). Entries are appended as research progresses; nothing is deleted.

Quantization (§4.1)#

From research/notes/quantization.md (69 sources).

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al., ICLR 2023) — https://arxiv.org/abs/2210.17323 (accessed 2026-08-11)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Lin et al., MLSys 2024) — https://arxiv.org/abs/2306.00978 (accessed 2026-08-11)
• SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs (Xiao et al., ICML 2023) — https://arxiv.org/abs/2211.10438 (accessed 2026-08-11)
• A Practical Guide to INT4 Quantization for SLMs: GPTQ vs AWQ (Microsoft Data Science, Medium) — https://medium.com/data-science-at-microsoft/a-practical-guide-to-int4-quantization-for-slms-gptq-vs-awq-olive-and-real-world-results-2f63d6963d1d (accessed 2026-08-11)
• Combining multiple post-training techniques to achieve most efficient quantized LLMs (MX formats + GPTQ/SmoothQuant) — https://arxiv.org/html/2405.07135v1 (accessed 2026-08-11)
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees (Chee et al., NeurIPS 2023) — https://neurips.cc/virtual/2023/poster/69982 (accessed 2026-08-11)
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (Tseng et al., ICML 2024) — slides — https://icml.cc/media/icml-2024/Slides/34816.pdf (accessed 2026-08-11)
• QuIP# codebase (Cornell RelaxML) — https://github.com/Cornell-RelaxML/quip-sharp (accessed 2026-08-11)
• QuIP# full text (PMC mirror, RVQ details) — https://pmc.ncbi.nlm.nih.gov/articles/PMC12395268 (accessed 2026-08-11)
• QTIP: Quantization with Trellises and Incoherence Processing (Tseng et al., NeurIPS 2024) — https://arxiv.org/html/2406.11235v1 (accessed 2026-08-11)
• Even Better, Even Faster Quantized LLMs with QTIP (Together AI blog) — https://www.together.ai/blog/even-better-even-faster-quantized-llms-with-qtip (accessed 2026-08-11)
• AQLM: Extreme Compression of Large Language Models via Additive Quantization (Egiazarian et al., ICML 2024) — https://arxiv.org/html/2401.06118v2 (accessed 2026-08-11)
• AQLM + PV-Tuning official repository — https://github.com/vahe1994/AQLM (accessed 2026-08-11)
• PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression (NeurIPS 2024) — https://proceedings.neurips.cc/paper_files/paper/2024/file/091166620a04a289c555f411d8899049-Paper-Conference.pdf (accessed 2026-08-11)
• The Evolution of Extreme LLM Compression: From QuIP to AQLM with PV-Tuning (Yandex, Medium) — https://medium.com/yandex/the-evolution-of-extreme-llm-compression-from-quip-to-aqlm-with-pv-tuning-19c44b91af96 (accessed 2026-08-11)
• GPTVQ: The Blessing of Dimensionality for LLM Quantization (van Baalen et al., Qualcomm) — https://arxiv.org/abs/2402.15319 (accessed 2026-08-11)
• GPTVQ repository — https://github.com/Qualcomm-AI-research/gptvq (accessed 2026-08-11)
• NestQuant: Nested Lattice Quantization for Matrix Products and LLMs (Savkin et al., ICML 2025) — https://arxiv.org/abs/2502.09720 (accessed 2026-08-11)
• Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression (NeurIPS 2025) — https://neurips.cc/virtual/2025/poster/117396 (accessed 2026-08-11)
• SqueezeLLM: Dense-and-Sparse Quantization (Kim et al., ICML 2024) — https://arxiv.org/html/2306.07629v4 (accessed 2026-08-11)
• Half-Quadratic Quantization of Large Machine Learning Models (Mobius Labs, via Dropbox Tech) — https://dropbox.tech/machine-learning/halfquadratic-quantization-of-large-machine-learning-models (accessed 2026-08-11)
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (Ashkboos et al., NeurIPS 2024) — https://neurips.cc/virtual/2024/poster/94328 (accessed 2026-08-11)
• SpinQuant: LLM Quantization with Learned Rotations (Liu et al., ICLR 2025) — https://proceedings.iclr.cc/paper_files/paper/2025/file/e5b1c0d4866f72393c522c8a00eed4eb-Paper-Conference.pdf (accessed 2026-08-11)
• Rotation-based quantization with QuaRot (AMD Quark docs, R1–R4 rotations) — https://quark.docs.amd.com/release-0.9/pytorch/tutorial_quarot.html (accessed 2026-08-11)
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (Ma et al., Microsoft) — https://arxiv.org/abs/2402.17764 (accessed 2026-08-11)
• 1-bit AI Infra Part 1.1: Fast and Lossless BitNet b1.58 Inference on CPUs (bitnet.cpp) — https://arxiv.org/html/2410.16144v1 (accessed 2026-08-11)
• Bitnet.cpp: Efficient Edge Inference for Ternary LLMs (ACL 2025; TL/I2_S kernels, M2 Ultra 100B result) — https://aclanthology.org/2025.acl-long.457.pdf (accessed 2026-08-11)
• microsoft/BitNet official inference framework — https://github.com/microsoft/BitNet (accessed 2026-08-11)
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (Huang et al., ICML 2024) — https://github.com/Aaronhuang-778/BiLLM (accessed 2026-08-11)
• ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization (Liu et al., Meta, NeurIPS 2025) — https://arxiv.org/html/2502.02631v2 (accessed 2026-08-11)
• ParetoQ (PyTorch blog) — https://pytorch.org/blog/paretoq-scaling-laws-in-extremely-low-bit-llm-quantization (accessed 2026-08-11)
• EfficientQAT: Efficient Quantization-Aware Training for LLMs (Chen et al., ACL 2025) — https://arxiv.org/abs/2407.11062 (accessed 2026-08-11)
• EfficientQAT repository (w2g64 PPL tables) — https://github.com/OpenGVLab/EfficientQAT (accessed 2026-08-11)
• Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (Park et al., ICML 2024 oral) — https://arxiv.org/html/2402.10517v4 (accessed 2026-08-11)
• Any-Precision LLM repository (bitplane engine) — https://github.com/SNU-ARC/any-precision-llm (accessed 2026-08-11)
• Matryoshka Quantization (Nair et al., Google DeepMind, ICLR 2025 oral) — https://iclr.cc/virtual/2025/10000114 (accessed 2026-08-11)
• Matryoshka Quantization topic overview (Emergent Mind) — https://www.emergentmind.com/topics/matryoshka-quantization-matquant (accessed 2026-08-11)
• Multi-Bitwidth Quantization for LLMs Using Additive Codebooks ("Drop-by-Drop", successive refinement) — https://arxiv.org/html/2606.12876v1 (accessed 2026-08-11)
• Progressive Mixed-Precision Decoding for Efficient LLM Inference (Chen et al., ICLR 2025) — https://arxiv.org/abs/2410.13461 (accessed 2026-08-11)
• Mixed-Precision Quantization for Language Models (survey, Oct 2025; PMDP/MPMLC taxonomy) — https://arxiv.org/html/2510.16805v1 (accessed 2026-08-11)
• SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators (Apple ML Research) — https://machinelearning.apple.com/research/seedlm-compressing (accessed 2026-08-11)
• SeedLM (arXiv full text) — https://arxiv.org/html/2410.10714v1 (accessed 2026-08-11)
• DFloat11: 70% Size, 100% Accuracy — Lossless LLM Compression via Dynamic-Length Float — https://huggingface.co/papers/2504.11651 (accessed 2026-08-11)
• DFloat11 repository (LeanModels, NeurIPS 2025) — https://github.com/LeanModels/DFloat11 (accessed 2026-08-11)
• DFloat11 throughput caveats (Hacker News discussion incl. appendix numbers) — https://news.ycombinator.com/item?id=43796935 (accessed 2026-08-11)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (Liu et al., ICML 2024) — https://arxiv.org/abs/2402.02750 (accessed 2026-08-11)
• KIVI repository — https://github.com/jy-yuan/KIVI (accessed 2026-08-11)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (Hooper et al., NeurIPS 2024) — https://arxiv.org/abs/2401.18079 (accessed 2026-08-11)
• KV Cache is 1 Bit Per Channel: Coupled Quantization (NeurIPS 2024) — https://proceedings.neurips.cc/paper_files/paper/2024/file/05d6b5b6901fb57d2c287e1d3ce6d63c-Paper-Conference.pdf (accessed 2026-08-11)
• mlx.core.quantize documentation (affine/mxfp4/mxfp8/nvfp4 modes) — https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.quantize.html (accessed 2026-08-11)
• mlx.nn.quantize documentation (quantize_input, class predicates) — https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.nn.quantize.html (accessed 2026-08-11)
• mlx-lm LEARNED_QUANTS.md (DWQ, dynamic_quant, AWQ/GPTQ recipes) — https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/LEARNED_QUANTS.md (accessed 2026-08-11)
• MLX Quantization on Apple Silicon: dynamic_quant vs AWQ vs GPTQ vs DWQ (Hannecke, Medium) — https://medium.com/@michael.hannecke/mlx-quantization-on-apple-silicon-dynamic-quant-vs-awq-vs-gptq-vs-dwq-8b2a5af2b53f (accessed 2026-08-11)
• Better inference quality and performance for MLX on Apple Silicon (Feldman; K-quant vs MLX affine KL measurements) — https://www.linkedin.com/pulse/better-inference-quality-performance-mlx-apple-silicon-asher-feldman-ztm0e (accessed 2026-08-11)
• Very slow IQ quant performance on Apple Silicon (llama.cpp discussion #5617, ikawrakow measurements) — https://github.com/ggml-org/llama.cpp/discussions/5617 (accessed 2026-08-11)
• Overview of GGUF quantization methods (r/LocalLLaMA; i-quant LUT bottleneck notes) — https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overview_of_gguf_quantization_methods (accessed 2026-08-11)
• LLM Quantization Formats Compared: GGUF vs MLX vs EXL3 vs GPTQ vs AWQ vs FP8 (D-Central; format inventories) — https://d-central.tech/llm-quantization-formats (accessed 2026-08-11)
• GGUF vs MLX Quantization Formats on Apple Silicon (Contra Collective, 2026) — https://contracollective.com/blog/gguf-vs-mlx-quantization-formats-apple-silicon-2026 (accessed 2026-08-11)
• llama.cpp Metal Backend vs MLX: Compute Path Comparison (Contra Collective, 2026) — https://contracollective.com/blog/llama-cpp-metal-vs-mlx-backend-apple-silicon-2026 (accessed 2026-08-11)
• llama.cpp supports gpt-oss in native MXFP4 (discussion #15095) — https://github.com/ggml-org/llama.cpp/discussions/15095 (accessed 2026-08-11)
• exllamav3 / EXL3 trellis format (turboderp) — https://github.com/turboderp-org/exllamav3 (accessed 2026-08-11)
• KV Cache and Context Length on Apple Silicon (Contra Collective, 2026; llama.cpp/mlx-lm KV flags) — https://contracollective.com/blog/kv-cache-context-length-apple-silicon-local-inference-2026 (accessed 2026-08-11)
• Running LLMs locally on a Mac (MacKinlay; KV cache quant flags across runtimes) — https://danmackinlay.name/notebook/local_llm_mac.html (accessed 2026-08-11)
• KVSplit: differentiated K/V precision on Apple Silicon (Show HN) — https://news.ycombinator.com/item?id=44009321 (accessed 2026-08-11)
• TurboQuant — Extreme KV Cache Quantization with Metal kernels (llama.cpp discussion #20969) — https://github.com/ggml-org/llama.cpp/discussions/20969 (accessed 2026-08-11)
• M2Cache: Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching — https://arxiv.org/abs/2410.14740 (accessed 2026-08-11)
• SSD Offloading for LLM MoE Weights Considered Harmful in Energy Efficiency — https://www.alphaxiv.org/overview/2508.06978 (accessed 2026-08-11)
• Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices (MLX Q4-KV state of play) — https://arxiv.org/html/2603.04428v1 (accessed 2026-08-11)
• MLX vs llama.cpp on Apple Silicon: Benchmarks, M5 Neural Accelerators, Ollama switch — https://yage.ai/share/mlx-apple-silicon-en-20260331.html (accessed 2026-08-11)

Activation & weight sparsity, pruning (§4.2–4.3)#

From research/notes/sparsity_pruning.md (48 sources).

• Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time — https://arxiv.org/abs/2310.17157 (accessed 2026-08-11)
• Deja Vu (OpenReview, ICML 2023) — https://openreview.net/forum?id=wIPIhHd00i (accessed 2026-08-11)
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU — https://arxiv.org/abs/2312.12456 (accessed 2026-08-11)
• PowerInfer (SOSP 2024 paper PDF, IPADS/SJTU) — https://ipads.se.sjtu.edu.cn/_media/publications/song-sosp24.pdf (accessed 2026-08-11)
• PowerInfer GitHub (macOS/Metal support status, supported ReLU models) — https://github.com/SJTU-IPADS/PowerInfer (accessed 2026-08-11)
• PowerInfer-2: Fast Large Language Model Inference on a Smartphone — https://arxiv.org/abs/2406.06282 (accessed 2026-08-11)
• PowerInfer-2 project page — https://powerinfer.ai/v2/ (accessed 2026-08-11)
• LLM in a flash: Efficient Large Language Model Inference with Limited Memory (Apple) — https://arxiv.org/abs/2312.11514 (accessed 2026-08-11)
• ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models (Apple, ICLR 2024) — https://arxiv.org/abs/2310.04564 (accessed 2026-08-11)
• ReLU Strikes Back — Apple Machine Learning Research page — https://machinelearning.apple.com/research/relu (accessed 2026-08-11)
• The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers — https://arxiv.org/abs/2210.06313 (accessed 2026-08-11)
• TEAL: Training-Free Activation Sparsity in Large Language Models — https://arxiv.org/abs/2408.14690 (accessed 2026-08-11)
• TEAL — Together AI blog — https://www.together.ai/blog/teal-training-free-activation-sparsity-in-large-language-models (accessed 2026-08-11)
• CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models (COLM 2024) — https://arxiv.org/abs/2404.08763 (accessed 2026-08-11)
• CATS GitHub — https://github.com/ScalingIntelligence/CATS (accessed 2026-08-11)
• GRIFFIN: Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation (ICML 2024) — https://arxiv.org/abs/2404.01365 (accessed 2026-08-11)
• GRIFFIN GitHub — https://github.com/hdong920/GRIFFIN (accessed 2026-08-11)
• ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models (EMNLP 2024) — https://arxiv.org/abs/2406.16635 (accessed 2026-08-11)
• ShadowLLM — ACL Anthology — https://aclanthology.org/2024.emnlp-main.1068/ (accessed 2026-08-11)
• ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models — https://arxiv.org/abs/2402.13516 (accessed 2026-08-11)
• Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters — https://arxiv.org/abs/2406.05955 (accessed 2026-08-11)
• Q-Sparse: All Large Language Models can be Fully Sparsely-Activated (NeurIPS 2024) — https://arxiv.org/abs/2407.10969 (accessed 2026-08-11)
• Sirius: Contextual Sparsity with Correction for Efficient LLMs (NeurIPS 2024) — https://arxiv.org/abs/2409.03856 (accessed 2026-08-11)
• SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024) — https://arxiv.org/abs/2312.04985 (accessed 2026-08-11)
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (ICML 2023) — https://arxiv.org/abs/2301.00774 (accessed 2026-08-11)
• Wanda: A Simple and Effective Pruning Approach for Large Language Models (ICLR 2024) — https://arxiv.org/abs/2306.11695 (accessed 2026-08-11)
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — https://arxiv.org/abs/2403.03853 (accessed 2026-08-11)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024) — https://arxiv.org/abs/2310.06694 (accessed 2026-08-11)
• Compact Language Models via Pruning and Knowledge Distillation (Minitron, NVIDIA) — https://arxiv.org/abs/2407.14679 (accessed 2026-08-11)
• LLM Pruning and Distillation in Practice: The Minitron Approach — https://arxiv.org/pdf/2408.11796 (accessed 2026-08-11)
• SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR 2024) — https://arxiv.org/abs/2401.15024 (accessed 2026-08-11)
• LLM-Pruner: On the Structural Pruning of Large Language Models (NeurIPS 2023) — https://arxiv.org/abs/2305.11627 (accessed 2026-08-11)
• M2Cache: Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching — https://arxiv.org/abs/2410.14740 (accessed 2026-08-11)
• Ripple/Neuralink: Accelerating LLM Inference on Smartphones with Correlation-Aware Neuron Management / Neuron Co-Activation Linking — https://arxiv.org/abs/2410.19274 (accessed 2026-08-11)
• DIP: Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking (Qualcomm AI Research) — https://arxiv.org/abs/2412.01380 (accessed 2026-08-11)
• Endor: Hardware-Friendly Sparse Format for Offloaded LLM Inference — https://arxiv.org/pdf/2406.11674 (accessed 2026-08-11)
• SparseInfer: Training-free Prediction of Activation Sparsity for Fast LLM Inference — https://arxiv.org/pdf/2411.12692 (accessed 2026-08-11)
• Post-Training Statistical Calibration for Higher Activation Sparsity — https://arxiv.org/pdf/2412.07174 (accessed 2026-08-11)
• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference — https://arxiv.org/abs/2504.19449 (accessed 2026-08-11)
• Spark Transformer: Reactivating Sparsity in FFN and Attention (NeurIPS 2025) — https://arxiv.org/html/2506.06644v2 (accessed 2026-08-11)
• Universal Properties of Activation Sparsity in Modern Large Language Models — https://arxiv.org/abs/2509.00454 (accessed 2026-08-11)
• RAP: Runtime Adaptive Pruning for LLM Inference — https://arxiv.org/pdf/2505.17138 (accessed 2026-08-11)
• DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs — https://arxiv.org/html/2506.20194 (accessed 2026-08-11)
• Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity — https://arxiv.org/pdf/2509.22166 (accessed 2026-08-11)
• VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking — https://arxiv.org/html/2511.18692 (accessed 2026-08-11)
• On-Demand Multi-Task Sparsity for Efficient Large-Model Deployment on Edge Devices — https://arxiv.org/pdf/2511.19986 (accessed 2026-08-11)
• Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsity — https://arxiv.org/pdf/2602.00397 (accessed 2026-08-11)
• Dynamic sparsity in tree-structured feed-forward layers at scale — https://arxiv.org/pdf/2604.08565 (accessed 2026-08-11)

Out-of-core inference & memory systems (§4.4, §4.8)#

From research/notes/out_of_core_memory_systems.md (59 sources).

• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — https://arxiv.org/abs/2303.06865 (accessed 2026-08-11)
• FlexLLMGen (FlexGen) README, FMInference — https://github.com/FMInference/FlexLLMGen/blob/main/README.md (accessed 2026-08-11)
• ZeRO-Inference: Democratizing massive model inference — https://www.deepspeed.ai/2022/09/09/zero-inference.html (accessed 2026-08-11)
• DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale — https://arxiv.org/pdf/2207.00032 (accessed 2026-08-11)
• DeepNVMe: Affordable I/O scaling for Deep Learning Applications (PyTorch blog) — https://pytorch.org/blog/deepnvme-affordable-i-o-scaling-for-deep-learning-applications/ (accessed 2026-08-11)
• LLM in a flash: Efficient Large Language Model Inference with Limited Memory — https://arxiv.org/abs/2312.11514 (accessed 2026-08-11)
• LLM in a flash (HTML full text, hardware/throughput details) — https://arxiv.org/html/2312.11514v3 (accessed 2026-08-11)
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU — https://arxiv.org/abs/2312.12456 (accessed 2026-08-11)
• PowerInfer (SOSP '24 proceedings) — https://dl.acm.org/doi/10.1145/3694715.3695964 (accessed 2026-08-11)
• PowerInfer-2: Fast Large Language Model Inference on a Smartphone — https://arxiv.org/abs/2406.06282 (accessed 2026-08-11)
• PowerInfer-2 project page — https://powerinfer.ai/v2/ (accessed 2026-08-11)
• M2Cache: Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching — https://arxiv.org/abs/2410.14740 (accessed 2026-08-11)
• SolidAttention: Low-Latency SSD-based Serving on Memory-Constrained PCs (FAST '26) — https://www.usenix.org/system/files/fast26-zheng.pdf (accessed 2026-08-11)
• InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference — https://arxiv.org/pdf/2409.04992 (accessed 2026-08-11)
• Swarm: Co-Activation Aware KVCache Offloading Across Multiple SSDs — https://arxiv.org/html/2603.17803v1 (accessed 2026-08-11)
• FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference — https://arxiv.org/abs/2503.03777 (accessed 2026-08-11)
• Glinthawk: A Two-Tiered Architecture for Offline LLM Inference — https://arxiv.org/pdf/2501.11779 (accessed 2026-08-11)
• Fast Inference of Mixture-of-Experts Language Models with Offloading (Eliseev & Mazur) — https://arxiv.org/pdf/2312.17238 (accessed 2026-08-11)
• In-Depth Analysis on Caching and Pre-Fetching in Mixture of Experts Offloading — https://arxiv.org/pdf/2511.05814 (accessed 2026-08-11)
• Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference — https://arxiv.org/pdf/2412.00099 (accessed 2026-08-11)
• MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts — https://arxiv.org/pdf/2510.12357 (accessed 2026-08-11)
• Petals: Run LLMs at home, BitTorrent-style — https://github.com/bigscience-workshop/petals (accessed 2026-08-11)
• Petals project page — https://petals.dev/ (accessed 2026-08-11)
• AirLLM and "70B on a 4GB GPU" — What's Actually Going On? — https://rohit-shirke.medium.com/airllm-and-70b-on-a-4gb-gpu-whats-actually-going-on-3bf0e102252e (accessed 2026-08-11)
• llama.cpp: Should use mmap for model loading (issue #91) — https://github.com/ggml-org/llama.cpp/issues/91 (accessed 2026-08-11)
• llama.cpp: Memory-mapping weights while loading the model (discussion #9999) — https://github.com/ggml-org/llama.cpp/discussions/9999 (accessed 2026-08-11)
• llama.cpp: Mmap faster than direct I/O for MoE models (discussion #18758, incl. M5 Pro/AP1024Z expert-layout measurements) — https://github.com/ggml-org/llama.cpp/discussions/18758 (accessed 2026-08-11)
• llama.cpp: Share readonly GPU model weights across processes — Metal reads mmap buffers via MTLResourceStorageModeShared (discussion #21223) — https://github.com/ggml-org/llama.cpp/discussions/21223 (accessed 2026-08-11)
• llama.cpp: Two-tier GPU+RAM expert cache for MoE offload, pluggable eviction (issue #20757) — https://github.com/ggml-org/llama.cpp/issues/20757 (accessed 2026-08-11)
• llama.cpp: Avoid memcpy for mmap-ed weights on Unified Memory architectures (issue #21827) — https://github.com/ggml-org/llama.cpp/issues/21827 (accessed 2026-08-11)
• Performant local mixture-of-experts CPU inference with GPU acceleration in llama.cpp (HF blog) — https://huggingface.co/blog/Doctor-Shotgun/llamacpp-moe-offload-guide (accessed 2026-08-11)
• MLX Unified Memory documentation — https://ml-explore.github.io/mlx/build/html/usage/unified_memory.html (accessed 2026-08-11)
• MLX: Loading models with mmap (discussion #615, incl. 70GB-on-64GB 0.025 tok/s prototype result) — https://github.com/ml-explore/mlx/discussions/615 (accessed 2026-08-11)
• mlx-swift wired-memory documentation (residency/wired limit) — https://github.com/ml-explore/mlx-swift/blob/main/Source/MLX/Documentation.docc/Articles/wired-memory.md (accessed 2026-08-11)
• mlx-lm: mlx_lm.server causes macOS kernel panic (IOGPUMemory) via unbounded wired growth (issue #883) — https://github.com/ml-explore/mlx-lm/issues/883 (accessed 2026-08-11)
• fcntl F_NOCACHE option behavior (Apple Developer Forums thread 25464) — https://developer.apple.com/forums/thread/25464 (accessed 2026-08-11)
• OSX fcntl(fd, F_NOCACHE, 1) not equivalent to O_DIRECT on Linux (fio issue #48) — https://github.com/axboe/fio/issues/48 (accessed 2026-08-11)
• ronomon/direct-io: Direct IO helpers for FreeBSD, Linux, macOS, Windows (F_NOCACHE alignment notes) — https://github.com/ronomon/direct-io (accessed 2026-08-11)
• makeBuffer(bytesNoCopy:length:options:deallocator:) — Apple Developer Documentation — https://developer.apple.com/documentation/metal/mtldevice/makebuffer(bytesnocopy:length:options:deallocator:) (accessed 2026-08-11)
• MTLStorageMode.shared — Apple Developer Documentation — https://developer.apple.com/documentation/metal/mtlstoragemode/shared (accessed 2026-08-11)
• MTLHeap (incl. setPurgeableState) — Apple Developer Documentation — https://developer.apple.com/documentation/metal/mtlheap (accessed 2026-08-11)
• newBufferWithBytesNoCopy pointer alignment requirement (Apple Developer Forums thread 8011) — https://developer.apple.com/forums/thread/8011 (accessed 2026-08-11)
• iOS/macOS writeback behavior for mmap(MAP_SHARED) dirty pages (Apple Developer Forums thread 763058) — https://developer.apple.com/forums/thread/763058 (accessed 2026-08-11)
• How to Increase VRAM Allocation on Apple Silicon Mac (iogpu.wired_limit_mb) — https://osxdaily.com/2025/05/07/how-to-increase-vram-allocation-on-apple-silicon-mac/ (accessed 2026-08-11)
• Adjust wired limits to allocate more memory to the GPU with Apple Silicon (gist) — https://gist.github.com/havenwood/f2f5c49c2c90c6787ae2295e9805adbe (accessed 2026-08-11)
• Disk speed testing on Apple Silicon: AmorphousDiskMark, Blackmagic, etc. (MacRumors, 4K QD1 results) — https://forums.macrumors.com/threads/disk-speed-testing-on-apple-silicon-amorphousdiskmark-blackmagic-etc-merged.2378298/ (accessed 2026-08-11)
• M1 Pro SSD speeds (MacRumors, 4K QD1 ~32 MB/s report) — https://forums.macrumors.com/threads/m1-pro-ssd-speeds.2319853/ (accessed 2026-08-11)
• MacBook Pro (16-inch, M5 Pro or M5 Max) — Tech Specs (memory bandwidth) — https://support.apple.com/en-us/126319 (accessed 2026-08-11)
• Unified Buffer Cache (UBC) — Mac OS X Internals: A Systems Approach (excerpt) — https://flylib.com/books/en/3.126.1.93/1/ (accessed 2026-08-11)
• Apple XNU WKdm fast memory page compressor (source mirror) — https://github.com/berkus/wkdm (accessed 2026-08-11)
• Virtual memory compression (WKdm background) — https://en.wikipedia.org/wiki/Virtual_memory_compression (accessed 2026-08-11)
• The working set model for program behavior (Denning, 1968; publications index) — http://denninginstitute.com/pjd/PUBS/Workingsets.html (accessed 2026-08-11)
• Working Set Analytics (Denning, ACM Computing Surveys) — https://dl.acm.org/doi/10.1145/3399709 (accessed 2026-08-11)
• ARC: A Self-Tuning, Low Overhead Replacement Cache (Megiddo & Modha, FAST '03) — https://www.usenix.org/legacy/events/fast03/tech/full_papers/megiddo/megiddo.pdf (accessed 2026-08-11)
• An Evaluation of Buffer Management Strategies for Relational Database Systems (Chou & DeWitt, VLDB '85 — DBMIN/QLSM) — https://www.cs.cmu.edu/~natassa/courses/15-721/papers/P127.PDF (accessed 2026-08-11)
• Anti-Caching: A New Approach to Database Management System Architecture (DeBrabant et al., VLDB 2013) — https://www.vldb.org/pvldb/vol6/p1942-debrabant.pdf (accessed 2026-08-11)
• TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory (ASPLOS '23) — https://arxiv.org/abs/2206.02878 (accessed 2026-08-11)
• Pythia: A Customizable Hardware Prefetching Framework Using Online Reinforcement Learning (MICRO 2021) — https://arxiv.org/pdf/2109.12021 (accessed 2026-08-11)
• Evolution of Buffer Management in Database Systems: From Classical Algorithms to Machine Learning and Disaggregated Memory (survey) — https://arxiv.org/pdf/2512.22995 (accessed 2026-08-11)

Decomposition & progressive computation (§4.5–4.6)#

From research/notes/decomposition_progressive.md (62 sources).

• SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression (ICLR 2025) — https://arxiv.org/html/2403.07378v3 (accessed 2026-08-11)
• SVD-LLM (ICLR 2025 proceedings abstract) — https://proceedings.iclr.cc/paper_files/paper/2025/hash/3104e1ab39875cf54fe1eb4473e7c5a1-Abstract-Conference.html (accessed 2026-08-11)
• SVD-LLM GitHub (AIoT-MLSys-Lab) — https://github.com/AIoT-MLSys-Lab/SVD-LLM (accessed 2026-08-11)
• ASVD: Activation-aware Singular Value Decomposition for Compressing LLMs — https://arxiv.org/abs/2312.05821 (accessed 2026-08-11)
• Language model compression with weighted low-rank factorization (FWSVD, ICLR 2022) — https://arxiv.org/abs/2207.00112 (accessed 2026-08-11)
• The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction (LASER, ICLR 2024) — https://arxiv.org/abs/2312.13558 (accessed 2026-08-11)
• LASER project page — https://pratyushasharma.github.io/laser (accessed 2026-08-11)
• SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR 2024) — https://arxiv.org/abs/2401.15024 (accessed 2026-08-11)
• Compressing Large Language Models using Low Rank and Low Precision Decomposition (CALDERA, NeurIPS 2024) — https://arxiv.org/abs/2405.18886 (accessed 2026-08-11)
• CALDERA GitHub (pilancilab) — https://github.com/pilancilab/caldera (accessed 2026-08-11)
• Matrix Compression via Randomized Low Rank and Low Precision Factorization (NeurIPS 2023) — https://neurips.cc/virtual/2023/poster/70291 (accessed 2026-08-11)
• Extreme Compression of Large Language Models via Additive Quantization (AQLM) — https://arxiv.org/html/2401.06118v2 (accessed 2026-08-11)
• AQLM GitHub (incl. ~1-bit Llama-2-7B result) — https://github.com/vahe1994/AQLM (accessed 2026-08-11)
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (ICML 2024) — https://proceedings.mlr.press/v235/tseng24a.html (accessed 2026-08-11)
• GPTVQ: The Blessing of Dimensionality for LLM Quantization — https://arxiv.org/abs/2402.15319 (accessed 2026-08-11)
• VPTQ: Extreme Low-bit Vector Post-Training Quantization for LLMs (Microsoft Research) — https://www.microsoft.com/en-us/research/publication/vptq-extreme-low-bit-vector-post-training-quantization-for-large-language-models (accessed 2026-08-11)
• Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025) — https://arxiv.org/html/2410.20672v1 (accessed 2026-08-11)
• Subformer: Exploring Weight Sharing for Parameter Efficiency (Findings of EMNLP 2021) — https://aclanthology.org/2021.findings-emnlp.344.pdf (accessed 2026-08-11)
• Basis Sharing: Cross-Layer Parameter Sharing for LLM Compression (ICLR 2025) — https://arxiv.org/abs/2410.03765 (accessed 2026-08-11)
• Basis Sharing (ICLR 2025 proceedings PDF) — https://proceedings.iclr.cc/paper_files/paper/2025/file/238c98450b1d9e8055f94d22f303bb57-Paper-Conference.pdf (accessed 2026-08-11)
• DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights — https://arxiv.org/abs/2501.18596 (accessed 2026-08-11)
• ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers (ICASSP 2024) — https://arxiv.org/abs/2310.02489 (accessed 2026-08-11)
• BitDelta: Your Fine-Tune May Only Be Worth One Bit (NeurIPS 2024) — https://arxiv.org/html/2402.10193v3 (accessed 2026-08-11)
• BitDelta NeurIPS poster page — https://neurips.cc/virtual/2024/poster/94736 (accessed 2026-08-11)
• DeltaZip: Compression for Foundation Models (EuroSys 2025; repo lists delta-compression literature) — https://github.com/eth-easl/deltazip (accessed 2026-08-11)
• Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for LLMs (NeurIPS 2024) — https://arxiv.org/abs/2406.08903 (accessed 2026-08-11)
• Kronecker Decomposition for GPT Compression (KnGPT2, ACL 2022) — https://aclanthology.org/2022.acl-short.24.pdf (accessed 2026-08-11)
• TensorGPT: Efficient Compression of LLMs based on Tensor-Train Decomposition — https://arxiv.org/html/2307.00526v2 (accessed 2026-08-11)
• BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025) — https://arxiv.org/abs/2410.23918 (accessed 2026-08-11)
• Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 oral) — https://arxiv.org/pdf/2402.10517 (accessed 2026-08-11)
• Any-Precision LLM GitHub (SNU-ARC) — https://github.com/SNU-ARC/any-precision-llm (accessed 2026-08-11)
• Matryoshka Quantization (MatQuant, ICLR 2025 oral) — https://openreview.net/forum?id=phVWcUSGYP (accessed 2026-08-11)
• Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs — https://arxiv.org/abs/2608.04048 (accessed 2026-08-11)
• Multi-Bitwidth Quantization for LLMs Using Additive Codebooks (Drop-by-Drop) — https://arxiv.org/html/2606.12876v1 (accessed 2026-08-11)
• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning — https://arxiv.org/abs/2012.13255 (accessed 2026-08-11)
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — https://arxiv.org/html/2403.03853v1 (accessed 2026-08-11)
• The Unreasonable Ineffectiveness of the Deeper Layers (ICLR 2025) — https://arxiv.org/abs/2403.17887 (accessed 2026-08-11)
• Your Transformer is Secretly Linear (ACL 2024) — https://arxiv.org/abs/2405.12250 (accessed 2026-08-11)
• Confident Adaptive Language Modeling (CALM, NeurIPS 2022) — https://proceedings.neurips.cc/paper_files/paper/2022/hash/6fac9e316a4ae75ea244ddcef1982c71-Abstract-Conference.html (accessed 2026-08-11)
• Google Research blog: Accelerating text generation with CALM — https://research.google/blog/accelerating-text-generation-with-confident-adaptive-language-modeling-calm (accessed 2026-08-11)
• LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding — https://arxiv.org/html/2404.16710v1 (accessed 2026-08-11)
• Depth-Adaptive Transformer (ICLR 2020) — https://arxiv.org/abs/1910.10073 (accessed 2026-08-11)
• Adaptive Computation Time for Recurrent Neural Networks (Graves 2016) — https://arxiv.org/abs/1603.08983 (accessed 2026-08-11)
• PonderNet: Learning to Ponder — https://arxiv.org/abs/2107.05407 (accessed 2026-08-11)
• Multi-Scale Dense Networks for Resource Efficient Image Classification (MSDNet) — https://arxiv.org/abs/1703.09844 (accessed 2026-08-11)
• Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — https://arxiv.org/abs/2404.02258 (accessed 2026-08-11)
• Multiplying Matrices Without Multiplying (MADDNESS, ICML 2021) — https://proceedings.mlr.press/v139/blalock21a/blalock21a.pdf (accessed 2026-08-11)
• Fast Monte Carlo Algorithms for Matrices I: Approximating Matrix Multiplication (Drineas, Kannan, Mahoney, SIAM J. Comput. 2006) — https://epubs.siam.org/doi/10.1137/S0097539704442684 (accessed 2026-08-11)
• Accelerating the Solution of Linear Systems by Iterative Refinement in Three Precisions (Carson & Higham, SIAM SISC 2018) — https://epubs.siam.org/doi/10.1137/17M1140819 (accessed 2026-08-11)
• Five-precision GMRES-based Iterative Refinement (Amestoy et al.) — https://eprints.maths.manchester.ac.uk/2852/1/paper.pdf (accessed 2026-08-11)
• What Is Iterative Refinement? (Nick Higham) — https://nhigham.com/2023/03/13/what-is-iterative-refinement (accessed 2026-08-11)
• zfp Compression Ratio and Quality (LLNL) — https://computing.llnl.gov/projects/zfp/zfp-compression-ratio-and-quality (accessed 2026-08-11)
• Error Analysis of ZFP Compression for Floating-Point Data (SIAM) — https://epubs.siam.org/doi/10.1137/18M1168832 (accessed 2026-08-11)
• Fast Error-bounded Lossy HPC Data Compression with SZ (Di & Cappello, IPDPS 2016) — https://www.mcs.anl.gov/papers/P5437-1115.pdf (accessed 2026-08-11)
• Embedded zerotrees of wavelet transforms (EZW) — https://en.wikipedia.org/wiki/Embedded_zerotrees_of_wavelet_transforms (accessed 2026-08-11)
• Wavelet and image compression: EZW / SPIHT / JPEG2000-EBCOT lecture notes (Cagnazzo, Télécom Paris) — https://perso.telecom-paristech.fr/tupin/ATHENS/COURSES/wavelet_athens_2012.pdf (accessed 2026-08-11)
• Progressive Meshes (Hoppe, SIGGRAPH 1996) — https://www.cs.jhu.edu/~misha/ReadingSeminar/Papers/Hoppe96.pdf (accessed 2026-08-11)
• Nanite Virtualized Geometry (Unreal Engine documentation) — https://dev.epicgames.com/documentation/unreal-engine/nanite-virtualized-geometry-in-unreal-engine?lang=en-US (accessed 2026-08-11)
• BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data (EuroSys 2013) — https://dl.acm.org/doi/10.1145/2465351.2465355 (accessed 2026-08-11)
• Readings in Database Systems (Red Book) ch. 8: Interactive Analytics — online aggregation & AQP context — http://www.redbook.io/ch8-interactive.html (accessed 2026-08-11)
• Unweight: how we compressed an LLM 22% without sacrificing quality (Cloudflare engineering, bandwidth-bound inference evidence) — https://blog.cloudflare.com/unweight-tensor-compression (accessed 2026-08-11)
• mlx.core.quantize documentation (supported modes, group sizes, bit widths) — https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.quantize.html (accessed 2026-08-11)

Speculation, error analysis, decision stability (§4.7, §4.9–4.10)#

From research/notes/speculation_error_stability.md (66 sources).

• Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman, Matias; ICML 2023) — https://arxiv.org/abs/2211.17192 (accessed 2026-08-11)
• Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., DeepMind) — https://arxiv.org/abs/2302.01318 (accessed 2026-08-11)
• Looking back at speculative decoding (Google Research blog) — https://research.google/blog/looking-back-at-speculative-decoding (accessed 2026-08-11)
• Speculative decoding — Wikipedia — https://en.wikipedia.org/wiki/Speculative_decoding (accessed 2026-08-11)
• Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (Xia et al., EMNLP 2023 Findings) — https://aclanthology.org/2023.findings-emnlp.257.pdf (accessed 2026-08-11)
• Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models — https://arxiv.org/html/2404.14897v1 (accessed 2026-08-11)
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (Cai et al.) — https://arxiv.org/abs/2401.10774 (accessed 2026-08-11)
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — https://arxiv.org/html/2503.01840v1 (accessed 2026-08-11)
• EAGLE-3 (NeurIPS 2025 poster) — https://neurips.cc/virtual/2025/poster/119930 (accessed 2026-08-11)
• Get 3× Faster LLM Inference with Speculative Decoding (BentoML; real-world EAGLE-3 acceptance rates) — https://www.bentoml.com/blog/3x-faster-llm-inference-with-speculative-decoding (accessed 2026-08-11)
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (Fu et al., ICML 2024) — https://arxiv.org/html/2402.02057v1 (accessed 2026-08-11)
• Lookahead decoding blog (LMSYS) — https://www.lmsys.org/blog/2023-11-21-lookahead-decoding (accessed 2026-08-11)
• Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding (Zhang et al.) — https://arxiv.org/abs/2309.08168 (accessed 2026-08-11)
• LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (Elhoushi et al., ACL 2024) — https://arxiv.org/html/2404.16710v1 (accessed 2026-08-11)
• Faster Text Generation with Self-Speculative Decoding (Hugging Face LayerSkip blog) — https://huggingface.co/blog/layerskip (accessed 2026-08-11)
• Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting (NeurIPS 2024) — https://neurips.cc/virtual/2024/poster/93829 (accessed 2026-08-11)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025) — https://arxiv.org/pdf/2410.06916 (accessed 2026-08-11)
• CLaSp: In-Context Layer Skip for Self-Speculative Decoding — https://arxiv.org/html/2505.24196v1 (accessed 2026-08-11)
• QSpec: Speculative Decoding with Complementary Quantization Schemes (EMNLP 2025) — https://aclanthology.org/2025.emnlp-main.240.pdf and https://arxiv.org/abs/2410.11305 (accessed 2026-08-11)
• QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache (Apple ML Research, ICML 2025) — https://machinelearning.apple.com/research/quantspec (accessed 2026-08-11)
• ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts — https://arxiv.org/html/2503.13565v1 (accessed 2026-08-11)
• Speculative Decoding with Big Little Decoder (Kim et al., NeurIPS 2023) — https://arxiv.org/abs/2302.07863 (accessed 2026-08-11)
• BigLittleDecoder repository — https://github.com/kssteven418/biglittledecoder (accessed 2026-08-11)
• SpecInfer: Accelerating LLM Serving with Tree-based Speculative Inference and Verification (ASPLOS 2024) — https://arxiv.org/abs/2305.09781 (accessed 2026-08-11)
• SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices (NeurIPS 2024) — https://arxiv.org/html/2406.02532v1 (accessed 2026-08-11)
• SpecExec results (Together AI blog) — https://www.together.ai/blog/specexec (accessed 2026-08-11)
• Recurrent Drafter for Fast Speculative Decoding in Large Language Models (Apple; MLX/Metal benchmarks) — https://arxiv.org/html/2403.09919v5 and https://machinelearning.apple.com/research/recurrent-drafter (accessed 2026-08-11)
• Speculative Streaming: Fast LLM Inference Without Auxiliary Models (Apple ML Research) — https://machinelearning.apple.com/research/llm-inference (accessed 2026-08-11)
• SPEED: Speculative Pipelined Execution for Efficient Decoding (Hooper et al., NeurIPS-W 2023) — https://arxiv.org/abs/2310.12072 (accessed 2026-08-11)
• LLM-42: Enabling Determinism in LLM Inference with Verified Speculation — https://arxiv.org/html/2601.17768v1 (accessed 2026-08-11)
• FrugalGPT / cascade & routing results summary — https://neuraltrust.ai/blog/llm-model-routing (accessed 2026-08-11)
• Regret Bounds for Model Cascades (survey of FrugalGPT/RouteLLM/Hybrid-LLM numbers) — https://www.tmls.nyc/research/cascade-regret-optimal-stopping (accessed 2026-08-11)
• Confident Adaptive Language Modeling (Schuster et al., NeurIPS 2022) — https://arxiv.org/abs/2207.07061 (PDF: https://www.proceedings.com/content/068/068431-1269open.pdf) (accessed 2026-08-11)
• Accelerating text generation with CALM (Google Research blog) — https://research.google/blog/accelerating-text-generation-with-confident-adaptive-language-modeling-calm (accessed 2026-08-11)
• Consistent Accelerated Inference via Confident Adaptive Transformers (Schuster et al., 2021) — https://neurips2021-nlp.github.io/papers/7/CameraReady/Confident_Early_Exit__Transformer___workshop.pdf (accessed 2026-08-11)
• Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (Liu et al., ICML 2023) — https://proceedings.mlr.press/v202/liu23am/liu23am.pdf (accessed 2026-08-11)
• LLM in a flash: Efficient Large Language Model Inference with Limited Memory (Apple, ACL 2024) — https://arxiv.org/html/2312.11514v2 (accessed 2026-08-11)
• MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading — https://ui.adsabs.harvard.edu/abs/2025arXiv251114102W/abstract (arXiv:2511.14102) (accessed 2026-08-11)
• Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate — https://arxiv.org/html/2502.12224v2 (accessed 2026-08-11)
• Speculating Experts Accelerates Inference for Mixture-of-Experts — https://arxiv.org/html/2603.19289v1 (accessed 2026-08-11)
• SpecMD: A Comprehensive Study on Speculative Expert Prefetching (Apple ML Research) — https://machinelearning.apple.com/research/specmd-expert-prefetching (accessed 2026-08-11)
• The Lipschitz Constant of Self-Attention (Kim, Papamakarios, Mnih; ICML 2021) — https://proceedings.mlr.press/v139/kim21i/kim21i.pdf (accessed 2026-08-11)
• How Smooth Is Attention? (Castin et al.; Apple ML Research) — https://arxiv.org/html/2312.14820v2 and https://machinelearning.apple.com/research/how-smooth-is-attention (accessed 2026-08-11)
• DeepT: Fast and Precise Certification of Transformers (PLDI 2021) — https://files.sri.inf.ethz.ch/website/papers/pldi21-transformers.pdf (accessed 2026-08-11)
• auto_LiRPA: Automatic Linear Relaxation based Perturbation Analysis (NeurIPS 2020; library) — https://github.com/Verified-Intelligence/auto_LiRPA (accessed 2026-08-11)
• Towards Tighter LiRPA-based Robustness Certification (COLING 2025; CROWN O(m²n³) complexity discussion) — https://aclanthology.org/2025.coling-main.415.pdf (accessed 2026-08-11)
• Mixed-precision iterative refinement using tensor cores (Haidar, Dongarra et al.; surveys Carson–Higham GMRES-IR guarantees) — https://www.netlib.org/utk/people/JackDongarra/PAPERS/mixed-rs-2020.pdf (accessed 2026-08-11)
• Three-Precision GMRES-Based Iterative Refinement for Least Squares Problems (Carson, Higham, Pranesh) — https://eprints.maths.manchester.ac.uk/2770/1/paper.pdf (accessed 2026-08-11)
• A New Approach to Probabilistic Rounding Error Analysis (Higham & Mary, SIAM SISC 2019) — https://epubs.siam.org/doi/10.1137/18M1226312 (accessed 2026-08-11)
• Stochastic Rounding and Its Probabilistic Backward Error Analysis (Connolly, Higham, Mary, SIAM SISC 2021) — https://epubs.siam.org/doi/10.1137/20M1334796 (accessed 2026-08-11)
• Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization (NeurIPS 2025) — https://arxiv.org/html/2504.09629v3 (accessed 2026-08-11)
• Why Do Some Inputs Break Low-Bit LLM Quantization? (EMNLP 2025) — https://aclanthology.org/2025.emnlp-main.168.pdf (accessed 2026-08-11)
• Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantizations — https://arxiv.org/html/2601.14277v1 (accessed 2026-08-11)
• Accuracy is Not All You Need (Microsoft; flips + KL under compression) — https://arxiv.org/html/2407.09141v1 (accessed 2026-08-11)
• Why accuracy is a misleading metric when evaluating compressed LLMs (flips summary) — https://bdtechtalks.com/2024/08/06/why-accuracy-is-a-misleading-metric-when-evaluating-compressed-llms (accessed 2026-08-11)
• llama.cpp quantizer discussion #23853 (KLD percentiles, "Same top p" ≈ 90.9–91.2%) — https://github.com/ggml-org/llama.cpp/discussions/23853 (accessed 2026-08-11)
• Blind testing different quants (llama.cpp discussion #5962) — https://github.com/ggml-org/llama.cpp/discussions/5962 (accessed 2026-08-11)
• Measuring Model Quantisation Quality with KL Divergence (MLX quant KLD measurements) — https://smcleod.net/2026/04/measuring-model-quantisation-quality-with-kl-divergence (accessed 2026-08-11)
• Eliciting Latent Predictions from Transformers with the Tuned Lens (Belrose et al.; "prediction depth") — https://arxiv.org/html/2303.08112v6 (accessed 2026-08-11)
• Defeating Nondeterminism in LLM Inference (Thinking Machines) — https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference (accessed 2026-08-11)
• Logit-Gap Steering (Palo Alto Networks Unit 42; measured refusal logit gaps) — https://unit42.paloaltonetworks.com/logit-gap-steering-impact (accessed 2026-08-11)
• QuickSilver / Adaptive Matryoshka Quantization (per-token entropy-gated bit-width) — https://arxiv.org/pdf/2506.22396 (accessed 2026-08-11)
• FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization — https://arxiv.org/html/2506.12024v3 (accessed 2026-08-11)
• DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment (NeurIPS 2025) — https://neurips.cc/virtual/2025/poster/115920 (accessed 2026-08-11)
• MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive LLM Inference — https://ui.adsabs.harvard.edu/abs/2026arXiv260220191W/abstract (accessed 2026-08-11)
• Speculative Decoding Papers (curated list, hemingkx) — https://github.com/hemingkx/SpeculativeDecodingPapers (accessed 2026-08-11)