AI & ML
LazyWeights: Decode-on-Demand Edge LLMs
arXiv math.IT0 views
arXiv:2611.17108v2 Announce Type: replace-cross Abstract: Memory decides which LLMs an edge device runs, and how fast: capacity on a 4 GB Jetson Nano, bandwidth on an 8 GB Jetson Orin Nano Super. Deployed formats (GPTQ, AWQ, llama.cpp's K- and I-quants) spend equal bits on every integer code of a matrix, although the codes are far from uniform. LazyWeights keeps these formats but tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4% smaller than the best llama.cpp format. A bit-exact 0.21 mm^2 ASIC decoder unit (1 GHz, 7 nm) cuts Orin latency and energy per token by up to 36% and 38% (54% and 44% with prompt-lookup speculation). On the Nano, LazyWeights needs 58-64% less time and 53-57% less energy per token than CPU llama.cpp, and runs a streamed Mistral-7B 6.6x faster at 3.89 bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput; the ASIC decoder unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).
Source:https://arxiv.org/abs/2611.17108PreviousDirectional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine VisionNextCoTTAGE: An Explainable Judge for Text-to-Audio Generation
Related posts
LVMH’s Media Division announces strategic partnership with OpenAI
0OpenAI Blog
Bharat Semi Raises $24M for AI Sensors That Think Where They See
0dev.to
Android 热更新实战:XopHotFix SDK 集成与补丁生成、发布完整教程
0dev.to

WeeklyReview#202646
0dev.to
evgmrf: An R Package For Gaussian Markov Random Field Extreme Value Models
0arXiv stat.CO
Limitations of nonreversible lifts for subgeometric diffusions
0arXiv stat.CO