AI & ML
Computed Permutation Trellis Coding: Near-Optimal Quantization from Two Multiplications and a 64-Byte Table
arXiv math.IT0 views
arXiv:2611.22019v1 Announce Type: new Abstract: We propose Computed Permutation Trellis Coding (CPTC), a low-bit quantization format for tensors written once and read many times. Formats that round each value on its own, such as INT4 and FP4, waste a large part of their bits against the information-theoretic ideal. Trellis codes recover this loss by encoding values jointly, but prior decoders either need a large lookup table for their best quality or compute values with a hash that restricts what they can decode. CPTC's decoder multiplies its state by two coefficients and adds the top bits of the products to index a 64-byte table. Because that sum is not uniformly distributed, the rarely used ends of a small table can hold values far out in the tails. One decoder serves every bit width from 1 to 6, reproduces the standard integer and floating-point formats exactly, and comes within 0.07 bits of the ideal at 4 bits. It matches a random-permutation trellis code whose table is thousands of times larger, and exceeds every QTIP codebook. In LLM weight quantization without fine-tuning, CPTC has the lowest KL divergence to the unquantized model at every tested bit width on Llama 3.1 8B and 70B Instruct against QTIP's codebooks (by up to 27%). The decoder has been built into the AWS Trainium 4 AI accelerator, where it runs at the throughput of the native FP4 and FP8 formats for a negligible fraction of the systolic array's area. Fused encoding kernels quantize an 8B model's linear layers in under two minutes on one GPU and, in cycle-level simulation, on one Trainium 4 chip.
Source:https://arxiv.org/abs/2611.22019PreviousAn upper bound on non-overlapping codesNextSubsystem Stabilizer $(r,\delta)$-Locally Recoverable Codes: Characterization, Bounds and Constructions
Related posts
AI-Powered Sashiko Introduces Experimental Patch Series Design Assessment
0Phoronix

Preparing for a post-quantum future
0TechRadar

75+ best Black Friday deals 2026 in the UK — GTA 6, Kindles, Switch 2, Shark vacuums, LG OLED TVs, and more
0TechRadar

Why no 40fps mode for GTA 6? — Rockstar says the ‘complexity of simulating and rendering our world’ requires it to ‘carefully balance how we spend every millisecond of both the central processor and t
0TechRadar

'Enterprises need to connect AI to the right data': Report finds over half of firms have abandoned AI projects over unclear goals
0TechRadar

$3.26 million in SpaceX AI data center revenue will be pumped back into the City of Memphis
0Tom's Hardware