BeRo1985/pasllm
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Sync updated files from pasllm source

Browse source
This commit is contained in:
Benjamin Rosseaux 2025-11-19 18:43:28 +01:00
commit a91bd6a87c
1 changed files with 2 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/quant_4bit_formats.md
View file

@ -4,9 +4,9 @@ This document specifies the **Q40NL**, **Q41NL**, **Q42NL**, and **Q43NL** forma
**Q40NL** is a **4.5 bits/weight** format with **non-linear decode**, designed to improve tail reconstruction while keeping the same storage as classic Q40/Q4_0 and IQ4_NL. **Q40NL** uses **4-bit signed symmetric codes** with a **per-block FP16 scale** and a non-linear decode function that allocates more resolution toward the tails. **Q40NL** is a **drop-in replacement** for Q40/Q4_0, with the same 18-byte block size (32 weights) and no external element LUTs.
**Q41NL** is an **alternative 4.5 bits/weight** format with the same block layout, per-block FP16 scale and kernels as Q40NL (18 bytes per 32 weights; no element LUTs). It differs only in the non-linearity, using \(f_{41}(x)=x\,|x|\) with inverse \(f_{41}^{-1}(y)=\mathrm{sign}(y)\sqrt{|y|}\), which further increases tail emphasis compared to Q40NL’s \(f(x)=\tfrac12(x|x|+x)\). Like Q40NL, it is a drop-in replacement for Q40/Q4_0 and IQ4_NL.
**Q41NL** is an **alternative 4.5 bits/weight** format with the same block layout, per-block FP16 scale and kernels as Q40NL (18 bytes per 32 weights; no element LUTs). It differs only in the non-linearity, using $\(f_{41}(x)=x\,|x|\)$ with inverse $\(f_{41}^{-1}(y)=\mathrm{sign}(y)\sqrt{|y|}\)$, which further increases tail emphasis compared to Q40NL’s $\(f(x)=\tfrac12(x|x|+x)\)$. Like Q40NL, it is a drop-in replacement for Q40/Q4_0 and IQ4_NL.
**Q42NL** extends the Q4*NL family with a **parametric non-linearity**: 18 bytes per 32 weights (16 nibbles + 1 FP8(E5M2) scale + 1 int8 curve parameter **c**). The decode is \(y=(1-c)x+c|x|x\), where **c** is optimized per-block to minimize MSE. This allows each block to adapt its nonlinearity between linear (c=0) and quadratic-tail (c=1), achieving better reconstruction than fixed-curve Q40NL/Q41NL.
**Q42NL** extends the Q4*NL family with a **parametric non-linearity**: 18 bytes per 32 weights (16 nibbles + 1 FP8(E5M2) scale + 1 int8 curve parameter **c**). The decode is $\(y=(1-c)x+c|x|x\)$, where **c** is optimized per-block to minimize MSE. This allows each block to adapt its nonlinearity between linear (c=0) and quadratic-tail (c=1), achieving better reconstruction than fixed-curve Q40NL/Q41NL.
**Q43NL** uses the same parametric approach as Q42NL but stores a **per-block FP16 scale** (instead of FP8) for higher precision: 19 bytes per 32 weights (16 nibbles + 2 FP16 scale + 1 int8 curve **c**). Both Q42NL and Q43NL support three optimization methods for finding optimal **c**: gradient-based (6x faster, 99.5-99.7% quality), coarse-to-fine grid (1.5x faster, 99.97% quality), or full grid search (slowest, 100% quality).