Sync updated files from pasllm source
This commit is contained in:
parent
a9dd9d6e63
commit
9ba0577d0b
2 changed files with 1 additions and 334 deletions
|
|
@ -1435,7 +1435,7 @@ FP32 gauss σ≈3.52563 r=1.000000 slope=1.000000 |slope-1|=0 |b|=0
|
|||
* **SIMD-friendly inner loop (Q42NL/Q43NL):** nibble-unpack → int8→f32 → multiply by $1/7$ → apply **$h(x,c) = (1-c)x + cx|x|$** with per-block $c$ → scale multiply.
|
||||
* **LUT option:** You can replace $f(x)$ or $h(x,c)$ with a small **signed-q** LUT (15 entries) to avoid transcendental ops.
|
||||
* **Scaling:** Absmax per block matched the baselines here. Per-channel/row variants are trivial if your kernels support them.
|
||||
* **Q42NL/Q43NL curve optimization:** Grid search over 255 candidate $c$ values (quantized to int8 range). See `tools/Q43NL_METHODS_GUIDE.md` for gradient and coarse-to-fine optimization methods.
|
||||
* **Curve optimization:** Q42NL uses grid search over 255 candidate $c$ values (quantized to int8 range). Q43NL uses exhaustive search over all valid FP16 values in [-1,1] (~32K candidates excluding NaN/Inf) for optimal quality.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,333 +0,0 @@
|
|||
# Q43NL Optimization Methods
|
||||
|
||||
## Summary
|
||||
|
||||
Three optimization methods are now available in `convert.py` for Q43NL quantization:
|
||||
|
||||
1. **`gradient`** (default) - Adam optimizer, 6x faster, 99.5% quality
|
||||
2. **`coarse_fine`** (new!) - Two-pass grid, 1.5x faster, 99.97% quality
|
||||
3. **`grid`** - Full grid search, slowest, 100% quality (reference)
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Default (Recommended)
|
||||
```python
|
||||
# Uses gradient method with Adam optimizer
|
||||
packed = q43nl(tensor)
|
||||
```
|
||||
|
||||
### Maximum Quality
|
||||
```python
|
||||
# Uses coarse-to-fine for 99.97% of grid quality
|
||||
packed = q43nl(tensor, method="coarse_fine")
|
||||
```
|
||||
|
||||
### Perfect Quality (Slow)
|
||||
```python
|
||||
# Uses full grid search (reference)
|
||||
packed = q43nl(tensor, method="grid")
|
||||
```
|
||||
|
||||
### Tuned for Best Quality
|
||||
```python
|
||||
# Gradient with more iterations and lower LR
|
||||
packed = q43nl(tensor, method="gradient", gd_iterations=20, gd_lr=0.1)
|
||||
# Achieves 99.7% grid quality, still 6x faster!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Detailed Comparison
|
||||
|
||||
| Method | Quality | Speed | Use Case |
|
||||
|--------|---------|-------|----------|
|
||||
| **gradient** (default) | 99.5% | 6x | General use, production default |
|
||||
| **coarse_fine** | 99.97% | 1.5x | Quality-critical applications |
|
||||
| **grid** | 100% | 1x | Benchmarking, reference only |
|
||||
|
||||
### Tested on 32K elements:
|
||||
|
||||
```
|
||||
Method MSE Ratio Time (s) Speedup Quality Stars
|
||||
------------------------------------------------------------
|
||||
gradient 1.0053x 0.018 6.34x ⭐⭐⭐⭐
|
||||
coarse_fine 1.0003x 0.076 1.46x ⭐⭐⭐⭐⭐
|
||||
grid 1.0000x 0.111 1.00x ⭐⭐⭐⭐⭐
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Gradient (v3 - Adam)
|
||||
|
||||
**Algorithm:**
|
||||
1. Try 12 initialization candidates:
|
||||
- Kurtosis-based
|
||||
- Fixed values: 0, ±0.3, ±0.6, ±0.9
|
||||
- Data-driven: mean-based, range-based
|
||||
- Percentile-based: 25th, 75th
|
||||
2. Pick best initialization per group
|
||||
3. Run Adam optimizer (10-20 iterations)
|
||||
4. Line search refinement (7 points)
|
||||
|
||||
**Parameters:**
|
||||
- `gd_iterations=5` (default, fast) → 1.01x ratio
|
||||
- `gd_iterations=10` (balanced) → 1.005x ratio
|
||||
- `gd_iterations=20, gd_lr=0.1` (best) → 1.0026x ratio
|
||||
|
||||
**Pros:**
|
||||
✅ Fastest (6x speedup)
|
||||
✅ Excellent quality (99.5%)
|
||||
✅ Scales well to large models
|
||||
✅ Production-ready
|
||||
|
||||
**Cons:**
|
||||
❌ Slightly worse than coarse-to-fine
|
||||
❌ Needs parameter tuning for best results
|
||||
|
||||
---
|
||||
|
||||
### Coarse-to-Fine (NEW!)
|
||||
|
||||
**Algorithm:**
|
||||
1. **Coarse pass**: Evaluate 17 candidates across full range [-1, 1]
|
||||
2. Find best candidate per group
|
||||
3. **Fine pass**: Evaluate 17 candidates around best coarse result
|
||||
4. Total: ~34 evaluations per group vs 255 for grid
|
||||
|
||||
**Why it works:**
|
||||
- Coarse pass eliminates poor regions quickly
|
||||
- Fine pass refines around optimal region
|
||||
- 93% fewer evaluations than full grid
|
||||
- Almost identical results (99.97% quality!)
|
||||
|
||||
**Pros:**
|
||||
✅ Nearly perfect quality (99.97% of grid)
|
||||
✅ Still faster than grid (1.5x)
|
||||
✅ Deterministic (no randomness)
|
||||
✅ No parameter tuning needed
|
||||
|
||||
**Cons:**
|
||||
❌ Slower than gradient method
|
||||
❌ Sequential fine pass (could be optimized)
|
||||
|
||||
**When to use:**
|
||||
- Quality is absolutely critical
|
||||
- Can't afford even 0.5% quality loss
|
||||
- Model inference quality matters more than conversion speed
|
||||
|
||||
---
|
||||
|
||||
### Grid (Reference)
|
||||
|
||||
**Algorithm:**
|
||||
- Exhaustively evaluate all 255 candidates
|
||||
- Fully vectorized
|
||||
- Deterministic
|
||||
|
||||
**Pros:**
|
||||
✅ Perfect quality (by definition)
|
||||
✅ No parameters to tune
|
||||
✅ Fully vectorized
|
||||
|
||||
**Cons:**
|
||||
❌ Slowest method
|
||||
❌ Wasteful on poor candidates
|
||||
|
||||
**When to use:**
|
||||
- Only for benchmarking
|
||||
- Quality reference
|
||||
- Not recommended for production
|
||||
|
||||
---
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### In Python Code
|
||||
|
||||
```python
|
||||
import torch
|
||||
from convert import q43nl
|
||||
|
||||
# Example tensor (must be divisible by 32)
|
||||
tensor = torch.randn(4096, dtype=torch.float32)
|
||||
|
||||
# Default (fast, good quality)
|
||||
packed_default = q43nl(tensor)
|
||||
|
||||
# Maximum quality
|
||||
packed_quality = q43nl(tensor, method="coarse_fine")
|
||||
|
||||
# Tuned gradient for best balance
|
||||
packed_tuned = q43nl(tensor, method="gradient",
|
||||
gd_iterations=10, gd_lr=0.1)
|
||||
|
||||
# Reference (slow, perfect)
|
||||
packed_ref = q43nl(tensor, method="grid")
|
||||
```
|
||||
|
||||
### From Command Line
|
||||
|
||||
The method is typically controlled internally by `convert.py`. To test different methods, you would modify the `q43nl()` calls in the conversion code.
|
||||
|
||||
---
|
||||
|
||||
## Performance Analysis
|
||||
|
||||
### Speed Breakdown (32K elements, 32 groups)
|
||||
|
||||
**Gradient (Adam):**
|
||||
- Candidate evaluation: ~50 evals (12 inits + 3×10 GD + 7 refine)
|
||||
- Time: 0.018s
|
||||
- Speedup: **6.34x**
|
||||
|
||||
**Coarse-to-Fine:**
|
||||
- Candidate evaluation: ~561 evals (17 coarse + 17×32 fine)
|
||||
- Time: 0.076s
|
||||
- Speedup: **1.46x**
|
||||
|
||||
**Grid:**
|
||||
- Candidate evaluation: 8160 evals (255×32)
|
||||
- Time: 0.111s
|
||||
- Baseline: 1.00x
|
||||
|
||||
### Quality Breakdown
|
||||
|
||||
**MSE Ratios (lower is better):**
|
||||
- Grid: 1.0000 (baseline)
|
||||
- Coarse-to-Fine: 1.0003 (0.03% worse)
|
||||
- Gradient (10 iter): 1.0053 (0.53% worse)
|
||||
- Gradient (5 iter): 1.0100 (1.00% worse)
|
||||
|
||||
---
|
||||
|
||||
## Recommendations by Scenario
|
||||
|
||||
### For Model Conversion (General)
|
||||
**Use:** `gradient` (default)
|
||||
- Fast enough for large models
|
||||
- Good quality for most use cases
|
||||
- Battle-tested
|
||||
|
||||
### For Production Models (Quality Critical)
|
||||
**Use:** `coarse_fine`
|
||||
- Near-perfect quality
|
||||
- Acceptable speed
|
||||
- Deterministic results
|
||||
|
||||
### For Research/Benchmarking
|
||||
**Use:** `grid`
|
||||
- Perfect quality reference
|
||||
- Measure quality loss of other methods
|
||||
|
||||
### For Experimentation
|
||||
**Use:** `gradient` with tuning
|
||||
```python
|
||||
# Try different configurations:
|
||||
gd_iterations=5, gd_lr=0.3 # Fast (default)
|
||||
gd_iterations=10, gd_lr=0.1 # Balanced
|
||||
gd_iterations=20, gd_lr=0.1 # Best quality
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Future Optimizations
|
||||
|
||||
### Coarse-to-Fine Improvements
|
||||
Current fine pass is sequential. Could vectorize:
|
||||
|
||||
```python
|
||||
# Create per-group fine grids
|
||||
C_fine = torch.stack([
|
||||
linspace(best_c[g] - delta, best_c[g] + delta, 17)
|
||||
for g in range(G)
|
||||
]) # [G, 17]
|
||||
|
||||
# Vectorize evaluation like grid search
|
||||
# Potential speedup: 2-3x → competitive with gradient!
|
||||
```
|
||||
|
||||
### Hybrid Method
|
||||
Combine best of both:
|
||||
```python
|
||||
1. Coarse grid (9 candidates)
|
||||
2. Gradient descent from best 3
|
||||
3. Pick absolute best
|
||||
```
|
||||
|
||||
### Learned Predictor
|
||||
Train tiny NN to predict optimal c:
|
||||
```python
|
||||
# Input: [kurtosis, skewness, mean, std]
|
||||
# Output: c estimate
|
||||
# Then: 3-iteration refinement
|
||||
# Potential: 10-100x faster!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
All methods are tested in `tools/tests/q43nl_gradient.py`:
|
||||
|
||||
```bash
|
||||
cd tools/tests
|
||||
python q43nl_gradient.py
|
||||
```
|
||||
|
||||
**Test suite includes:**
|
||||
1. Basic quantization roundtrip
|
||||
2. Various distributions (normal, uniform, heavy-tailed, etc.)
|
||||
3. Large tensor (32K elements)
|
||||
4. Parameter sweep
|
||||
5. Method comparison
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### v3 (Current)
|
||||
- ✅ Adam optimizer for gradient descent
|
||||
- ✅ 12 initialization candidates
|
||||
- ✅ Line search refinement
|
||||
- ✅ **Coarse-to-fine method added**
|
||||
- ✅ Comprehensive documentation
|
||||
|
||||
### v2
|
||||
- ✅ Momentum optimizer
|
||||
- ✅ 8 initialization candidates
|
||||
- ✅ Faster LR decay
|
||||
|
||||
### v1
|
||||
- ✅ 4 initialization candidates
|
||||
- ✅ Central difference gradients
|
||||
- ✅ Adaptive learning rate
|
||||
|
||||
### Original
|
||||
- ❌ Single initialization
|
||||
- ❌ Forward difference only
|
||||
- ❌ Poor quality (1.3-2.5x worse)
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
**For `convert.py` production use:**
|
||||
|
||||
1. **Default:** Keep `gradient` (v3 Adam)
|
||||
- Fast, reliable, good quality
|
||||
- 99.5% of grid quality
|
||||
- 6x speedup
|
||||
|
||||
2. **Quality option:** Add `coarse_fine`
|
||||
- Use when quality is critical
|
||||
- 99.97% of grid quality
|
||||
- Still 1.5x faster than grid
|
||||
|
||||
3. **Reference:** Keep `grid`
|
||||
- For benchmarking only
|
||||
- Perfect quality baseline
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue