github-actions[bot]andnilbuild
1288e97c42
Co-authored-by: nilbuild <4921183+nilbuild@users.noreply.github.com>
1.3 KiB
1.3 KiB
Post-Training Quantization
Post-training quantization converts finished model weights to lower precision after training is complete, using a small calibration dataset to compute per-layer scale factors that minimize the gap between quantized and original outputs. It is the standard approach for most deployments since it requires no retraining and can be applied to any existing model. The leading tool is NVIDIA ModelOpt, which exports quantized models in formats compatible with all major inference engines.
Visit the following resources to learn more:
- @opensource@AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- @opensource@bitsandbytes
- @opensource@SmoothQuant
- @article@Optimizing LLMs for Performance and Accuracy with Post-Training Quantization
- @article@Post-training quantization
- @article@Reverse-engineering GGUF | Post-Training Quantization