What Is Model Quantization?
Model quantization is a compression technique that reduces the precision of a neural network's weights and activations from standard 32-bit or 16-bit floating point to lower-precision formats such as INT8 (8-bit integer) or INT4 (4-bit integer). This can shrink model size by 2-8x and significantly accelerate inference, often with minimal impact on output quality.
Why Quantization Matters for Deployment
Several quantization approaches have emerged. Post-training quantization (PTQ) converts an already-trained model without retraining. Quantization-aware training (QAT) simulates low-precision during training for better accuracy. Advanced methods like GPTQ and AWQ use calibration data to optimize which weights can tolerate lower precision, achieving remarkable compression with negligible quality loss.
Trade-offs and Best Practices
Running large language models at full precision requires expensive GPU hardware with substantial memory. Quantization enables organizations to deploy powerful models on more modest infrastructure, including consumer-grade GPUs or even CPUs. A 70-billion parameter model that normally requires multiple high-end GPUs can run on a single GPU when quantized to 4-bit precision.