EDUCATIONAL VIDEO
LLM Quantization: Same Model, Fewer Bits
A short animated explainer on how quantization maps higher-precision weights and activations to smaller representations - FP16, BF16, INT8, INT4, GGUF, AWQ, EXL2, and NVFP4 microscaling - and how to choose between them.
What's inside
Eight chapters, each with a spoken caption shown on screen.
The idea
Quantization maps high-precision weights and activations to fewer bits. The model stays the same, but needs less memory and bandwidth, at the cost of some quality and runtime constraints.
The 16-bit baselines
FP16 and BF16 are the 16-bit baselines, about two bytes per parameter. FP16 keeps more mantissa precision; BF16 keeps FP32-like exponent range with fewer mantissa bits. BF16 is not universally more accurate.
INT8
8-bit integers, roughly half the weight storage of 16-bit, generally high quality when calibrated well. A scale and zero-point map the integer back to a real value. Weight-only versus weights-and-activations.
INT4
About a quarter of the raw weight storage, though real files add scales, metadata and overhead. Quality depends strongly on calibration, outliers, layer sensitivity, and runtime kernels.
Formats vs methods
GGUF is a container file format for llama.cpp that can hold many quantization types. AWQ is an activation-aware weight quantization method for GPU serving. EXL2 is an ExLlamaV2 format and runtime with flexible bitrates.
NVFP4 & microscaling
On NVIDIA Blackwell, 4-bit E2M1 FP4 values with microscaling: one FP8 E4M3 scale per 16-value block, plus an FP32 scale per tensor or group. Many small rulers instead of one global ruler.
Decision guide
FP16 or BF16 for maximum compatibility and a quality baseline. INT8 for conservative compression. INT4, AWQ, GPTQ, EXL2 and GGUF variants depending on runtime and hardware. NVFP4 when hardware and kernels support it.
The takeaway
Quantization is a systems choice. The right format depends on your hardware, runtime, and the exact workload you ship.
Key concepts
Weight-only vs W/A
Quantizing just the weights, or weights and activations together. W/A saves more but needs more care.
Scales & zero-points
An integer plus a scale maps back to a real value. Calibration picks the range that keeps error small.
Group & block scales
A small group of values shares one scale. Finer grouping stores more scales but preserves more range.
Microscaling
Many small rulers instead of one global ruler, preserving local dynamic range across a tensor.