Quantization
Also known as LLM quantization, Model quantization
Definition
Quantization represents model values with lower numerical precision than the original model. It can reduce memory requirements and sometimes improve serving efficiency, with possible effects on output quality and supported operations.
What lower precision changes
Model weights and intermediate values use numerical representations. Quantization stores some of those values with fewer bits, which can reduce memory use and make it possible to serve a model on different hardware.
The exact result depends on the quantization method, the model architecture, the runtime, and the hardware. Some approaches are applied after training. Others incorporate representative data or quantization-aware training to preserve behavior more carefully.
Check quality on real tasks
A small benchmark may show no visible change while a specialized coding or reasoning task reveals one. Compare the original and quantized versions on a held-out evaluation dataset. Include structured output validity, tool use, refusal behavior, and task completion when those outcomes matter.
Also measure loading time, memory use, throughput, and latency under the expected concurrency. A quantized model can be cheaper to serve while producing more retries if quality falls for the target workflow.
Treat the configuration as part of the model
Record the quantization format, runtime, hardware, context length, and generation settings. Without those details, another team cannot reproduce the reported result or tell whether a change came from precision or from serving configuration.
How this relates to Weave
A quantized model can have a different speed, cost, and completion profile from its full-precision counterpart. When evaluating routes with Weave Router, compare task success and complete workflow cost instead of assuming that smaller memory use means a better route.
Explore Router