LLM Quantization Guide: Run 70B Models on Consumer GPUs with GPTQ, AWQ & GGUF

Running a 70-billion parameter language model requires 140 GB of memory in 16-bit precision. That's more than most consumer GPUs can handle, and even enterprise deployments struggle with the costs. Quantization - the art of reducing numerical precision while preserving model quality - has become essential for making large language models practical. Recent breakthroughs have pushed the boundaries from 8-bit to 4-bit, and now even 2-bit precision, achieving compression ratios that seemed impossible just two years ago.

1import numpy as np
2import matplotlib.pyplot as plt
3 
4# Simulate neural network weights (normally distributed)
5np.random.seed(42)
6weights_fp32 = np.random.randn(10000) * 0.5
7 
8# Quantize to int8 (simulated)
9def quantize_int8(weights):
10    scale = (weights.max() - weights.min()) / 255
11    zero_point = -weights.min() / scale
12    quantized = np.round(weights / scale + zero_point).astype(np.int8)
13    # Dequantize back
14    dequantized = (quantized.astype(float) - zero_point) * scale
15    return quantized, dequantized, scale
16 
17quantized, dequantized, scale = quantize_int8(weights_fp32)
18error = weights_fp32 - dequantized
19 
20fig, axes = plt.subplots(2, 2, figsize=(12, 8))
21 
22# Original weights distribution
23ax = axes[0, 0]
24ax.hist(weights_fp32, bins=100, color='C0', alpha=0.8, edgecolor='none')
25ax.set_title('Original Weights (FP32)')
26ax.set_xlabel('Weight Value')
27ax.set_ylabel('Count')
28ax.axvline(x=0, color='white', linestyle='--', alpha=0.5)
29 
30# Quantized weights
31ax = axes[0, 1]
32ax.hist(quantized, bins=50, color='C1', alpha=0.8, edgecolor='none')
33ax.set_title('Quantized Weights (INT8)')
34ax.set_xlabel('Quantized Value')
35ax.set_ylabel('Count')
36 
37# Dequantized vs Original
38ax = axes[1, 0]
39sample = 200
40ax.scatter(weights_fp32[:sample], dequantized[:sample], alpha=0.5, c='C2', s=10)
41ax.plot([-2, 2], [-2, 2], 'w--', alpha=0.5, label='Perfect')
42ax.set_xlabel('Original (FP32)')
43ax.set_ylabel('Dequantized')
44ax.set_title('Reconstruction Quality')
45ax.legend()
46 
47# Quantization error
48ax = axes[1, 1]
49ax.hist(error, bins=100, color='C3', alpha=0.8, edgecolor='none')
50ax.set_title(f'Quantization Error (RMSE: {np.sqrt(np.mean(error**2)):.4f})')
51ax.set_xlabel('Error')
52ax.set_ylabel('Count')
53 
54plt.tight_layout()
55plt.show()
56 
57print(f"Memory: FP32 = {weights_fp32.nbytes/1024:.1f}KB → INT8 = {quantized.nbytes/1024:.1f}KB")
58print(f"Compression ratio: {weights_fp32.nbytes / quantized.nbytes:.1f}x")
59print(f"Mean absolute error: {np.mean(np.abs(error)):.5f}")
60 

---

The Fundamentals of Quantization

What Is Quantization?

Quantization maps high-precision floating-point numbers to lower-precision representations. In the context of neural networks, this typically means converting 32-bit or 16-bit weights to 8-bit, 4-bit, or even lower bit-widths.

The core quantization formula is:

$$Q(w) = \text{round}\left(\frac{w - z}{s}\right) \cdot s + z$$

Where:

• $w$ is the original weight value
• $s$ is the scale factor (determines the step size between quantized values)
• $z$ is the zero-point (offset for asymmetric quantization)
• $Q(w)$ is the quantized-then-dequantized value

Symmetric vs. Asymmetric Quantization

Symmetric quantization assumes weights are centered around zero:

$$Q_{\text{sym}}(w) = \text{round}\left(\frac{w}{s}\right) \cdot s$$

Symmetric quantization, 8-bit

Negative weightsPositive weights

-1270+127

• Scale: s = max(|w|) / 127
• Zero-point: z = 0

Asymmetric quantization allows the zero-point to shift, better capturing distributions that aren't centered at zero:

$$Q_{\text{asym}}(w) = \text{round}\left(\frac{w - w_{\min}}{s}\right) \cdot s + w_{\min}$$

Asymmetric quantization, 8-bit

Full range

0 (w_min)255 (w_max)

• Scale: s = (w_max - w_min) / 255
• Zero-point: z = w_min

Method	Pros	Cons
Symmetric	Simpler computation, faster inference	Wastes precision if distribution is skewed
Asymmetric	Better utilizes full quantization range	Requires storing zero-point, slightly slower

---

Quantization Approaches: PTQ vs. QAT

Post-Training Quantization (PTQ)

PTQ quantizes a pre-trained model without additional training. It's fast and convenient but can suffer accuracy loss, especially at low bit-widths.

Post-training quantization pipeline

1. Pre-trained modelFull precision (FP16/FP32)
2. Calibration datasetRun a small dataset to determine weight and activation ranges
3. Compute scales and zero-pointsScale s and zero-point z per layer
4. Quantized modelINT8/INT4 weights

Quantization-Aware Training (QAT)

QAT simulates quantization during training, allowing the model to adapt to reduced precision. It achieves better accuracy but requires significant computational resources.

Quantization-aware training

Forward pass

1. w_float
2. Quantize
3. w_quant
4. Dequantize
5. w_fake

Backward pass

1. Gradient at w_fake

   ∂L/∂w_fake

2. STE
   Gradient for w_float

   ∂L/∂w ≈ ∂L/∂w_fake

Approach	Training Cost	Accuracy at 4-bit	Use Case
PTQ	None	Good (with calibration)	Quick deployment
QAT	High (full retraining)	Excellent	Production systems

---

Quantization Error Analysis

Understanding quantization error is crucial for designing effective compression schemes. The mean squared error (MSE) from quantization is:

$$\text{MSE} = \mathbb{E}[(w - Q(w))^2]$$

For uniform quantization with step size $\Delta$, the error is bounded by:

$$\text{MSE} \leq \frac{\Delta^2}{12}$$

This assumes the quantization error is uniformly distributed over $[-\Delta/2, \Delta/2]$.

The Outlier Problem

LLM weights often contain outliers - values significantly larger than the typical range. These outliers devastate naive quantization:

Weight distribution with outliers

99% of weightsOutliers

-0.500.52.0

• Outliers are about 0.1% of weights.
• If the scale is set by the outliers (max = 2.0), most weights get very few quantization levels.

Solutions to the outlier problem include:

• Per-channel quantization: Different scales for different output channels
• Mixed-precision: Keep outlier channels in higher precision
• Outlier-aware methods: Explicitly handle outliers (e.g., SmoothQuant, AWQ)

---

QuIP#: State-of-the-Art Extreme Compression

The QuIP# paper from Cornell (2024) achieved a breakthrough in extreme quantization, enabling 2-bit precision with minimal accuracy loss. With 215 citations in under two years, it has become a foundational technique for LLM compression.

The Key Insight: Incoherence

QuIP#'s central innovation is using Hadamard transforms to make weight matrices incoherent before quantization. Incoherence means the weights are "spread out" uniformly, without concentrated outliers.

Incoherence via the Hadamard transform

Before (coherent)

After H× (incoherent)

Before: outliers concentrated on specific entries. After the Hadamard transform: energy spread uniformly across all entries.

The Hadamard Transform

The Hadamard matrix $H_n$ is a square matrix with entries $\pm 1/\sqrt{n}$ satisfying:

$$H_n H_n^T = I$$

For dimension $n = 2^k$, it's constructed recursively:

$$H_2 = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}$$ $$H_{2n} = \frac{1}{\sqrt{2}} \begin{pmatrix} H_n & H_n \\ H_n & -H_n \end{pmatrix}$$

QuIP# applies Hadamard transforms to both rows and columns of weight matrices:

$$W' = H_m W H_n$$

Because $H$ is orthogonal, the transformation is lossless (can be perfectly reversed) and can be fused into adjacent layers with minimal overhead.

"The Hadamard transform spreads the energy of outlier weights across all entries, creating a more uniform distribution that is far easier to quantize accurately." - Tseng et al., QuIP#, 2024

Lattice Codebooks

The second innovation in QuIP# is using lattice codebooks instead of uniform quantization grids. A lattice is a regular geometric arrangement of points that provides optimal quantization for incoherent vectors.

Uniform grid vs. lattice codebook

Uniform 2D grid

1. Evenly spaced pointsWasted space between points

E8 lattice (8D)

1. Optimal sphere packing in 8 dimensionsQuantization points arranged to minimize the average distance to any weight vector

QuIP# uses the E8 lattice for 2-bit quantization. The E8 lattice achieves the optimal packing density in 8 dimensions, meaning quantized weights are, on average, closer to their original values than any other arrangement.

QuIP# Results

Model	Method	Bits	Perplexity (WikiText-2)
Llama-2-70B	FP16	16	3.32
Llama-2-70B	GPTQ	4	3.85
Llama-2-70B	QuIP	4	3.48
Llama-2-70B	QuIP#	2	4.16

QuIP# achieves 2-bit quantization with perplexity within 25% of full precision - a remarkable result that was considered impossible just a few years ago. This translates to 8x compression, reducing a 140 GB model to under 18 GB.

---

KV Cache Compression with GEAR

While weight quantization reduces model storage, KV cache compression addresses a different bottleneck: memory usage during inference. The GEAR paper (2024), with 128 citations, presents an elegant solution.

The KV Cache Problem

During autoregressive generation, transformers cache the key and value tensors from all previous tokens to avoid recomputation. For long sequences, this cache dominates memory usage:

Memory at 128K context (Llama-3)

Model weights70B params × 2 bytes, constant140 GB

KV cacheGrows with sequence length × batch160 GB

GEAR: Quantization + Low-Rank + Sparse Residuals

GEAR (Generative Efficient Attention caching with Residual approximation) combines three compression techniques:

GEAR compression pipeline

1. Original KV cache (FP16)
2. Uniform quantization (INT4)Captures the bulk of the information cheaply
3. Low-rank approximation (SVD)Captures systematic quantization error

   K̂ ≈ U × S × V^T  (top-r singular values)

4. Sparse outlier storageStores only the top-k largest residual entries to handle the remaining outliers

The reconstruction is:

$$\text{KV} \approx \text{Dequant}(Q_{\text{int4}}(\text{KV})) + U \Sigma V^T + S$$

Where:

• $Q_{\text{int4}}$ is 4-bit uniform quantization
• $U \Sigma V^T$ is the low-rank correction (rank 2-4)
• $S$ is the sparse residual (top 1-2% of entries)

GEAR Results

Method	Bits	Memory Reduction	Accuracy Loss
FP16 (baseline)	16	1x	0%
Naive INT4	4	4x	8-15%
KIVI	2	8x	3-5%
GEAR	~3	5-6x	<1%

"GEAR achieves near-lossless compression of the KV cache with 5-6x memory reduction, enabling longer context windows and larger batch sizes without quality degradation." - Kang et al., GEAR, 2024

---

Practical Quantization: Choosing the Right Approach

Decision Tree for Quantization Strategy

Quantization strategy selection

1. What's your constraint?
2. Memory (edge)

   QuIP# (2-bit)

   Accuracy critical

   QAT (8-bit)

   Both

   GPTQ/AWQ (4-bit)

3. for long-context inference
   Add GEAR for the KV cache

Compression Comparison

Technique	Compression	Quality	Complexity	Best For
INT8 PTQ	2x	Excellent	Low	Quick deployment
GPTQ 4-bit	4x	Good	Medium	Consumer GPUs
AWQ 4-bit	4x	Very Good	Medium	Production
QuIP# 2-bit	8x	Good	High	Maximum compression
GEAR (KV)	5-6x	Excellent	Medium	Long context

---

The Mathematics of Optimal Quantization

Rate-Distortion Theory

The fundamental limit of quantization is described by rate-distortion theory. For a source with distribution $p(x)$ and distortion measure $d(x, \hat{x})$, the minimum achievable rate (bits) at distortion $D$ is:

$$R(D) = \min_{p(\hat{x}|x): \mathbb{E}[d(x,\hat{x})] \leq D} I(X; \hat{X})$$

For Gaussian sources with mean squared error distortion:

$$R(D) = \frac{1}{2} \log_2 \frac{\sigma^2}{D}$$

This means each halving of the distortion requires one additional bit per value.

Why Lattices Are Optimal

Vector quantization outperforms scalar quantization because it exploits correlations. For $n$-dimensional vectors, the normalized second moment measures quantization efficiency:

$$G_n = \frac{1}{n} \frac{\mathbb{E}[\|X - Q(X)\|^2]}{V^{2/n}}$$

Where $V$ is the volume of the quantization cell. The E8 lattice achieves:

$$G_{E8} \approx 0.0717$$

Compared to uniform scalar quantization:

$$G_1 = \frac{1}{12} \approx 0.0833$$

This 14% reduction in quantization error per dimension compounds significantly over millions of weights.

---

Future Directions

Emerging Techniques

1. 1-bit quantization: Binary and ternary networks are approaching practical quality for specific tasks

2. Dynamic quantization: Adjusting precision based on input difficulty or layer importance

3. Quantization-aware architecture design: Models designed from scratch for efficient quantization

4. Hardware co-design: Custom accelerators optimized for specific quantization formats

The Path to Sub-2-Bit

Theoretical compression limits

FP16 (baseline)Standard16-bit

INT8 PTQMature8-bit

GPTQ/AWQMainstream4-bit

QuIP#State of the art2-bit

Ternary + entropyResearch1.5-bit

BinaryEmerging1-bit

---

Conclusion

Quantization has transformed from a nice-to-have optimization into a critical enabling technology for large language models. The journey from 16-bit to 2-bit precision - an 8x compression - makes models like Llama-2-70B accessible on consumer hardware.

The key insights from recent research are:

1. Incoherence matters: QuIP# showed that preprocessing weights with Hadamard transforms enables extreme quantization by eliminating outliers

2. Optimal codebooks: Lattice-based quantization (E8 lattice) provides theoretically optimal compression for incoherent weight distributions

3. KV cache is the next frontier: GEAR demonstrates that combining quantization, low-rank, and sparse approximations achieves near-lossless compression

4. Theory guides practice: Rate-distortion theory and optimal transport provide principled frameworks for understanding quantization limits

As models continue to grow - GPT-4 class systems likely exceed 1 trillion parameters - efficient compression becomes ever more critical. The mathematics of quantization, from Hadamard transforms to lattice codebooks, will remain essential knowledge for anyone deploying large language models at scale.

---

This article cites peer-reviewed research from Semantic Scholar, including studies from leading institutions in machine learning and systems research. For complete bibliographic information, see the hyperlinked references throughout the text.