Quantization

Introduction

Quantization is the process of mapping a continuous range of amplitude values to a finite set of discrete levels. It is an inherently lossy operation — the only irreversible step in the analog-to-digital conversion chain. While sampling (if done above the Nyquist rate) preserves all information and is perfectly reversible, quantization permanently discards fine amplitude details that fall between the available levels. This lost information manifests as quantization noise, setting a fundamental limit on the fidelity of any digital representation.

Understanding quantization is essential because every digital system — from a 1-bit comparator to a 32-bit floating-point processor — operates on quantized values. The number of quantization levels determines the dynamic range, the noise floor, and ultimately the quality of the digital representation.

Uniform Quantization

In uniform (linear) quantization, the input range $[-V_{max}, V_{max}]$ is divided into $L$ equal intervals, each of width:

$$\Delta = \frac{2V_{max}}{L} = \frac{2V_{max}}{2^B}$$

where $B$ is the number of bits and $L = 2^B$ is the number of levels.

Each input sample is rounded to the nearest quantization level: $$x_q[n] = \Delta \cdot \text{round}\left(\frac{x[n]}{\Delta}\right)$$

Midtread vs. Midrise Quantizers

Midtread quantizer: Has a quantization level at zero. The output can be exactly zero. Used when zero-input should produce zero-output (most common in audio).

Midrise quantizer: Has a quantization boundary (decision threshold) at zero. No output level equals exactly zero. Both types have $L$ levels and produce the same quantization error statistics.

Quantization Error

The quantization error (or quantization noise) is defined as: $$e[n] = x_q[n] - x[n]$$

For a properly-designed uniform quantizer (no clipping), the error is bounded: $$-\frac{\Delta}{2} \leq e[n] \leq \frac{\Delta}{2}$$

Statistical Model

When the input signal is complex relative to the quantizer step size (many levels are actively used), the quantization error can be modeled as:

• Uniformly distributed over $[-\Delta/2, \Delta/2]$
• Uncorrelated with the input signal
• White noise (flat power spectral density)

Under these assumptions: $$\text{Mean}: E[e] = 0$$ $$\text{Variance (Power)}: \sigma_e^2 = \frac{\Delta^2}{12}$$

This statistical model is called the additive white noise model of quantization.

Signal-to-Quantization-Noise Ratio (SQNR)

For a full-scale sinusoidal input $x(t) = V_{max}\sin(\omega t)$:

Signal power: $P_s = V_{max}^2 / 2$

Quantization noise power: $P_e = \Delta^2/12 = (2V_{max}/2^B)^2 / 12 = V_{max}^2/(3 \cdot 2^{2B-2})$

$$\text{SQNR} = \frac{P_s}{P_e} = \frac{V_{max}^2/2}{V_{max}^2/(3 \cdot 2^{2B-2})} = \frac{3}{2} \cdot 2^{2B}$$

In decibels: $$\text{SQNR}_{dB} = 6.02B + 1.76 \text{ dB}$$

The 6 dB per bit rule: Each additional bit of resolution improves the SQNR by approximately 6 dB, which corresponds to doubling the amplitude precision.

SQNR for Non-Sinusoidal Signals

The 6.02B + 1.76 formula assumes a full-scale sinusoid. For other signal types:

• If the signal does not use the full range: SQNR decreases (lower signal power)
• For a signal using only fraction $f$ of full scale: subtract $20\log_{10}(1/f)$ dB from SQNR

Clipping (Overload) Distortion

When the input exceeds the quantizer range ($|x| > V_{max}$), the output saturates at the maximum or minimum level. This clipping creates severe harmonic distortion that is far more objectionable than quantization noise. Proper system design ensures that clipping occurs rarely (e.g., less than 0.01% of the time for audio).

The trade-off: setting $V_{max}$ too high reduces quantization noise (more levels in the active range) but wastes dynamic range. Setting it too low causes frequent clipping.

Dynamic Range

The dynamic range of a B-bit quantizer is the ratio between the largest representable signal and the smallest detectable signal:

$$DR_{dB} \approx 6.02B \text{ dB}$$

For common bit depths:

Human hearing has a dynamic range of about 120 dB, which explains why 24-bit audio (144 dB) is considered more than sufficient for any recording application.

Non-Uniform Quantization

When the signal probability distribution is not uniform (as with speech, which is often quiet and rarely loud), uniform quantization is suboptimal. Non-uniform quantization places more levels where the signal spends more time:

Approach 1: Companding: Compress the signal before uniform quantization, then expand after reconstruction. This effectively creates smaller step sizes for small amplitudes.

Approach 2: Direct non-uniform spacing: Place quantization levels according to the signal PDF using the Lloyd-Max algorithm, which minimizes mean-squared quantization error for a given number of levels.

Dithering

For low-level signals that span only a few quantization levels, the quantization error becomes correlated with the signal (creating audible distortion rather than noise-like artifacts). Adding a small amount of random noise (dither) before quantization decorrelates the error, converting distortion into benign noise:

$$x_{dithered}[n] = x[n] + d[n]$$

where $d[n]$ is noise with amplitude approximately $\Delta/2$. This technique is essential in audio mastering when reducing bit depth (e.g., from 24-bit to 16-bit for CD release).

Oversampling and Noise Shaping

By sampling at $M$ times the Nyquist rate (oversampling factor $M$), the quantization noise power spreads over a wider bandwidth. Since only the baseband portion ($f_s/2M$) is retained after decimation filtering, the effective SQNR improves:

$$\text{SQNR improvement} = 10\log_{10}(M) \text{ dB}$$

Each doubling of oversampling ratio gives 3 dB improvement (half a bit). Sigma-delta ADCs combine high oversampling (64× to 256×) with noise shaping to push quantization noise out of the signal band, achieving high effective resolution with only 1-bit quantization.

Key Takeaways

• Quantization maps continuous amplitudes to discrete levels — the only irreversible step in ADC
• Uniform quantization: step size $\Delta = 2V_{max}/2^B$; error bounded by $\pm\Delta/2$
• Quantization noise power: $\sigma_e^2 = \Delta^2/12$ (uniform distribution model)
• SQNR for full-scale sinusoid: $6.02B + 1.76$ dB — each bit adds ~6 dB
• Clipping occurs when input exceeds the quantizer range — far worse than quantization noise
• Non-uniform quantization (companding) improves performance for non-uniform signal distributions
• Dithering converts quantization distortion into noise; oversampling spreads noise over wider bandwidth