Positional Encoding (Fourier Features) — Visual Experiments¶
Goal: Understand positional encoding as a bandwidth knob. We watch spectral bias happen (a plain MLP learns low frequencies first and stalls on high ones), then show how Fourier features cure it, how the encoding scale trades blur against noise, why it works (the induced kernel is stationary with a tunable width), and how Mip-NeRF's integrated encoding makes it scale-aware.
What we'll explore¶
| # | Experiment | Question answered |
|---|---|---|
| 1 | Spectral bias in action | Does a plain MLP really learn low frequencies first and stall on high ones? |
| 2 | The bandwidth knob | How does the Fourier scale trade underfit (blur) vs overfit (noise)? |
| 3 | Why it works (kernel) | What does the encoding do to the similarity the network sees? |
| 4 | Integrated PE | How does Mip-NeRF make encoding aware of how big a region a sample covers? |
Linked theory: research/04-positional-encoding.md · Chạy: uv run jupyter lab từ thư mục latent-anything-theory/ (cần các package trong pyproject.toml).
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
import warnings
warnings.filterwarnings('ignore')
np.random.seed(42)
torch.manual_seed(42)
plt.rcParams.update({
'figure.dpi': 110, 'font.size': 11, 'axes.titlesize': 12, 'axes.labelsize': 11,
'figure.facecolor': 'white', 'axes.facecolor': '#f8f9fa', 'axes.grid': True, 'grid.alpha': 0.35,
})
print('Setup complete.')
Setup complete.
A shared toolkit: a 1D target, an MLP, and Fourier features¶
We fit a 1D signal $f(x)$ with both a low- and a high-frequency component. RFF is a random Fourier-feature map $\gamma(x)=[\cos(2\pi Bx),\sin(2\pi Bx)]$ with $B\sim\mathcal N(0,s^2)$ — the scale $s$ is the bandwidth knob. train fits an MLP and can snapshot its prediction during training.
def target(x):
"""low-frequency + high-frequency content."""
return np.sin(2 * np.pi * 3 * x) + 0.5 * np.sin(2 * np.pi * 12 * x)
class RFF:
"""random Fourier features: gamma(x) = [cos(2 pi B x), sin(2 pi B x)], B ~ N(0, s^2)."""
def __init__(self, scale, m=128, seed=0):
g = torch.Generator().manual_seed(seed)
self.B = torch.randn(m, 1, generator=g) * scale
self.out_dim = 2 * m
def __call__(self, x):
proj = 2 * np.pi * x @ self.B.T
return torch.cat([torch.cos(proj), torch.sin(proj)], dim=1)
def make_mlp(in_dim, hidden=128, layers=3):
mods = [nn.Linear(in_dim, hidden), nn.ReLU()]
for _ in range(layers - 1):
mods += [nn.Linear(hidden, hidden), nn.ReLU()]
mods += [nn.Linear(hidden, 1)]
return nn.Sequential(*mods)
def train(model, X, y, steps=3000, lr=2e-3, snaps=(), Xeval=None, seed=0):
torch.manual_seed(seed)
opt = torch.optim.Adam(model.parameters(), lr=lr)
snapshots = {}
for step in range(steps + 1):
if step in snaps and Xeval is not None:
with torch.no_grad():
snapshots[step] = model(Xeval).numpy().ravel()
opt.zero_grad()
loss = F.mse_loss(model(X), y)
loss.backward(); opt.step()
return snapshots
print('target has a frequency-3 and a frequency-12 component.')
target has a frequency-3 and a frequency-12 component.
Experiment 1: Spectral bias in action¶
Question: The theory says a plain coordinate MLP suffers from spectral bias — it fits low frequencies first and is very slow on high ones. If we snapshot a plain MLP's reconstruction during training, do we see the low-frequency part appear first and the high-frequency wiggles lag far behind?
We fit a plain ReLU MLP (no encoding) to the two-frequency target and capture its prediction at several training steps.
np.random.seed(1); torch.manual_seed(1)
xs = np.linspace(0, 1, 200)
ys = target(xs)
X = torch.tensor(xs[:, None], dtype=torch.float32)
y = torch.tensor(ys[:, None], dtype=torch.float32)
plain = make_mlp(in_dim=1, hidden=128, layers=3)
snaps = train(plain, X, y, steps=4000, lr=2e-3, snaps=(50, 300, 1000, 4000), Xeval=X, seed=1)
print('captured snapshots at steps:', list(snaps.keys()))
captured snapshots at steps: [50, 300, 1000, 4000]
fig, axes = plt.subplots(1, 4, figsize=(16, 4), sharey=True)
fig.suptitle('Experiment 1: a plain MLP fits the low frequency first; the high frequency lags',
fontsize=12.5, fontweight='bold')
for ax, step in zip(axes, [50, 300, 1000, 4000]):
ax.plot(xs, ys, color='black', lw=2, label='target')
ax.plot(xs, snaps[step], color='#c0392b', lw=2, label='MLP')
ax.set_title(f'step {step}'); ax.set_xlabel('x'); ax.legend(fontsize=8)
axes[0].set_ylabel('f(x)')
plt.tight_layout()
plt.show()
What you see¶
What the plot shows: Early in training the plain MLP captures only the slow, frequency-3 wave; the fast frequency-12 ripples are missing. They fill in slowly over thousands of steps, and even at step 4000 the high-frequency detail is weaker than the target.
Why it looks this way: A ReLU MLP on raw coordinates corresponds to a kernel whose spectrum decays fast, so high-frequency modes are learned far more slowly than low-frequency ones — spectral bias. The network is not incapable of high frequencies, it is just heavily biased against them within a practical training budget.
Key takeaway: Spectral bias is a dynamics phenomenon you can watch: low first, high (maybe) much later. Positional encoding changes the kernel so high frequencies are learned on equal footing — the rest of this notebook.
Experiment 2: The bandwidth knob¶
Question: Fourier features fix spectral bias, but their scale $s$ (the spread of the random frequencies) must be chosen. What happens at a scale that is too low, about right, or too high?
To isolate the encoding from the network, we fit a plain linear model on the Fourier features — this is kernel ridge regression with the induced kernel of Experiment 3, with no MLP nonlinearity to mask the effect. We fit from a sparse set of points at three scales and evaluate densely to expose interpolation behaviour.
np.random.seed(2)
xtr = np.linspace(0, 1, 60) # sparse training points
ytr = target(xtr)
xde = np.linspace(0, 1, 600) # dense evaluation
def rff_np(x, s, m=128, seed=0):
B = np.random.default_rng(seed).standard_normal(m) * s
p = 2 * np.pi * np.outer(x, B)
return np.concatenate([np.cos(p), np.sin(p)], axis=1)
scales = [1.0, 5.0, 30.0]
lam = 1e-4 # tiny ridge for numerical stability
fits, train_err = {}, {}
for s in scales:
Phi = rff_np(xtr, s)
w = np.linalg.solve(Phi.T @ Phi + lam * np.eye(Phi.shape[1]), Phi.T @ ytr)
fits[s] = rff_np(xde, s) @ w
train_err[s] = np.sqrt(np.mean((Phi @ w - ytr) ** 2))
interp = np.sqrt(np.mean((fits[s] - target(xde)) ** 2))
print(f'scale s={s:4.0f}: train RMSE={train_err[s]:.3f} interpolation RMSE={interp:.3f}')
scale s= 1: train RMSE=0.350 interpolation RMSE=0.350 scale s= 5: train RMSE=0.005 interpolation RMSE=0.006 scale s= 30: train RMSE=0.000 interpolation RMSE=0.212
fig, axes = plt.subplots(1, 3, figsize=(16, 4.4), sharey=True)
fig.suptitle('Experiment 2: the Fourier scale is a bandwidth knob — too low blurs, too high overfits',
fontsize=12.5, fontweight='bold')
labels = ['s=1 (too low: underfits)', 's=5 (about right)', 's=30 (too high: overfits)']
for ax, s, lab in zip(axes, scales, labels):
ax.plot(xde, target(xde), color='black', lw=2, label='target')
ax.plot(xde, fits[s], color='#2980b9', lw=1.8, label='fit')
ax.plot(xtr, target(xtr), 'o', color='#e67e22', ms=3.5, label='train points')
ax.set_ylim(-2, 2)
ax.set_title(lab); ax.set_xlabel('x'); ax.legend(fontsize=8)
axes[0].set_ylabel('f(x)')
plt.tight_layout()
plt.show()
What you see¶
What the plot shows: At $s=1$ the fit is too smooth to match even the training points — it cannot represent the frequency-12 component (underfit / blurry, large train and interpolation error). At $s=5$ it matches the target almost exactly. At $s=30$ it passes through every training point (train error ≈ 0) but oscillates wildly between them, giving the worst interpolation error.
Why it looks this way: The scale sets which frequencies the features can represent. Too low and the high-frequency basis is simply absent. Too high and the model has frequencies far above the signal, so it interpolates the sparse samples with high-frequency garbage — classic overfitting.
Key takeaway: Positional encoding is not free — its bandwidth must match the signal. Using a linear model makes the knob unmistakable (no network nonlinearity to hide behind). This manual tuning is a real weakness of fixed encodings, motivating the learned features of Instant-NGP.
Experiment 3: Why it works — the induced kernel¶
Question: The theory frames the effect through the kernel the network sees. For Fourier features, the dot product $\langle\gamma(x),\gamma(x')\rangle$ depends only on $x-x'$ — a stationary kernel — and its width is set by the scale. Can we compute it and see the bandwidth?
We evaluate $k(d)=\langle\gamma(x),\gamma(x+d)\rangle$ for several scales (no training needed).
d = np.linspace(-0.5, 0.5, 400)
kernels = {}
for s in [1.0, 5.0, 30.0]:
B = (np.random.default_rng(0).standard_normal(4000) * s)
# <gamma(x), gamma(x+d)> = sum_k cos(2 pi B_k d) (cos·cos + sin·sin)
k = np.mean(np.cos(2 * np.pi * np.outer(d, B)), axis=1)
kernels[s] = k / k.max()
# theoretical width: exp(-2 pi^2 s^2 d^2) -> half-width ~ 1/(2 pi s)
print(f's={s:4.0f}: kernel half-width ≈ {1/(2*np.pi*s):.3f}')
s= 1: kernel half-width ≈ 0.159 s= 5: kernel half-width ≈ 0.032 s= 30: kernel half-width ≈ 0.005
fig, ax = plt.subplots(figsize=(10, 5.2))
colors = {1.0: '#27ae60', 5.0: '#2980b9', 30.0: '#c0392b'}
for s in [1.0, 5.0, 30.0]:
ax.plot(d, kernels[s], color=colors[s], lw=2.2, label=f'scale s={s:.0f}')
ax.set_xlabel('coordinate gap d = x − x′')
ax.set_ylabel('induced similarity k(d) / k(0)')
ax.set_title('Fourier features induce a stationary kernel; larger scale = narrower = higher bandwidth')
ax.legend()
plt.tight_layout()
plt.show()
What you see¶
What the plot shows: Each curve is the similarity the network assigns to two inputs a distance $d$ apart. It depends only on $d$ (stationary), peaks at $d=0$, and gets narrower as the scale grows — $s=30$ considers only very close points similar, $s=1$ treats a wide neighbourhood as similar.
Why it looks this way: Averaging $\cos(2\pi B d)$ over Gaussian frequencies $B\sim\mathcal N(0,s^2)$ gives $\exp(-2\pi^2 s^2 d^2)$ — a Gaussian kernel of width $\sim 1/(2\pi s)$. This is exactly the NTK view: Fourier features turn the MLP's rapidly-decaying kernel into a stationary one whose bandwidth we set with $s$.
Key takeaway: The bandwidth knob from Experiment 2 is this kernel width. A narrow kernel (high $s$) can represent fine detail but generalizes little between points (overfit); a wide kernel (low $s$) is smooth but blurry (underfit). Choosing $s$ = choosing the kernel.
Experiment 4: Integrated positional encoding (Mip-NeRF)¶
Question: Standard PE encodes an infinitesimal point, so it cannot tell whether a sample covers a tiny or a wide region — causing aliasing. Mip-NeRF's integrated PE encodes the expectation over a Gaussian region. The claim: high-frequency components automatically shrink toward zero as the region widens. Can we show that attenuation?
We compute the amplitude of each encoding frequency under integration over a Gaussian of varying width $\sigma_r$: $\mathbb E[\sin(2\pi b x)] \propto \exp(-2\pi^2 b^2 \sigma_r^2)$.
freqs = 2.0 ** np.arange(0, 10) # encoding frequencies (octaves)
region_sigmas = [0.0, 0.02, 0.08, 0.2] # how wide a region the sample covers
atten = {sr: np.exp(-2 * np.pi ** 2 * (freqs ** 2) * sr ** 2) for sr in region_sigmas}
for sr in region_sigmas:
n_alive = int(np.sum(atten[sr] > 0.5))
print(f'region σ={sr:.2f}: {n_alive}/{len(freqs)} frequencies survive (amplitude > 0.5)')
region σ=0.00: 10/10 frequencies survive (amplitude > 0.5) region σ=0.02: 4/10 frequencies survive (amplitude > 0.5) region σ=0.08: 2/10 frequencies survive (amplitude > 0.5) region σ=0.20: 0/10 frequencies survive (amplitude > 0.5)
fig, ax = plt.subplots(figsize=(10, 5.2))
for sr in region_sigmas:
lbl = 'point (standard PE)' if sr == 0.0 else f'region σ={sr}'
ax.plot(np.arange(len(freqs)), atten[sr], 'o-', lw=2, label=lbl)
ax.set_xlabel('encoding frequency index (octave: 1, 2, 4, …, 512)')
ax.set_ylabel('surviving amplitude after integration')
ax.set_title('Integrated PE: the wider the region a sample covers, the more high frequencies vanish')
ax.legend()
plt.tight_layout()
plt.show()
What you see¶
What the plot shows: Standard PE (region $\sigma=0$) keeps every frequency at full amplitude. As the region a sample covers widens, the high-frequency encodings are damped toward zero, while the low ones stay — the wider the region, the fewer surviving frequencies.
Why it looks this way: Averaging a sinusoid over a region wider than its period cancels it out; the closed form is the factor $\exp(-2\pi^2 b^2\sigma_r^2)$, which kills frequency $b$ once its period is small relative to $\sigma_r$. The encoding therefore encodes the scale of the sample.
Key takeaway: Integrated PE makes the network see a low-pass version of the input when it looks at a coarse (far/large) region and full detail when it looks at a fine one — automatic anti-aliasing, the mipmap idea brought into neural fields.
Summary¶
| Experiment | Theme | Key insight |
|---|---|---|
| 1. Spectral bias | The problem | A plain MLP learns low frequencies first and stalls on high ones |
| 2. Bandwidth knob | The trade-off | The Fourier scale sets bandwidth: too low blurs, too high overfits |
| 3. Induced kernel | Why it works | Fourier features give a stationary kernel of width $\sim 1/(2\pi s)$ |
| 4. Integrated PE | Anti-aliasing | Integrating over a region damps high frequencies → scale-aware encoding |
Connection to the broader theory¶
Positional encoding is the fix that makes coordinate networks — Neural Implicit Representation and NeRF — render sharp detail despite spectral bias (Experiment 1). It is best understood as a bandwidth knob (Experiment 2) whose effect is to reshape the network's kernel into a stationary one of controllable width (Experiment 3); Mip-NeRF's integrated version makes that bandwidth depend on the size of the region each sample covers, giving anti-aliasing for free (Experiment 4). The knob is manual, though — the right scale must be guessed per signal — which is exactly the limitation that Instant-NGP removes by learning spatial features on a multiresolution hash grid instead of fixing sinusoids.