Riemannian Geometry — Parallel Transport, Curvature, and Why Lerp Fails¶
Goal: See the two foundations of Riemannian geometry — parallel transport and curvature — on a real curved surface, then watch exactly how and when naive linear interpolation (Lerp) breaks down in a curved latent space, and why Slerp and geodesics repair it.
In flat Euclidean space, carrying a vector "without turning it" is trivial and the straight line is the shortest path. On a curved manifold neither is true: a vector dragged around a closed loop comes back rotated (holonomy), and the straight chord between two points slices off the manifold into regions the model never learned. This notebook builds a sphere from scratch to make curvature tangible, then transfers the lesson to latent-space interpolation.
What we'll explore¶
| # | Experiment | Question answered |
|---|---|---|
| 1 | Parallel transport & holonomy | What happens to a vector dragged around a closed loop on a sphere? |
| 2 | Holonomy = curvature × area | How does the rotation angle encode the curvature enclosed by the loop? |
| 3 | Geodesic vs chord | Why is the "straight line" between two points not the shortest path on a curved space? |
| 4 | Lerp goes off-manifold | When does a straight interpolation cut through an empty, low-density void? |
| 5 | Hypersphere shell & norm decay | Why does a Lerp midpoint lose norm and fall off the data shell in high dimensions? |
| 6 | Lerp vs Slerp in-distribution | Does Slerp actually keep interpolations on the high-density manifold? |
Linked theory: research/04-riemannian-geometry.md · Chạy: uv run jupyter lab từ thư mục latent-anything-theory/ (cần các package trong pyproject.toml).
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from mpl_toolkits.mplot3d import Axes3D # noqa: F401 (enables 3-D projection)
from scipy.stats import gaussian_kde
import warnings
warnings.filterwarnings('ignore')
np.random.seed(42)
plt.rcParams.update({
'figure.dpi': 110,
'font.size': 11,
'axes.titlesize': 12,
'axes.labelsize': 11,
'figure.facecolor': 'white',
'axes.facecolor': '#f8f9fa',
'axes.grid': True,
'grid.alpha': 0.35,
})
print('Setup complete. NumPy:', np.__version__)
Setup complete. NumPy: 2.4.6
A shared toolkit: the sphere, geodesics, transport, and interpolation¶
The unit sphere is the simplest non-flat manifold, so we use it as our laboratory. We need a handful of exact operations, defined once.
- Geodesic / Slerp — the great-circle arc between two points is the sphere's "straight line". The same spherical-linear-interpolation formula $\mathrm{slerp}(a,b,t)=\dfrac{\sin((1-t)\Omega)}{\sin\Omega}\,a+\dfrac{\sin(t\Omega)}{\sin\Omega}\,b$ (with $\Omega$ the angle between the vectors) doubles as our latent-space interpolator later.
- Parallel transport along a geodesic — on the sphere this is exactly the rotation about the great circle's axis that carries the start point to the end point. Composing these rotations around a closed loop gives the holonomy.
- Spherical-triangle area — by Girard's theorem, the area equals the angle excess $(\alpha+\beta+\gamma)-\pi$, which (for $K=1$) is exactly what the holonomy should equal.
def nrm(v):
return v / (np.linalg.norm(v) + 1e-15)
def slerp(a, b, t):
"""Spherical linear interpolation; works for unit points and for general vectors."""
na, nb = np.linalg.norm(a), np.linalg.norm(b)
omega = np.arccos(np.clip((a @ b) / (na * nb + 1e-15), -1, 1))
if omega < 1e-8: # nearly parallel -> plain lerp
return (1 - t) * a + t * b
s = np.sin(omega)
return np.sin((1 - t) * omega) / s * a + np.sin(t * omega) / s * b
def lerp(a, b, t):
return (1 - t) * a + t * b
def rodrigues(axis, theta):
"""Rotation matrix by angle theta about a 3-D axis (Rodrigues' formula)."""
a = nrm(axis)
K = np.array([[0, -a[2], a[1]], [a[2], 0, -a[0]], [-a[1], a[0], 0]])
return np.eye(3) + np.sin(theta) * K + (1 - np.cos(theta)) * (K @ K)
def transport_rotation(p, q):
"""Parallel transport along the minimizing geodesic p->q = rotation about the great-circle axis."""
axis = np.cross(p, q)
if np.linalg.norm(axis) < 1e-12:
return np.eye(3)
theta = np.arccos(np.clip(p @ q, -1, 1))
return rodrigues(axis, theta)
def triangle_area(A, B, C):
"""Area of a spherical triangle on the unit sphere via Girard's angle-excess theorem."""
def vertex_angle(P, Q, R):
tq = nrm(Q - (Q @ P) * P) # tangent at P toward Q
tr = nrm(R - (R @ P) * P) # tangent at P toward R
return np.arccos(np.clip(tq @ tr, -1, 1))
return vertex_angle(A, B, C) + vertex_angle(B, A, C) + vertex_angle(C, A, B) - np.pi
def loop_holonomy(vertices, v0):
"""Net rotation angle of a tangent vector transported around a closed polygon of geodesics."""
R = np.eye(3)
for i in range(len(vertices)):
R = transport_rotation(vertices[i], vertices[(i + 1) % len(vertices)]) @ R
vf = R @ v0
angle = np.arccos(np.clip(nrm(v0) @ nrm(vf), -1, 1))
return angle, vf
# quick self-check: the octant triangle should give a 90-degree holonomy = its area
A = np.array([1., 0, 0]); B = np.array([0, 1., 0]); C = np.array([0, 0, 1.])
ang, _ = loop_holonomy([A, B, C], np.array([0, 1., 0]))
print(f'octant: holonomy = {np.degrees(ang):.1f} deg area = {np.degrees(triangle_area(A, B, C)):.1f} deg')
octant: holonomy = 90.0 deg area = 90.0 deg
What you see¶
The self-check transports a tangent vector around the spherical octant (the triangle joining the three axis points $(1,0,0)$, $(0,1,0)$, $(0,0,1)$) and reports a 90° holonomy that exactly equals the triangle's area. That single line already contains the whole story of curvature: on a flat plane the loop would return the vector unchanged (0°), but the sphere's curvature rotates it by an amount equal to the area enclosed. Every tool above is exact — no discretization — so the experiments that follow measure real geometry, not numerical artefacts.
Experiment 1: Parallel transport and holonomy — a vector that comes back rotated¶
Question: If we hold a tangent vector and slide it around a closed loop on the sphere — keeping it "as parallel as possible" at every step (zero covariant derivative) — does it return to its starting orientation? The research file says no: on a curved surface the vector comes back deflected, a phenomenon called holonomy.
We transport a vector around the spherical octant: from $(1,0,0)$ on the equator, up to the north pole $(0,0,1)$, across to $(0,1,0)$, and back along the equator. Each leg is a geodesic, so transport is the exact great-circle rotation from our toolkit.
np.random.seed(1)
A = np.array([1., 0, 0]); B = np.array([0, 0, 1.]); C = np.array([0, 1., 0]) # octant loop
v0 = nrm(np.array([0, 1., 0]) - (np.array([0, 1., 0]) @ A) * A) # tangent at A
# carry the vector leg by leg, sampling points + transported vectors for the quiver
path_pts, path_vecs = [], []
carried = v0.copy()
for P, Q in [(A, B), (B, C), (C, A)]:
axis = np.cross(P, Q); theta = np.arccos(np.clip(P @ Q, -1, 1))
for t in np.linspace(0, 1, 9)[:-1]:
Rt = rodrigues(axis, t * theta)
path_pts.append(Rt @ P)
path_vecs.append(Rt @ carried)
carried = rodrigues(axis, theta) @ carried # vector at the next corner
v_final = carried
hol_angle = np.degrees(np.arccos(np.clip(nrm(v0) @ nrm(v_final), -1, 1)))
path_pts = np.array(path_pts); path_vecs = np.array(path_vecs)
print(f'holonomy deflection after the closed loop: {hol_angle:.1f} degrees')
holonomy deflection after the closed loop: 90.0 degrees
fig = plt.figure(figsize=(8.5, 7.5))
ax = fig.add_subplot(111, projection='3d')
# faint sphere
u, vv = np.mgrid[0:2 * np.pi:40j, 0:np.pi:20j]
ax.plot_surface(np.cos(u) * np.sin(vv), np.sin(u) * np.sin(vv), np.cos(vv),
color='#d6eaf8', alpha=0.25, linewidth=0, zorder=0)
# the three geodesic edges
for P, Q in [(A, B), (B, C), (C, A)]:
arc = np.array([slerp(P, Q, t) for t in np.linspace(0, 1, 40)])
ax.plot(arc[:, 0], arc[:, 1], arc[:, 2], color='#34495e', lw=2)
# transported vectors along the path
ax.quiver(path_pts[:, 0], path_pts[:, 1], path_pts[:, 2],
path_vecs[:, 0], path_vecs[:, 1], path_vecs[:, 2],
length=0.35, color='#2980b9', normalize=True, label='transported vector')
# start vs final at A, highlighted
ax.quiver(*A, *v0, length=0.5, color='#27ae60', lw=3, normalize=True, label='start vector')
ax.quiver(*A, *v_final, length=0.5, color='#c0392b', lw=3, normalize=True, label='returned vector')
for p, txt in [(A, 'A'), (B, 'pole'), (C, 'C')]:
ax.text(p[0] * 1.1, p[1] * 1.1, p[2] * 1.1, txt, fontsize=11, fontweight='bold')
ax.set_title(f'Parallel transport around a closed loop\nholonomy deflection = {hol_angle:.0f}°',
fontsize=12.5, fontweight='bold')
ax.set_xlabel('x'); ax.set_ylabel('y'); ax.set_zlabel('z')
ax.legend(loc='upper left', fontsize=9)
ax.view_init(elev=22, azim=35)
plt.tight_layout()
plt.show()
What you see¶
What the plot shows: A vector (green) starts tangent to the sphere at point A on the equator. Dragged around the three-legged loop A → pole → C → A, staying parallel at every step (blue arrows), it returns to A pointing in a visibly different direction (red) — deflected by 90°.
Why it looks this way: Parallel transport keeps the vector's angle to the path constant, but the path itself bends with the surface. Because the loop encloses curved territory, those local "no-turn" rules do not cancel globally: the accumulated effect is a net rotation. On a flat plane the same loop would bring the vector back exactly aligned — the deflection is a pure signature of curvature.
Key takeaway: Holonomy is the operational definition of curvature you can actually watch. It also explains why a single global notion of "the same direction" does not exist on a manifold — a fact that, in latent space, is exactly why moving along a straight line can quietly walk you into the wrong neighbourhood.
Experiment 2: Holonomy = curvature × area — measuring $K$ from loops¶
Question: The research file states the deflection angle is proportional to the curvature enclosed by the loop: $\text{holonomy} = \iint_{\text{loop}} K\,dA = K \cdot \text{Area}$. If that is literally true, then plotting holonomy against enclosed metric area should be a straight line whose slope is the curvature $K$ — steeper for a tightly curved (small) sphere, flat (zero) for a plane.
We measure holonomy for a family of spherical triangles of growing size, on a unit sphere ($K=1$) and a larger sphere of radius $R=1.5$ ($K=1/R^2$), and compare against a flat plane ($K=0$).
np.random.seed(2)
pole = np.array([0, 0, 1.])
solid_angles, holo = [], []
for col in np.linspace(0.25, 1.25, 9): # colatitude of the two base vertices
P2 = np.array([np.sin(col), 0, np.cos(col)])
P3 = np.array([np.sin(col) * np.cos(0.95), np.sin(col) * np.sin(0.95), np.cos(col)])
v = nrm(np.cross(pole, np.array([1., 0, 0])))
ang, _ = loop_holonomy([pole, P2, P3], v)
solid_angles.append(triangle_area(pole, P2, P3)) # = enclosed solid angle (unit sphere)
holo.append(ang)
solid_angles, holo = np.array(solid_angles), np.array(holo)
# metric area depends on the sphere radius: A = R^2 * solid_angle ; holonomy stays = solid_angle = K*A
R_big = 1.5
area_unit = solid_angles * 1.0 ** 2 # unit sphere, K = 1
area_big = solid_angles * R_big ** 2 # radius-1.5 sphere, K = 1/R^2
slope_unit = np.polyfit(area_unit, holo, 1)[0]
slope_big = np.polyfit(area_big, holo, 1)[0]
print(f'unit sphere : fitted slope = {slope_unit:.3f} (curvature K = 1.000)')
print(f'R=1.5 sphere: fitted slope = {slope_big:.3f} (curvature K = 1/R^2 = {1 / R_big**2:.3f})')
unit sphere : fitted slope = 1.000 (curvature K = 1.000) R=1.5 sphere: fitted slope = 0.444 (curvature K = 1/R^2 = 0.444)
fig, ax = plt.subplots(figsize=(9.5, 5.8))
ax.plot(area_unit, holo, 'o-', color='#c0392b', lw=2.2, ms=7,
label=f'unit sphere (slope K = {slope_unit:.2f})')
ax.plot(area_big, holo, 's-', color='#2980b9', lw=2.2, ms=7,
label=f'sphere R=1.5 (slope K = {slope_big:.2f})')
ax.plot(area_big, np.zeros_like(area_big), '^--', color='#27ae60', lw=2.2, ms=7,
label='flat plane (K = 0)')
ax.set_xlabel('metric area enclosed by the loop')
ax.set_ylabel('holonomy deflection (radians)')
ax.set_title('Holonomy is linear in enclosed area; the slope IS the curvature')
ax.legend(fontsize=10)
plt.tight_layout()
plt.show()
What you see¶
What the plot shows: Both spheres trace straight lines through the origin — holonomy grows in exact proportion to the area enclosed. The unit sphere's line has slope ≈ 1; the larger $R=1.5$ sphere's line is gentler, slope ≈ 0.44 ≈ $1/1.5^2$. The flat plane stays pinned at zero deflection for every area.
Why it looks this way: The same angular loop encloses more metric area on a bigger sphere ($A = R^2\Omega$), but its intrinsic curvature is correspondingly weaker ($K = 1/R^2$), so the product $K\cdot A = \Omega$ — the deflection — is identical. Plotting against metric area therefore exposes the curvature directly as the slope. The plane has $K=0$, so no area ever produces a deflection.
Key takeaway: Curvature is not an abstract tensor entry here — it is a measurable exchange rate between area swept and angle gained. This is the Riemann curvature tensor's scalar shadow, and it is the quantity that decides whether a region of latent space is "flat enough" to interpolate through naively or curved enough to demand a geodesic.
Experiment 3: Geodesic vs straight chord — the shortest path bends¶
Question: On a curved manifold the genuine shortest path is a geodesic, not the Euclidean straight line. For two points on a sphere, how different are the great-circle geodesic and the straight chord through the interior — and what happens to the chord's norm as it cuts inside?
This is the exact bridge to latent interpolation: the chord is what Lerp computes, the great circle is what Slerp computes.
np.random.seed(3)
p = nrm(np.array([1., 0.2, 0.3]))
q = nrm(np.array([-0.3, 1., 0.4]))
omega = np.arccos(np.clip(p @ q, -1, 1))
ts = np.linspace(0, 1, 60)
geo = np.array([slerp(p, q, t) for t in ts]) # on the sphere (norm 1)
chord = np.array([lerp(p, q, t) for t in ts]) # straight line through the interior
geo_norm = np.linalg.norm(geo, axis=1)
chord_norm = np.linalg.norm(chord, axis=1)
arc_len = omega # great-circle length on unit sphere
chord_len = np.linalg.norm(q - p)
print(f'angle between points : {np.degrees(omega):.1f} deg')
print(f'geodesic (great-circle) len : {arc_len:.3f}')
print(f'straight chord length : {chord_len:.3f}')
print(f'chord norm at midpoint : {chord_norm[30]:.3f} (geodesic stays at 1.000)')
angle between points : 89.0 deg geodesic (great-circle) len : 1.554 straight chord length : 1.402 chord norm at midpoint : 0.713 (geodesic stays at 1.000)
fig = plt.figure(figsize=(14, 5.6))
# --- left: 3-D geodesic vs chord ---
ax = fig.add_subplot(121, projection='3d')
u, vv = np.mgrid[0:2 * np.pi:40j, 0:np.pi:20j]
ax.plot_surface(np.cos(u) * np.sin(vv), np.sin(u) * np.sin(vv), np.cos(vv),
color='#d6eaf8', alpha=0.22, linewidth=0)
ax.plot(geo[:, 0], geo[:, 1], geo[:, 2], color='#27ae60', lw=3, label='geodesic (Slerp)')
ax.plot(chord[:, 0], chord[:, 1], chord[:, 2], color='#c0392b', lw=3, ls='--',
label='straight chord (Lerp)')
ax.scatter(*p, color='black', s=40); ax.scatter(*q, color='black', s=40)
ax.scatter(*chord[30], color='#c0392b', s=60, label='chord midpoint (sunk inside)')
ax.set_title('Geodesic hugs the surface; the chord dives through the interior')
ax.set_xlabel('x'); ax.set_ylabel('y'); ax.set_zlabel('z')
ax.legend(fontsize=8.5, loc='upper left'); ax.view_init(elev=20, azim=40)
# --- right: norm along the path ---
ax2 = fig.add_subplot(122)
ax2.plot(ts, geo_norm, color='#27ae60', lw=2.4, label='geodesic / Slerp norm')
ax2.plot(ts, chord_norm, color='#c0392b', lw=2.4, ls='--', label='chord / Lerp norm')
ax2.axvline(0.5, color='gray', ls=':', lw=1)
ax2.set_xlabel('interpolation parameter t')
ax2.set_ylabel('vector norm')
ax2.set_title('The chord loses norm, deepest at the midpoint')
ax2.legend()
plt.tight_layout()
plt.show()
What you see¶
What the plot shows: The green geodesic rides along the sphere's surface from $p$ to $q$; the red dashed chord is the straight Euclidean line, which plunges inside the sphere. The norm plot on the right makes the cost explicit: the geodesic keeps norm exactly 1, while the chord's norm sags to a minimum right at $t=0.5$.
Why it looks this way: A straight line between two surface points is a chord, and every chord of a sphere passes through points closer to the centre than the surface — the midpoint norm is $\cos(\Omega/2)<1$. The geodesic, by construction, never leaves the manifold, so it preserves the norm that defines membership on the sphere.
Key takeaway: "Straight" is a property of the ambient space, not the manifold. The shortest on-manifold path is the geodesic, and the gap between chord and geodesic — especially the midpoint norm collapse — is precisely the failure mode that the next experiments show wrecking latent-space interpolation.
Experiment 4: Lerp goes off-manifold — straight lines cross the void¶
Question: The research file's first Lerp failure: a latent manifold is a curved sheet that hugs the data distribution, so a straight interpolation between two distant points can cut clean through an off-manifold void — a low-density region the decoder never learned, producing blurry, distorted samples. Can we see the straight path leave the data and a manifold-following path stay on it?
We build a strongly curved 2-D data manifold (a crescent) and compare the Lerp chord with a path that follows the manifold's arc.
np.random.seed(4)
rng4 = np.random.default_rng(4)
# crescent-shaped data manifold: points on an arc of radius 4 with a little normal noise
n = 1500
phi = rng4.uniform(0.15 * np.pi, 0.85 * np.pi, n)
radius = 4.0 + 0.18 * rng4.standard_normal(n)
data = np.column_stack([radius * np.cos(phi), radius * np.sin(phi)])
# two endpoints near the tips of the crescent
z1 = np.array([4.0 * np.cos(0.20 * np.pi), 4.0 * np.sin(0.20 * np.pi)])
z2 = np.array([4.0 * np.cos(0.80 * np.pi), 4.0 * np.sin(0.80 * np.pi)])
ts = np.linspace(0, 1, 50)
lerp_path = np.array([lerp(z1, z2, t) for t in ts])
# a manifold-following path: interpolate the ANGLE and keep radius on the arc
phi1, phi2 = 0.20 * np.pi, 0.80 * np.pi
arc_path = np.array([[4.0 * np.cos(p), 4.0 * np.sin(p)]
for p in (1 - ts) * phi1 + ts * phi2])
kde = gaussian_kde(data.T)
log_dens_lerp = kde.logpdf(lerp_path.T)
log_dens_arc = kde.logpdf(arc_path.T)
print(f'min log-density along Lerp chord : {log_dens_lerp.min():.1f}')
print(f'min log-density along arc path : {log_dens_arc.min():.1f}')
print(f'Lerp midpoint sits {log_dens_arc.min() - log_dens_lerp[25]:.1f} log-units below the manifold')
min log-density along Lerp chord : -11.5 min log-density along arc path : -2.6 Lerp midpoint sits 8.9 log-units below the manifold
fig, axes = plt.subplots(1, 2, figsize=(14.5, 5.8))
fig.suptitle('Experiment 4: a straight Lerp slices across the off-manifold void',
fontsize=12.5, fontweight='bold')
# --- left: density field + the two paths ---
ax = axes[0]
gx, gy = np.mgrid[-5:5:120j, -1:5:120j]
grid = np.vstack([gx.ravel(), gy.ravel()])
dens = np.exp(kde.logpdf(grid)).reshape(gx.shape)
cf = ax.contourf(gx, gy, dens, levels=14, cmap='viridis')
ax.scatter(data[:, 0], data[:, 1], s=4, color='white', alpha=0.25)
ax.plot(lerp_path[:, 0], lerp_path[:, 1], color='#e74c3c', lw=2.6, ls='--', label='Lerp (straight)')
ax.plot(arc_path[:, 0], arc_path[:, 1], color='#f1c40f', lw=2.6, label='manifold path')
ax.scatter(*lerp_path[25], color='#e74c3c', s=80, zorder=5, edgecolor='black')
ax.scatter([z1[0], z2[0]], [z1[1], z2[1]], color='black', s=50, zorder=5)
ax.set_xlabel('latent dim 1'); ax.set_ylabel('latent dim 2')
ax.set_title('Lerp midpoint lands in the dark, zero-density gap')
ax.legend(loc='lower center', fontsize=9)
fig.colorbar(cf, ax=ax, fraction=0.046, pad=0.04, label='data density')
# --- right: density along the interpolation ---
ax2 = axes[1]
ax2.plot(ts, log_dens_lerp, color='#e74c3c', lw=2.4, ls='--', label='Lerp (straight)')
ax2.plot(ts, log_dens_arc, color='#f1c40f', lw=2.4, label='manifold path')
ax2.axvline(0.5, color='gray', ls=':', lw=1)
ax2.set_xlabel('interpolation parameter t')
ax2.set_ylabel('log data density')
ax2.set_title('Lerp plunges into low density; the manifold path stays high')
ax2.legend()
plt.tight_layout()
plt.show()
What you see¶
What the plot shows: The bright crescent is where the data (and thus the decoder's competence) lives; the dark interior is empty. The red dashed Lerp chord shoots straight across that dark gap, with its midpoint marooned in near-zero density, while the yellow manifold-following path stays on the bright ridge the whole way. The density curve on the right shows the Lerp path crashing to a deep minimum at $t=0.5$.
Why it looks this way: The manifold is curved, so the straight line connecting its tips does not stay inside it — it short-cuts through the concave void. Points sampled there are out of distribution: the decoder was never trained on them, so it produces blurry, malformed outputs. The arc path respects the curvature and never leaves the supported region.
Key takeaway: Lerp implicitly assumes the latent space is flat and convex. When the true manifold curves, that assumption sends interpolations through territory the model cannot render — the geometric root cause of "muddy in-between" samples, and the motivation for geodesic interpolation that bends around the void.
Experiment 5: The hypersphere shell and norm degradation¶
Question: The research file's second Lerp failure is subtler and purely high-dimensional. Under an isotropic Gaussian prior $\mathcal{N}(0, I)$, concentration of measure pins almost all the probability mass onto a thin shell at radius $\approx\sqrt{D}$ — not near the origin. A Lerp chord between two shell points dives toward the empty core, and its norm is crushed most at the midpoint (the "tent-pole" effect). Does Slerp avoid this?
We sample Gaussian endpoints in several dimensions and track the norm of Lerp vs Slerp along $t$, against the shell radius $\sqrt{D}$.
np.random.seed(5)
rng5 = np.random.default_rng(5)
dims = [2, 10, 100]
ts = np.linspace(0, 1, 50)
lerp_curves, slerp_curves = {}, {}
for D in dims:
# place both endpoints exactly on the shell (norm = sqrt(D)) -- the realistic high-density region
a = nrm(rng5.standard_normal(D)) * np.sqrt(D)
b = nrm(rng5.standard_normal(D)) * np.sqrt(D)
lerp_curves[D] = np.array([np.linalg.norm(lerp(a, b, t)) for t in ts]) / np.sqrt(D)
slerp_curves[D] = np.array([np.linalg.norm(slerp(a, b, t)) for t in ts]) / np.sqrt(D)
for D in dims:
print(f'D={D:4d}: Lerp midpoint norm / sqrt(D) = {lerp_curves[D][25]:.3f} '
f'Slerp = {slerp_curves[D][25]:.3f}')
D= 2: Lerp midpoint norm / sqrt(D) = 0.514 Slerp = 1.000 D= 10: Lerp midpoint norm / sqrt(D) = 0.612 Slerp = 1.000 D= 100: Lerp midpoint norm / sqrt(D) = 0.677 Slerp = 1.000
fig, ax = plt.subplots(figsize=(10, 5.8))
colors = {2: '#5dade2', 10: '#2980b9', 100: '#1b2631'}
for D in dims:
ax.plot(ts, lerp_curves[D], '--', color=colors[D], lw=2.2, label=f'Lerp (D={D})')
ax.plot(ts, slerp_curves[D], '-', color=colors[D], lw=2.2, label=f'Slerp (D={D})')
ax.axhline(1.0, color='#c0392b', ls=':', lw=1.6, label='shell radius √D')
ax.set_xlabel('interpolation parameter t')
ax.set_ylabel('norm / √D (1.0 = on the shell)')
ax.set_title('Lerp norm collapses at the midpoint; Slerp rides the shell at every D')
ax.legend(fontsize=9, ncol=2)
plt.tight_layout()
plt.show()
What you see¶
What the plot shows: Every dashed Lerp curve sags into a U with its trough at $t=0.5$, dropping to roughly $0.7\sqrt{D}$ — about 30% of the norm gone at the midpoint. Every solid Slerp curve stays flat on the red shell line at $1.0$, regardless of dimension. The effect is identical in shape from $D=2$ to $D=100$.
Why it looks this way: Two independent Gaussian vectors are nearly orthogonal in high dimensions, so the Lerp midpoint $\tfrac12(a+b)$ has norm $\tfrac12\sqrt{\|a\|^2+\|b\|^2+2a\cdot b}\approx \sqrt{D/2}=0.71\sqrt{D}$ — it falls toward the empty core. Slerp travels along the great circle of constant radius, so by construction it never leaves the shell.
Key takeaway: In high-dimensional generative priors, "halfway between two samples" in the Lerp sense is a point the prior considers wildly unlikely — abnormally small norm, deep in the low-density core. This is the "tent-pole" artefact: crisp endpoints, mushy middle. Slerp fixes it for free by preserving norm.
Experiment 6: Lerp vs Slerp — staying in-distribution¶
Question: Experiment 5 tracked a single pair. Now the statistical version: across many random pairs, where do the midpoints land relative to the prior's own norm distribution? If Slerp is the right interpolator, its midpoints should be statistically indistinguishable from fresh prior samples, while Lerp midpoints should pile up at too-small norms with much lower likelihood.
We draw thousands of Gaussian pairs in $D=100$, collect Lerp and Slerp midpoint norms, and compare them to the prior shell and to the radial likelihood $\log p(r) = (D-1)\log r - r^2/2$ — the density of a sample's norm, which (unlike the pointwise density) accounts for the volume factor and so reveals the typical set the decoder actually saw.
np.random.seed(6)
rng6 = np.random.default_rng(6)
D = 100
m = 4000
A = rng6.standard_normal((m, D))
B = rng6.standard_normal((m, D))
lerp_mid = 0.5 * (A + B)
slerp_mid = np.array([slerp(A[i], B[i], 0.5) for i in range(m)])
prior = rng6.standard_normal((m, D))
norm_lerp = np.linalg.norm(lerp_mid, axis=1)
norm_slerp = np.linalg.norm(slerp_mid, axis=1)
norm_prior = np.linalg.norm(prior, axis=1)
# In high dimensions the *mass* lives on the shell, not at the high-density core. The likelihood of
# a sample's NORM under N(0, I) follows the radial density log p(r) = (D-1) log r - r^2 / 2 (the
# (D-1) log r term is the volume of the sphere of radius r). This -- not the pointwise density
# -0.5||x||^2 -- is what says whether a point is in the typical set the decoder was trained on.
def log_radial(X):
r = np.linalg.norm(X, axis=1)
return (D - 1) * np.log(r) - 0.5 * r ** 2
print(f'mean norm -> prior: {norm_prior.mean():.2f} Slerp: {norm_slerp.mean():.2f} '
f'Lerp: {norm_lerp.mean():.2f} (shell sqrt(D) = {np.sqrt(D):.2f})')
print(f'mean log p(r) -> prior: {log_radial(prior).mean():.1f} Slerp: {log_radial(slerp_mid).mean():.1f} '
f'Lerp: {log_radial(lerp_mid).mean():.1f}')
mean norm -> prior: 9.99 Slerp: 9.99 Lerp: 7.05 (shell sqrt(D) = 10.00) mean log p(r) -> prior: 177.4 Slerp: 177.7 Lerp: 168.1
fig, axes = plt.subplots(1, 2, figsize=(14.5, 5.6))
fig.suptitle('Experiment 6: Slerp midpoints match the prior; Lerp midpoints fall off the shell',
fontsize=12.5, fontweight='bold')
# --- left: midpoint norm distributions ---
ax = axes[0]
bins = np.linspace(6, 13, 50)
ax.hist(norm_prior, bins=bins, density=True, alpha=0.5, color='#7f8c8d', label='prior samples')
ax.hist(norm_slerp, bins=bins, density=True, alpha=0.55, color='#27ae60', label='Slerp midpoints')
ax.hist(norm_lerp, bins=bins, density=True, alpha=0.55, color='#e74c3c', label='Lerp midpoints')
ax.axvline(np.sqrt(D), color='black', ls=':', lw=1.6, label='shell √D')
ax.set_xlabel('midpoint norm'); ax.set_ylabel('density')
ax.set_title('Lerp midpoints cluster at too-small norm')
ax.legend(fontsize=9)
# --- right: radial log-density (likelihood of the norm under the prior) ---
ax2 = axes[1]
labels = ['prior', 'Slerp\nmidpoint', 'Lerp\nmidpoint']
means = [log_radial(prior).mean(), log_radial(slerp_mid).mean(), log_radial(lerp_mid).mean()]
cols = ['#7f8c8d', '#27ae60', '#e74c3c']
bars = ax2.bar(labels, means, color=cols, edgecolor='black', width=0.6)
ax2.set_ylim(min(means) - 4, max(means) + 1)
for b, v in zip(bars, means):
ax2.text(b.get_x() + b.get_width() / 2, v, f'{v:.0f}', ha='center', va='bottom', fontsize=11)
ax2.set_ylabel('mean radial log-density log p(r)')
ax2.set_title('Lerp midpoints are far less likely on the shell')
plt.tight_layout()
plt.show()
What you see¶
What the plot shows: The Slerp midpoint norms (green) sit right on top of the prior's own norm distribution (grey), centred on the shell at $\sqrt{D}\approx 10$. The Lerp midpoints (red) form a separate bump shifted to clearly smaller norms (mean ≈ 7). The right panel turns this into likelihood: under the radial density, Slerp midpoints are as probable as real prior samples, while Lerp midpoints score about 10 log-units lower.
Why it looks this way — the concentration-of-measure twist: Moving toward the origin actually raises the pointwise density $-\tfrac12\|x\|^2$, so it is tempting (and wrong) to call Lerp midpoints "more likely". What matters is probability mass, and mass carries a volume factor $r^{D-1}$ that in high dimensions concentrates almost everything onto a thin shell at $\sqrt{D}$. The radial density $\log p(r)=(D-1)\log r-\tfrac{r^2}{2}$ encodes this: it peaks on the shell, and Lerp's shrunken-norm midpoints fall off it. Slerp, preserving norm and rotational symmetry, stays a genuine shell sample.
Key takeaway: Interpolation quality is an in-distribution question, and "in-distribution" in high dimensions means on the typical-set shell, not at the high-density core. Slerp keeps every intermediate point on the shell the decoder was trained on; Lerp quietly slides into the empty core. For a project that manipulates latent codes, this is the difference between a smooth, faithful traversal and a sequence of degraded, off-manifold samples — and where the manifold is irregular rather than a clean sphere, the pullback-geodesic methods from the research file generalise the same fix.
Summary¶
| Experiment | Theme | Key insight |
|---|---|---|
| 1. Parallel transport | Curvature foundations | A vector dragged around a closed loop returns rotated — holonomy is curvature you can watch |
| 2. Holonomy = K·Area | Curvature foundations | The deflection is linear in enclosed area; its slope is the curvature (0 for a plane) |
| 3. Geodesic vs chord | Bridge to Lerp | The shortest on-manifold path is the geodesic; the straight chord leaves the surface and loses norm |
| 4. Off-manifold Lerp | Lerp failure #1 | On a curved data manifold, a straight interpolation cuts through a zero-density void → blurry samples |
| 5. Shell & norm decay | Lerp failure #2 | In high-D Gaussian priors, the Lerp midpoint norm collapses to ≈0.7√D, off the data shell |
| 6. Lerp vs Slerp | The fix | Slerp midpoints match the prior shell and likelihood; Lerp midpoints are off-shell and improbable |
The big picture¶
Riemannian geometry gives latent space its true shape. Parallel transport and holonomy (Experiments 1–2) show that curvature is real and measurable — there is no global "same direction" or "straight line" on a curved manifold. That abstraction has teeth the moment we interpolate: geodesics ≠ chords (Experiment 3), so naive Lerp fails in two distinct ways — by slicing through off-manifold voids on a curved data sheet (Experiment 4) and by collapsing the norm off the high-dimensional probability shell (Experiments 5–6). Slerp repairs the spherical case by riding the shell at constant norm, and pullback geodesics generalise the fix to irregular manifolds by bending the path toward high-density ridges. For any system that manipulates latent codes, respecting this geometry is the precondition for interpolations that stay sharp, plausible, and on-manifold.