Lerp — Linear Interpolation: Visual Experiments¶

Goal: Build geometric intuition for lerp — what it computes, when it works, and when it silently breaks.

What we'll explore¶

#	Experiment	Question answered
1	The Lerp Path	What does lerp actually compute geometrically?
2	Norm Dip on the Unit Circle	Why does lerp fail for unit-norm vectors?
3	Norm Drop vs Angle	How bad is the dip, and for which angles?
4	Off-Manifold Pass-Through	What happens when lerp crosses the void between clusters?
5	High-Dimensional Concentration	Why does this matter in realistic latent dimensions?

---

Linked theory: research/01-lerp.md

In [1]:

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import gaussian_kde
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
plt.rcParams.update({
    'figure.dpi': 110,
    'font.size': 11,
    'axes.titlesize': 12,
    'axes.labelsize': 11,
    'figure.facecolor': 'white',
    'axes.facecolor': '#f8f9fa',
    'axes.grid': True,
    'grid.alpha': 0.35,
})
print('Setup complete.')

import numpy as np import matplotlib.pyplot as plt from scipy.stats import gaussian_kde import warnings warnings.filterwarnings('ignore') np.random.seed(42) plt.rcParams.update({ 'figure.dpi': 110, 'font.size': 11, 'axes.titlesize': 12, 'axes.labelsize': 11, 'figure.facecolor': 'white', 'axes.facecolor': '#f8f9fa', 'axes.grid': True, 'grid.alpha': 0.35, }) print('Setup complete.')

Setup complete.

---

Experiment 1: The Lerp Path¶

Question: What does lerp actually compute in 2D space?

Lerp between $z_1$ and $z_2$ traces the unique straight line connecting them, parameterized at constant speed: $z(t) = (1-t)\cdot z_1 + t\cdot z_2$. In flat Euclidean space this is the geodesic — the shortest path — which is why lerp is the natural baseline for latent interpolation.

In [2]:

np.random.seed(1)

z1 = np.array([-1.5, -1.0])
z2 = np.array([2.0, 1.5])
t_values = np.linspace(0, 1, 11)
path = np.array([(1 - t) * z1 + t * z2 for t in t_values])
step_sizes = np.linalg.norm(np.diff(path, axis=0), axis=1)

print(f"z1 = {z1}")
print(f"z2 = {z2}")
print(f"Step sizes along the path (should all be equal): {step_sizes.round(4)}")
print(f"Std of step sizes: {step_sizes.std():.6f}  (zero = perfectly uniform)")

np.random.seed(1) z1 = np.array([-1.5, -1.0]) z2 = np.array([2.0, 1.5]) t_values = np.linspace(0, 1, 11) path = np.array([(1 - t) * z1 + t * z2 for t in t_values]) step_sizes = np.linalg.norm(np.diff(path, axis=0), axis=1) print(f"z1 = {z1}") print(f"z2 = {z2}") print(f"Step sizes along the path (should all be equal): {step_sizes.round(4)}") print(f"Std of step sizes: {step_sizes.std():.6f} (zero = perfectly uniform)")

z1 = [-1.5 -1. ]
z2 = [2.  1.5]
Step sizes along the path (should all be equal): [0.4301 0.4301 0.4301 0.4301 0.4301 0.4301 0.4301 0.4301 0.4301 0.4301]
Std of step sizes: 0.000000  (zero = perfectly uniform)

In [3]:

fig, axes = plt.subplots(1, 2, figsize=(13, 5))
fig.suptitle('Experiment 1: Lerp = Constant-Speed Straight Line', fontsize=13, fontweight='bold')

# --- Left: path in 2D ---
ax = axes[0]
cmap = plt.get_cmap('plasma')
ax.plot([z1[0], z2[0]], [z1[1], z2[1]], 'k--', lw=1.0, alpha=0.4, label='Lerp chord')
for i, (pt, t) in enumerate(zip(path, t_values)):
    ax.scatter(*pt, color=cmap(t), s=80, zorder=5)
    ax.annotate(f't={t:.1f}', pt, textcoords='offset points',
                xytext=(5, 5), fontsize=8)
ax.scatter(*z1, color='blue', s=180, zorder=6, label='$z_1$', marker='*')
ax.scatter(*z2, color='red',  s=180, zorder=6, label='$z_2$', marker='*')
ax.set_xlabel('Dimension 1')
ax.set_ylabel('Dimension 2')
ax.set_title('Lerp path (colored by t)')
ax.legend()
ax.set_aspect('equal')

# --- Right: step sizes ---
ax2 = axes[1]
ax2.bar(range(len(step_sizes)), step_sizes, color='steelblue', alpha=0.8)
ax2.axhline(step_sizes.mean(), color='red', ls='--',
            label=f'Mean = {step_sizes.mean():.4f}')
ax2.set_xlabel('Step index')
ax2.set_ylabel('Euclidean step size')
ax2.set_title('Step sizes along the lerp path')
ax2.legend()

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(13, 5)) fig.suptitle('Experiment 1: Lerp = Constant-Speed Straight Line', fontsize=13, fontweight='bold') # --- Left: path in 2D --- ax = axes[0] cmap = plt.get_cmap('plasma') ax.plot([z1[0], z2[0]], [z1[1], z2[1]], 'k--', lw=1.0, alpha=0.4, label='Lerp chord') for i, (pt, t) in enumerate(zip(path, t_values)): ax.scatter(*pt, color=cmap(t), s=80, zorder=5) ax.annotate(f't={t:.1f}', pt, textcoords='offset points', xytext=(5, 5), fontsize=8) ax.scatter(*z1, color='blue', s=180, zorder=6, label='$z_1$', marker='*') ax.scatter(*z2, color='red', s=180, zorder=6, label='$z_2$', marker='*') ax.set_xlabel('Dimension 1') ax.set_ylabel('Dimension 2') ax.set_title('Lerp path (colored by t)') ax.legend() ax.set_aspect('equal') # --- Right: step sizes --- ax2 = axes[1] ax2.bar(range(len(step_sizes)), step_sizes, color='steelblue', alpha=0.8) ax2.axhline(step_sizes.mean(), color='red', ls='--', label=f'Mean = {step_sizes.mean():.4f}') ax2.set_xlabel('Step index') ax2.set_ylabel('Euclidean step size') ax2.set_title('Step sizes along the lerp path') ax2.legend() plt.tight_layout() plt.show()

What you see¶

1. What the plot shows: The left panel shows the lerp path as a straight chord from $z_1$ (blue star) to $z_2$ (red star), with intermediate points colored from purple (t=0) to yellow (t=1). The right panel shows the Euclidean step size for each of the 10 equal-$t$ intervals — they are all identical.

2. Why it looks this way: Lerp is linear in $t$, so the velocity $dz/dt = z_2 - z_1$ is constant. Equal $\Delta t$ steps produce equal Euclidean displacements. This makes lerp the definition of "constant speed" in Euclidean geometry — it is the geodesic.

3. Key takeaway: In flat space lerp is optimal — it is both the shortest path and the most uniform path. The trouble starts when the latent space is not flat.

---

Experiment 2: Norm Dip on the Unit Circle¶

Question: What happens to the norm of a lerp midpoint when $z_1$ and $z_2$ are unit vectors?

For unit vectors with angle $\theta$ between them, the midpoint norm is exactly $\|z(0.5)\| = \cos(\theta/2)$, which is strictly less than 1 whenever $\theta > 0$. Lerp pulls the midpoint inside the sphere — into a region the prior has never trained the decoder to handle.

In [4]:

np.random.seed(2)

angles_deg = [45, 90, 135]
t_fine = np.linspace(0, 1, 300)

results = {}
for deg in angles_deg:
    theta = np.radians(deg)
    z1 = np.array([1.0, 0.0])
    z2 = np.array([np.cos(theta), np.sin(theta)])
    path_t = np.array([(1 - t) * z1 + t * z2 for t in t_fine])
    norms = np.linalg.norm(path_t, axis=1)
    theory_min = np.cos(theta / 2)
    results[deg] = {'path': path_t, 'norms': norms, 'min_norm': theory_min}
    print(f"theta = {deg:3d}° | midpoint norm = {theory_min:.4f} "
          f"| theory cos({deg//2}°) = {np.cos(theta/2):.4f}")

np.random.seed(2) angles_deg = [45, 90, 135] t_fine = np.linspace(0, 1, 300) results = {} for deg in angles_deg: theta = np.radians(deg) z1 = np.array([1.0, 0.0]) z2 = np.array([np.cos(theta), np.sin(theta)]) path_t = np.array([(1 - t) * z1 + t * z2 for t in t_fine]) norms = np.linalg.norm(path_t, axis=1) theory_min = np.cos(theta / 2) results[deg] = {'path': path_t, 'norms': norms, 'min_norm': theory_min} print(f"theta = {deg:3d}° | midpoint norm = {theory_min:.4f} " f"| theory cos({deg//2}°) = {np.cos(theta/2):.4f}")

theta =  45° | midpoint norm = 0.9239 | theory cos(22°) = 0.9239
theta =  90° | midpoint norm = 0.7071 | theory cos(45°) = 0.7071
theta = 135° | midpoint norm = 0.3827 | theory cos(67°) = 0.3827

In [5]:

fig, axes = plt.subplots(1, 2, figsize=(13, 5))
fig.suptitle('Experiment 2: Lerp Midpoint Dips Inside the Unit Circle', fontsize=13, fontweight='bold')

colors_2 = ['#1f77b4', '#ff7f0e', '#2ca02c']

# --- Left: unit circle + chords ---
ax = axes[0]
circle_t = np.linspace(0, 2 * np.pi, 400)
ax.plot(np.cos(circle_t), np.sin(circle_t), 'k-', lw=2, label='Unit circle', zorder=1)
ax.fill_between(np.cos(circle_t), np.sin(circle_t), alpha=0.05, color='gray')

for deg, col in zip(angles_deg, colors_2):
    path = results[deg]['path']
    min_n = results[deg]['min_norm']
    ax.plot(path[:, 0], path[:, 1], '-', color=col, lw=2.0,
            label=f'Chord θ={deg}°', zorder=3)
    midpt = path[len(path) // 2]
    ax.scatter(*midpt, color=col, s=130, zorder=5, marker='D')
    ax.annotate(f'‖mid‖={min_n:.3f}', midpt,
                textcoords='offset points', xytext=(-60, 6), fontsize=8.5, color=col)

ax.scatter(1, 0, color='black', s=120, zorder=6, label='$z_1 = (1,0)$')
ax.set_xlim(-1.25, 1.25)
ax.set_ylim(-0.25, 1.55)
ax.set_aspect('equal')
ax.set_xlabel('Dimension 1')
ax.set_ylabel('Dimension 2')
ax.set_title('Lerp chords dive inside the unit circle')
ax.legend(loc='upper left', fontsize=9)

# --- Right: norm(t) curves ---
ax2 = axes[1]
for deg, col in zip(angles_deg, colors_2):
    ax2.plot(t_fine, results[deg]['norms'], '-', color=col, lw=2, label=f'θ={deg}°')
    ax2.axhline(results[deg]['min_norm'], color=col, ls=':', lw=1.0, alpha=0.5)

ax2.axhline(1.0, color='black', ls='--', lw=1.5, label='Unit norm = 1')
ax2.set_xlabel('Interpolation parameter t')
ax2.set_ylabel('‖z(t)‖')
ax2.set_title('Norm of lerp path vs t')
ax2.legend(fontsize=9)

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(13, 5)) fig.suptitle('Experiment 2: Lerp Midpoint Dips Inside the Unit Circle', fontsize=13, fontweight='bold') colors_2 = ['#1f77b4', '#ff7f0e', '#2ca02c'] # --- Left: unit circle + chords --- ax = axes[0] circle_t = np.linspace(0, 2 * np.pi, 400) ax.plot(np.cos(circle_t), np.sin(circle_t), 'k-', lw=2, label='Unit circle', zorder=1) ax.fill_between(np.cos(circle_t), np.sin(circle_t), alpha=0.05, color='gray') for deg, col in zip(angles_deg, colors_2): path = results[deg]['path'] min_n = results[deg]['min_norm'] ax.plot(path[:, 0], path[:, 1], '-', color=col, lw=2.0, label=f'Chord θ={deg}°', zorder=3) midpt = path[len(path) // 2] ax.scatter(*midpt, color=col, s=130, zorder=5, marker='D') ax.annotate(f'‖mid‖={min_n:.3f}', midpt, textcoords='offset points', xytext=(-60, 6), fontsize=8.5, color=col) ax.scatter(1, 0, color='black', s=120, zorder=6, label='$z_1 = (1,0)$') ax.set_xlim(-1.25, 1.25) ax.set_ylim(-0.25, 1.55) ax.set_aspect('equal') ax.set_xlabel('Dimension 1') ax.set_ylabel('Dimension 2') ax.set_title('Lerp chords dive inside the unit circle') ax.legend(loc='upper left', fontsize=9) # --- Right: norm(t) curves --- ax2 = axes[1] for deg, col in zip(angles_deg, colors_2): ax2.plot(t_fine, results[deg]['norms'], '-', color=col, lw=2, label=f'θ={deg}°') ax2.axhline(results[deg]['min_norm'], color=col, ls=':', lw=1.0, alpha=0.5) ax2.axhline(1.0, color='black', ls='--', lw=1.5, label='Unit norm = 1') ax2.set_xlabel('Interpolation parameter t') ax2.set_ylabel('‖z(t)‖') ax2.set_title('Norm of lerp path vs t') ax2.legend(fontsize=9) plt.tight_layout() plt.show()

What you see¶

1. What the plot shows: Three lerp chords (θ = 45°, 90°, 135°) all dive inside the unit circle. The diamond markers show the midpoints; their norms drop from 0.924 to 0.383. The right panel shows symmetric norm-dip curves, hitting minimum exactly at t = 0.5.

2. Why it looks this way: A chord connecting two boundary points of a convex set passes through the interior. The formula $\|z(0.5)\| = \cos(\theta/2)$ follows from the half-angle identity $(1+\cos\theta)/2 = \cos^2(\theta/2)$. Larger angle → deeper dip.

3. Key takeaway: For unit-norm latent vectors, lerp always constructs midpoints outside the support of the prior — the model has never seen these interior points during training, so decoding produces unreliable results.

---

Experiment 3: Norm Drop vs Angle — The Full Picture¶

Question: How does the midpoint norm vary across all angles, and what angles do random unit vectors in a latent space actually have?

Random unit vectors in $\mathbb{R}^d$ tend toward $\theta \approx 90°$ as $d$ grows (concentration of measure), placing the predicted midpoint norm near $\cos(45°) \approx 0.707$ regardless of which vectors are chosen.

In [6]:

np.random.seed(3)

# Theoretical curve: min norm = cos(theta/2)
theta_deg = np.linspace(0, 180, 400)
min_norms_theory = np.cos(np.radians(theta_deg) / 2)

# Mean angle between random unit vectors across dimensions
dimensions = [2, 5, 10, 50, 200, 1000]
n_pairs = 2000
avg_angles = {}

for d in dimensions:
    z1s = np.random.randn(n_pairs, d)
    z2s = np.random.randn(n_pairs, d)
    z1s /= np.linalg.norm(z1s, axis=1, keepdims=True)
    z2s /= np.linalg.norm(z2s, axis=1, keepdims=True)
    cos_a = np.clip(np.sum(z1s * z2s, axis=1), -1.0, 1.0)
    angles = np.degrees(np.arccos(cos_a))
    avg_angles[d] = angles.mean()
    pred_norm = np.cos(np.radians(angles.mean()) / 2)
    print(f"d={d:5d} | mean angle = {angles.mean():.2f}° | predicted midpoint norm = {pred_norm:.4f}")

np.random.seed(3) # Theoretical curve: min norm = cos(theta/2) theta_deg = np.linspace(0, 180, 400) min_norms_theory = np.cos(np.radians(theta_deg) / 2) # Mean angle between random unit vectors across dimensions dimensions = [2, 5, 10, 50, 200, 1000] n_pairs = 2000 avg_angles = {} for d in dimensions: z1s = np.random.randn(n_pairs, d) z2s = np.random.randn(n_pairs, d) z1s /= np.linalg.norm(z1s, axis=1, keepdims=True) z2s /= np.linalg.norm(z2s, axis=1, keepdims=True) cos_a = np.clip(np.sum(z1s * z2s, axis=1), -1.0, 1.0) angles = np.degrees(np.arccos(cos_a)) avg_angles[d] = angles.mean() pred_norm = np.cos(np.radians(angles.mean()) / 2) print(f"d={d:5d} | mean angle = {angles.mean():.2f}° | predicted midpoint norm = {pred_norm:.4f}")

d=    2 | mean angle = 91.25° | predicted midpoint norm = 0.6993
d=    5 | mean angle = 90.60° | predicted midpoint norm = 0.7034
d=   10 | mean angle = 89.90° | predicted midpoint norm = 0.7077
d=   50 | mean angle = 89.67° | predicted midpoint norm = 0.7091
d=  200 | mean angle = 89.90° | predicted midpoint norm = 0.7077
d= 1000 | mean angle = 90.02° | predicted midpoint norm = 0.7070

In [7]:

fig, axes = plt.subplots(1, 2, figsize=(13, 5))
fig.suptitle('Experiment 3: Norm Drop vs Angle θ', fontsize=13, fontweight='bold')

# --- Left: theoretical curve ---
ax = axes[0]
ax.plot(theta_deg, min_norms_theory, 'b-', lw=2.5,
        label=r'$\|z(0.5)\| = \cos(\theta/2)$')
ax.axhline(np.cos(np.pi / 4), color='orange', ls='--', lw=1.5,
           label=r'$\cos(45°) \approx 0.707$  [θ=90°]')
ax.axhline(np.cos(np.pi / 6), color='green',  ls='--', lw=1.5,
           label=r'$\cos(30°) \approx 0.866$  [θ=60°]')
ax.axvline(90, color='gray', ls=':', lw=1.0, alpha=0.7)

for angle_d in [45, 90, 120, 150]:
    norm_val = np.cos(np.radians(angle_d / 2))
    ax.scatter(angle_d, norm_val, s=90, zorder=5)
    ax.annotate(f'{norm_val:.3f}', (angle_d, norm_val),
                textcoords='offset points', xytext=(6, 4), fontsize=8.5)

ax.set_xlabel('Angle θ between z₁ and z₂ (degrees)')
ax.set_ylabel('Midpoint norm ‖z(0.5)‖')
ax.set_title('Theoretical norm drop vs angle')
ax.legend(fontsize=9)

# --- Right: where random pairs land per dimension ---
ax2 = axes[1]
ax2.plot(theta_deg, min_norms_theory, 'k-', lw=1.5, alpha=0.3, label='Theory curve')
ax2.axvline(90, color='gray', ls=':', lw=1.0)
ax2.axhline(np.cos(np.pi / 4), color='orange', ls='--', lw=1.0, alpha=0.8)

colors_d3 = plt.cm.viridis(np.linspace(0.05, 0.9, len(dimensions)))
for d, col in zip(dimensions, colors_d3):
    ang = avg_angles[d]
    norm_pred = np.cos(np.radians(ang) / 2)
    ax2.scatter(ang, norm_pred, color=col, s=140, zorder=5,
                label=f'd={d}  (θ̄={ang:.0f}°, ‖mid‖≈{norm_pred:.3f})')

ax2.set_xlabel('Mean angle θ̄ between random unit vectors (degrees)')
ax2.set_ylabel('Predicted midpoint norm')
ax2.set_title('Where random pairs land by dimension')
ax2.legend(fontsize=8.5, loc='upper right')

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(13, 5)) fig.suptitle('Experiment 3: Norm Drop vs Angle θ', fontsize=13, fontweight='bold') # --- Left: theoretical curve --- ax = axes[0] ax.plot(theta_deg, min_norms_theory, 'b-', lw=2.5, label=r'$\|z(0.5)\| = \cos(\theta/2)$') ax.axhline(np.cos(np.pi / 4), color='orange', ls='--', lw=1.5, label=r'$\cos(45°) \approx 0.707$ [θ=90°]') ax.axhline(np.cos(np.pi / 6), color='green', ls='--', lw=1.5, label=r'$\cos(30°) \approx 0.866$ [θ=60°]') ax.axvline(90, color='gray', ls=':', lw=1.0, alpha=0.7) for angle_d in [45, 90, 120, 150]: norm_val = np.cos(np.radians(angle_d / 2)) ax.scatter(angle_d, norm_val, s=90, zorder=5) ax.annotate(f'{norm_val:.3f}', (angle_d, norm_val), textcoords='offset points', xytext=(6, 4), fontsize=8.5) ax.set_xlabel('Angle θ between z₁ and z₂ (degrees)') ax.set_ylabel('Midpoint norm ‖z(0.5)‖') ax.set_title('Theoretical norm drop vs angle') ax.legend(fontsize=9) # --- Right: where random pairs land per dimension --- ax2 = axes[1] ax2.plot(theta_deg, min_norms_theory, 'k-', lw=1.5, alpha=0.3, label='Theory curve') ax2.axvline(90, color='gray', ls=':', lw=1.0) ax2.axhline(np.cos(np.pi / 4), color='orange', ls='--', lw=1.0, alpha=0.8) colors_d3 = plt.cm.viridis(np.linspace(0.05, 0.9, len(dimensions))) for d, col in zip(dimensions, colors_d3): ang = avg_angles[d] norm_pred = np.cos(np.radians(ang) / 2) ax2.scatter(ang, norm_pred, color=col, s=140, zorder=5, label=f'd={d} (θ̄={ang:.0f}°, ‖mid‖≈{norm_pred:.3f})') ax2.set_xlabel('Mean angle θ̄ between random unit vectors (degrees)') ax2.set_ylabel('Predicted midpoint norm') ax2.set_title('Where random pairs land by dimension') ax2.legend(fontsize=8.5, loc='upper right') plt.tight_layout() plt.show()

What you see¶

1. What the plot shows: The left panel traces $\|z(0.5)\| = \cos(\theta/2)$ from 1 (identical vectors) to 0 (antipodal). The right panel overlays where random unit-vector pairs land for each dimension: as $d$ grows, the mean angle converges to ~90°, placing the midpoint norm near 0.707.

2. Why it looks this way: In high dimensions, two independently sampled unit vectors are nearly orthogonal with overwhelming probability — concentration of measure. This is not a worst case; it is the typical case for latent spaces with d ≥ 50.

3. Key takeaway: For normalized latent vectors in realistic dimensions, the norm dip is not occasional — it is structural. Lerp midpoints will land around ‖mid‖ ≈ 0.707 for almost any pair of points.

---

Experiment 4: Off-Manifold Pass-Through — The Semantic Void¶

Question: What happens when lerp crosses the gap between two distinct clusters?

Even with non-normalized vectors, lerp fails if the two endpoints lie in different semantic modes separated by a low-density region. The straight line passes through points the model has never seen — the decoder produces incoherent output there.

In [8]:

np.random.seed(4)

n = 300
cluster1 = np.random.randn(n, 2) * 0.6 + np.array([-2.5, 0.0])
cluster2 = np.random.randn(n, 2) * 0.6 + np.array([2.5, 0.0])
all_pts = np.vstack([cluster1, cluster2])

z_a = cluster1[12]
z_b = cluster2[25]
t_vals = np.linspace(0, 1, 120)
lerp_path = np.array([(1 - t) * z_a + t * z_b for t in t_vals])

kde = gaussian_kde(all_pts.T, bw_method=0.3)
path_density = kde(lerp_path.T)

print(f"KDE density at z_a (cluster 1) : {kde(z_a[:, None])[0]:.4f}")
print(f"KDE density at z_b (cluster 2) : {kde(z_b[:, None])[0]:.4f}")
print(f"KDE density at midpoint (t=0.5): {path_density[60]:.4f}")

np.random.seed(4) n = 300 cluster1 = np.random.randn(n, 2) * 0.6 + np.array([-2.5, 0.0]) cluster2 = np.random.randn(n, 2) * 0.6 + np.array([2.5, 0.0]) all_pts = np.vstack([cluster1, cluster2]) z_a = cluster1[12] z_b = cluster2[25] t_vals = np.linspace(0, 1, 120) lerp_path = np.array([(1 - t) * z_a + t * z_b for t in t_vals]) kde = gaussian_kde(all_pts.T, bw_method=0.3) path_density = kde(lerp_path.T) print(f"KDE density at z_a (cluster 1) : {kde(z_a[:, None])[0]:.4f}") print(f"KDE density at z_b (cluster 2) : {kde(z_b[:, None])[0]:.4f}") print(f"KDE density at midpoint (t=0.5): {path_density[60]:.4f}")

KDE density at z_a (cluster 1) : 0.0585
KDE density at z_b (cluster 2) : 0.0489
KDE density at midpoint (t=0.5): 0.0479

In [9]:

fig, axes = plt.subplots(1, 2, figsize=(13, 5))
fig.suptitle('Experiment 4: Lerp Through the Semantic Void', fontsize=13, fontweight='bold')

# --- Left: scatter + lerp path ---
ax = axes[0]
ax.scatter(cluster1[:, 0], cluster1[:, 1], alpha=0.4, s=18, color='#1f77b4', label='Cluster A')
ax.scatter(cluster2[:, 0], cluster2[:, 1], alpha=0.4, s=18, color='#ff7f0e', label='Cluster B')
ax.plot(lerp_path[:, 0], lerp_path[:, 1], 'k--', lw=2.0, label='Lerp path', zorder=4)

sc = ax.scatter(lerp_path[5:-5, 0], lerp_path[5:-5, 1],
                c=path_density[5:-5], cmap='YlOrRd_r',
                s=25, zorder=5, vmin=0, vmax=path_density.max())
plt.colorbar(sc, ax=ax, label='KDE density')

ax.scatter(*z_a, s=220, color='#1f77b4', marker='*', zorder=6, label='$z_a$ (start)')
ax.scatter(*z_b, s=220, color='#ff7f0e', marker='*', zorder=6, label='$z_b$ (end)')
ax.set_xlabel('Dimension 1')
ax.set_ylabel('Dimension 2')
ax.set_title('Lerp crosses the inter-cluster void')
ax.legend(fontsize=9)

# --- Right: density along path ---
ax2 = axes[1]
ax2.fill_between(t_vals, path_density, alpha=0.25, color='red')
ax2.plot(t_vals, path_density, 'r-', lw=2.5)
ax2.axvline(0.5, color='gray', ls='--', lw=1.5, label='Midpoint t=0.5')
ax2.axhline(path_density[60], color='red', ls=':', lw=1.0,
            label=f'Density at midpoint ≈ {path_density[60]:.4f}')
ax2.scatter([0, 1], [path_density[0], path_density[-1]],
            s=110, color=['#1f77b4', '#ff7f0e'], zorder=5)
ax2.set_xlabel('Interpolation parameter t')
ax2.set_ylabel('KDE density at z(t)')
ax2.set_title('Data density along the lerp path')
ax2.legend(fontsize=9)

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(13, 5)) fig.suptitle('Experiment 4: Lerp Through the Semantic Void', fontsize=13, fontweight='bold') # --- Left: scatter + lerp path --- ax = axes[0] ax.scatter(cluster1[:, 0], cluster1[:, 1], alpha=0.4, s=18, color='#1f77b4', label='Cluster A') ax.scatter(cluster2[:, 0], cluster2[:, 1], alpha=0.4, s=18, color='#ff7f0e', label='Cluster B') ax.plot(lerp_path[:, 0], lerp_path[:, 1], 'k--', lw=2.0, label='Lerp path', zorder=4) sc = ax.scatter(lerp_path[5:-5, 0], lerp_path[5:-5, 1], c=path_density[5:-5], cmap='YlOrRd_r', s=25, zorder=5, vmin=0, vmax=path_density.max()) plt.colorbar(sc, ax=ax, label='KDE density') ax.scatter(*z_a, s=220, color='#1f77b4', marker='*', zorder=6, label='$z_a$ (start)') ax.scatter(*z_b, s=220, color='#ff7f0e', marker='*', zorder=6, label='$z_b$ (end)') ax.set_xlabel('Dimension 1') ax.set_ylabel('Dimension 2') ax.set_title('Lerp crosses the inter-cluster void') ax.legend(fontsize=9) # --- Right: density along path --- ax2 = axes[1] ax2.fill_between(t_vals, path_density, alpha=0.25, color='red') ax2.plot(t_vals, path_density, 'r-', lw=2.5) ax2.axvline(0.5, color='gray', ls='--', lw=1.5, label='Midpoint t=0.5') ax2.axhline(path_density[60], color='red', ls=':', lw=1.0, label=f'Density at midpoint ≈ {path_density[60]:.4f}') ax2.scatter([0, 1], [path_density[0], path_density[-1]], s=110, color=['#1f77b4', '#ff7f0e'], zorder=5) ax2.set_xlabel('Interpolation parameter t') ax2.set_ylabel('KDE density at z(t)') ax2.set_title('Data density along the lerp path') ax2.legend(fontsize=9) plt.tight_layout() plt.show()

What you see¶

1. What the plot shows: The left panel shows two Gaussian clusters (blue and orange) with the lerp path (black dashed) cutting straight through the empty space between them. Intermediate path points are colored by KDE density — bright red = near zero density. The right panel quantifies: density at start and end is high, collapsing to near zero at the midpoint.

2. Why it looks this way: The lerp formula only sees the two endpoints — it has no information about the data distribution. A straight line between two distant clusters inevitably crosses the inter-cluster gap: a region of zero training density. A decoder trained only on cluster A and cluster B has no guidance for what to produce there.

3. Key takeaway: Lerp is context-free. Even with non-normalized vectors, it fails whenever the latent manifold has disconnected regions. This failure mode is independent of the norm problem — it requires a density-aware interpolation method instead.

---

Experiment 5: High-Dimensional Concentration¶

Question: How does the lerp midpoint norm behave as latent dimension grows?

In $\mathbb{R}^d$ with a Gaussian prior, random unit vectors tend to be nearly orthogonal ($\theta \approx 90°$) as $d$ grows. This makes the norm dip from Experiment 2 not an edge case — it is the structural default for high-dimensional normalized latent spaces.

In [10]:

np.random.seed(5)

dimensions = [2, 8, 32, 128, 512, 2048]
n_pairs = 2000
midpoint_norms = {}

for d in dimensions:
    z1s = np.random.randn(n_pairs, d)
    z2s = np.random.randn(n_pairs, d)
    z1s /= np.linalg.norm(z1s, axis=1, keepdims=True)
    z2s /= np.linalg.norm(z2s, axis=1, keepdims=True)
    mids = 0.5 * (z1s + z2s)
    norms = np.linalg.norm(mids, axis=1)
    midpoint_norms[d] = norms
    print(f"d={d:5d} | mean midpoint norm = {norms.mean():.4f} ± {norms.std():.4f}")

np.random.seed(5) dimensions = [2, 8, 32, 128, 512, 2048] n_pairs = 2000 midpoint_norms = {} for d in dimensions: z1s = np.random.randn(n_pairs, d) z2s = np.random.randn(n_pairs, d) z1s /= np.linalg.norm(z1s, axis=1, keepdims=True) z2s /= np.linalg.norm(z2s, axis=1, keepdims=True) mids = 0.5 * (z1s + z2s) norms = np.linalg.norm(mids, axis=1) midpoint_norms[d] = norms print(f"d={d:5d} | mean midpoint norm = {norms.mean():.4f} ± {norms.std():.4f}")

d=    2 | mean midpoint norm = 0.6471 ± 0.3118
d=    8 | mean midpoint norm = 0.6959 ± 0.1329
d=   32 | mean midpoint norm = 0.7038 ± 0.0637
d=  128 | mean midpoint norm = 0.7063 ± 0.0309
d=  512 | mean midpoint norm = 0.7074 ± 0.0159

d= 2048 | mean midpoint norm = 0.7071 ± 0.0079

In [11]:

fig, axes = plt.subplots(1, 2, figsize=(13, 5))
fig.suptitle('Experiment 5: Midpoint Norm Concentrates as Dimension Grows', fontsize=13, fontweight='bold')

colors_5 = plt.cm.plasma(np.linspace(0.1, 0.9, len(dimensions)))

# --- Left: distributions ---
ax = axes[0]
for d, col in zip(dimensions, colors_5):
    ax.hist(midpoint_norms[d], bins=60, density=True, alpha=0.55, color=col, label=f'd={d}')
ax.axvline(np.cos(np.pi / 4), color='black', ls='--', lw=2,
           label=r'$\cos(45°) \approx 0.707$')
ax.set_xlabel('Lerp midpoint norm ‖z(0.5)‖')
ax.set_ylabel('Density')
ax.set_title('Distribution of midpoint norms by dimension')
ax.legend(fontsize=8.5)

# --- Right: mean ± std vs dimension ---
ax2 = axes[1]
means = [midpoint_norms[d].mean() for d in dimensions]
stds  = [midpoint_norms[d].std()  for d in dimensions]
ax2.errorbar(range(len(dimensions)), means, yerr=stds,
             fmt='o-', color='steelblue', capsize=5, lw=2, markersize=7)
ax2.axhline(np.cos(np.pi / 4), color='red', ls='--', lw=1.5,
            label=r'Limit: $\cos(45°) \approx 0.707$')
ax2.set_xticks(range(len(dimensions)))
ax2.set_xticklabels([str(d) for d in dimensions], rotation=30)
ax2.set_xlabel('Latent dimension d')
ax2.set_ylabel('Mean midpoint norm (± std)')
ax2.set_title('Mean midpoint norm converges with dimension')
ax2.legend()

plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(13, 5)) fig.suptitle('Experiment 5: Midpoint Norm Concentrates as Dimension Grows', fontsize=13, fontweight='bold') colors_5 = plt.cm.plasma(np.linspace(0.1, 0.9, len(dimensions))) # --- Left: distributions --- ax = axes[0] for d, col in zip(dimensions, colors_5): ax.hist(midpoint_norms[d], bins=60, density=True, alpha=0.55, color=col, label=f'd={d}') ax.axvline(np.cos(np.pi / 4), color='black', ls='--', lw=2, label=r'$\cos(45°) \approx 0.707$') ax.set_xlabel('Lerp midpoint norm ‖z(0.5)‖') ax.set_ylabel('Density') ax.set_title('Distribution of midpoint norms by dimension') ax.legend(fontsize=8.5) # --- Right: mean ± std vs dimension --- ax2 = axes[1] means = [midpoint_norms[d].mean() for d in dimensions] stds = [midpoint_norms[d].std() for d in dimensions] ax2.errorbar(range(len(dimensions)), means, yerr=stds, fmt='o-', color='steelblue', capsize=5, lw=2, markersize=7) ax2.axhline(np.cos(np.pi / 4), color='red', ls='--', lw=1.5, label=r'Limit: $\cos(45°) \approx 0.707$') ax2.set_xticks(range(len(dimensions))) ax2.set_xticklabels([str(d) for d in dimensions], rotation=30) ax2.set_xlabel('Latent dimension d') ax2.set_ylabel('Mean midpoint norm (± std)') ax2.set_title('Mean midpoint norm converges with dimension') ax2.legend() plt.tight_layout() plt.show()

What you see¶

1. What the plot shows: As dimension grows from d=2 to d=2048, the distribution of lerp midpoint norms narrows and concentrates at ≈ 0.707. At d=2 there is wide variation; at d=512 nearly all midpoints fall in a tight band around the theoretical limit.

2. Why it looks this way: In high dimensions, independently sampled unit vectors are nearly orthogonal with overwhelming probability — their dot product concentrates near 0, meaning $\theta \approx 90°$ and $\cos(\theta/2) \approx \cos(45°) = 1/\sqrt{2}$. This is concentration of measure applied to the dot product.

3. Key takeaway: For any modern latent space with d ≥ 128 and normalized vectors (VAE, diffusion model, hyperspherical prior), lerp midpoints always have norm ≈ 0.707. There is no lucky pair of points where lerp avoids this — the problem is structural. This is precisely why slerp was proposed as a replacement.

---

Summary¶

Experiment	Key insight
1. The Lerp Path	Lerp = constant-speed geodesic in flat space — optimal when geometry is Euclidean
2. Norm Dip on Unit Circle	Midpoint norm drops to $\cos(\theta/2)$ — the chord dives inside the sphere
3. Norm Drop vs Angle	At θ=90° the midpoint loses 29% norm; at θ=135° it loses 62%
4. Off-Manifold Pass-Through	Lerp through a void between clusters hits near-zero training density
5. High-Dimensional Concentration	In d ≥ 128, random unit-vector midpoints always concentrate at ≈ 0.707

Connection to the broader theory¶

Lerp is the mandatory starting point for understanding latent manipulation — simple, interpretable, and fast. Its failure modes map directly onto the structure needed for safe interpolation: flat geometry (or use slerp), connected density (or avoid cross-cluster paths), and ideally non-unit-norm vectors. The next notebook in this tier is Slerp, which fixes the norm-dip problem by following the geodesic on the hypersphere instead of the Euclidean chord.