Latent Arithmetic — Conditions for Meaningful Vector Operations¶

Learning objectives

1. Visualise the analogy formula $z_d \approx z_c + (z_b - z_a)$ geometrically in 2-D.
2. Extract a concept direction from labelled examples and apply steering.
3. Quantify linear structure by measuring how consistent concept directions are.
4. Show that arithmetic breaks across different coordinate systems (different training runs).
5. Measure concept leakage caused by non-orthogonal concept directions.

In [1]:

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from matplotlib.patches import FancyArrowPatch
from scipy.spatial.distance import cosine as cosine_dist
import warnings
warnings.filterwarnings("ignore")

np.random.seed(42)
plt.rcParams.update({
    "figure.dpi": 110,
    "font.size": 11,
    "axes.titlesize": 12,
    "axes.labelsize": 11,
    "figure.facecolor": "white",
    "axes.facecolor": "#f8f9fa",
    "axes.grid": True,
    "grid.alpha": 0.35,
})
print("Setup complete.")

import numpy as np import matplotlib.pyplot as plt import matplotlib.patches as mpatches from matplotlib.patches import FancyArrowPatch from scipy.spatial.distance import cosine as cosine_dist import warnings warnings.filterwarnings("ignore") np.random.seed(42) plt.rcParams.update({ "figure.dpi": 110, "font.size": 11, "axes.titlesize": 12, "axes.labelsize": 11, "figure.facecolor": "white", "axes.facecolor": "#f8f9fa", "axes.grid": True, "grid.alpha": 0.35, }) print("Setup complete.")

Setup complete.

Experiment 1: Analogy Arithmetic in 2-D¶

Question: What does $z_d \approx z_c + (z_b - z_a)$ look like geometrically?

The classic word2vec analogy king − man + woman ≈ queen assumes that the concept direction $\delta = z_{\text{king}} - z_{\text{man}}$ is the same as $z_{\text{queen}} - z_{\text{woman}}$. In other words, the four points form a parallelogram in latent space.

Here we build a toy 2-D latent space with two orthogonal semantic axes — gender (x) and royalty (y) — and verify the arithmetic geometrically.

In [2]:

np.random.seed(1)

# Toy latent vectors — perfect linear structure
# axis 0: gender  (0 = male, 1 = female)
# axis 1: royalty (0 = commoner, 1 = royal)
words = {
    "man":   np.array([0.0, 0.0]),
    "woman": np.array([1.0, 0.0]),
    "king":  np.array([0.0, 1.0]),
    "queen": np.array([1.0, 1.0]),
}

delta = words["king"] - words["man"]        # concept direction: royalty
predicted_queen = words["woman"] + delta    # analogy: woman + royalty = queen

fig, ax = plt.subplots(figsize=(6, 6))

# Points
colors = {"man": "#4393c3", "woman": "#d6604d", "king": "#4393c3", "queen": "#d6604d"}
for name, vec in words.items():
    ax.scatter(*vec, s=120, color=colors[name], zorder=5)
    ax.annotate(name, vec, textcoords="offset points", xytext=(8, 5),
                fontsize=13, fontweight="bold")

# Concept direction arrows
arrow_kw = dict(arrowstyle="-|>", lw=1.8, mutation_scale=15)
ax.annotate("", xy=words["king"], xytext=words["man"],
            arrowprops=dict(**arrow_kw, color="#2166ac"))
ax.annotate("", xy=predicted_queen, xytext=words["woman"],
            arrowprops=dict(**arrow_kw, color="#b2182b"))

# Mark prediction
ax.scatter(*predicted_queen, s=200, marker="*", color="gold",
           edgecolors="black", zorder=6, label="predicted queen")

# Label arrows
mid_m2k = (words["man"] + words["king"]) / 2
ax.annotate(r"$\delta$ = king − man", mid_m2k, xytext=(-80, 5),
            textcoords="offset points", fontsize=10, color="#2166ac")
mid_w2q = (words["woman"] + predicted_queen) / 2
ax.annotate(r"$\delta$ applied", mid_w2q, xytext=(8, -15),
            textcoords="offset points", fontsize=10, color="#b2182b")

ax.set_xlim(-0.4, 1.8)
ax.set_ylim(-0.4, 1.6)
ax.set_xlabel("Gender axis  (0 = male, 1 = female)")
ax.set_ylabel("Royalty axis  (0 = commoner, 1 = royal)")
ax.set_title("Analogy arithmetic: man : king :: woman : queen")
ax.legend(loc="upper left")
plt.tight_layout()
plt.show()

err = np.linalg.norm(predicted_queen - words["queen"])
print(f"Prediction error: {err:.4f}  (0 = perfect)")
print(f"delta = {delta}  (should point in +royalty direction)")

np.random.seed(1) # Toy latent vectors — perfect linear structure # axis 0: gender (0 = male, 1 = female) # axis 1: royalty (0 = commoner, 1 = royal) words = { "man": np.array([0.0, 0.0]), "woman": np.array([1.0, 0.0]), "king": np.array([0.0, 1.0]), "queen": np.array([1.0, 1.0]), } delta = words["king"] - words["man"] # concept direction: royalty predicted_queen = words["woman"] + delta # analogy: woman + royalty = queen fig, ax = plt.subplots(figsize=(6, 6)) # Points colors = {"man": "#4393c3", "woman": "#d6604d", "king": "#4393c3", "queen": "#d6604d"} for name, vec in words.items(): ax.scatter(*vec, s=120, color=colors[name], zorder=5) ax.annotate(name, vec, textcoords="offset points", xytext=(8, 5), fontsize=13, fontweight="bold") # Concept direction arrows arrow_kw = dict(arrowstyle="-|>", lw=1.8, mutation_scale=15) ax.annotate("", xy=words["king"], xytext=words["man"], arrowprops=dict(**arrow_kw, color="#2166ac")) ax.annotate("", xy=predicted_queen, xytext=words["woman"], arrowprops=dict(**arrow_kw, color="#b2182b")) # Mark prediction ax.scatter(*predicted_queen, s=200, marker="*", color="gold", edgecolors="black", zorder=6, label="predicted queen") # Label arrows mid_m2k = (words["man"] + words["king"]) / 2 ax.annotate(r"$\delta$ = king − man", mid_m2k, xytext=(-80, 5), textcoords="offset points", fontsize=10, color="#2166ac") mid_w2q = (words["woman"] + predicted_queen) / 2 ax.annotate(r"$\delta$ applied", mid_w2q, xytext=(8, -15), textcoords="offset points", fontsize=10, color="#b2182b") ax.set_xlim(-0.4, 1.8) ax.set_ylim(-0.4, 1.6) ax.set_xlabel("Gender axis (0 = male, 1 = female)") ax.set_ylabel("Royalty axis (0 = commoner, 1 = royal)") ax.set_title("Analogy arithmetic: man : king :: woman : queen") ax.legend(loc="upper left") plt.tight_layout() plt.show() err = np.linalg.norm(predicted_queen - words["queen"]) print(f"Prediction error: {err:.4f} (0 = perfect)") print(f"delta = {delta} (should point in +royalty direction)")

Prediction error: 0.0000  (0 = perfect)
delta = [0. 1.]  (should point in +royalty direction)

What you see¶

The two blue arrows (man→king and woman→queen) are identical in direction and length — they both represent $\delta$, the "royalty" concept vector. The gold star lands exactly on queen because the latent space was constructed with perfect orthogonal axes, so arithmetic is exact.

Key insight: Analogy arithmetic works when the four points form a parallelogram. That happens when the concept is encoded as a translation-invariant direction — the same additive offset regardless of which base word you start from. Real latent spaces are rarely this clean, but good models approximate it for many relations.

Experiment 2: Concept Direction Extraction and Steering¶

Question: Given labelled examples (with/without an attribute), how do we extract a steering direction, and what happens as we scale $\alpha$?

The mean-difference estimator $\delta = \bar{z}_{\text{with}} - \bar{z}_{\text{without}}$ is both the simplest and most robust method. We simulate a 2-D latent space where class 1 is offset by a "smile" direction of $[0.8,\ 0.3]$ (non-axis-aligned), extract $\delta$, and apply steering at several $\alpha$ values.

In [3]:

np.random.seed(2)

n = 40
# Ground-truth concept direction (normalised)
true_dir = np.array([0.8, 0.3])
true_dir /= np.linalg.norm(true_dir)

# Two clusters: neutral (class 0) and "smile" (class 1)
z_neutral = np.random.randn(n, 2) * 0.35
z_smile   = z_neutral + true_dir * 1.2 + np.random.randn(n, 2) * 0.15

# Estimated concept direction via mean difference
delta_hat  = z_smile.mean(axis=0) - z_neutral.mean(axis=0)
delta_unit = delta_hat / np.linalg.norm(delta_hat)

# Steer a single neutral point at several alpha values
z_base   = np.array([0.0, 0.0])
alphas   = [-1.0, 0.0, 0.5, 1.0, 1.5, 2.0]
steered  = [z_base + a * delta_hat for a in alphas]

fig, axes = plt.subplots(1, 2, figsize=(13, 5))

# Left: scatter + direction
ax = axes[0]
ax.scatter(*z_neutral.T, c="#4393c3", alpha=0.7, s=50, label="no smile (class 0)")
ax.scatter(*z_smile.T,   c="#d6604d", alpha=0.7, s=50, label="smile (class 1)")
ax.annotate("", xy=z_neutral.mean(axis=0) + delta_hat,
            xytext=z_neutral.mean(axis=0),
            arrowprops=dict(arrowstyle="-|>", lw=2, color="black", mutation_scale=14))
ax.text(*(z_neutral.mean(axis=0) + delta_hat * 0.5 + np.array([-0.15, 0.1])),
        r"$\hat{\delta}$", fontsize=14)
ax.set_xlabel("Latent dim 0")
ax.set_ylabel("Latent dim 1")
ax.set_title("Concept direction from mean difference")
ax.legend(fontsize=9)

# Right: steering trajectory
ax = axes[1]
ax.scatter(*z_neutral.T, c="#4393c3", alpha=0.3, s=30)
ax.scatter(*z_smile.T,   c="#d6604d", alpha=0.3, s=30)

traj = np.array(steered)
ax.plot(traj[:, 0], traj[:, 1], "o-", color="purple", lw=2, zorder=5,
        label="steered point")
for pt, al in zip(steered, alphas):
    ax.annotate(f"α={al}", pt, textcoords="offset points",
                xytext=(5, 5), fontsize=8)

ax.set_xlabel("Latent dim 0")
ax.set_ylabel("Latent dim 1")
ax.set_title(r"Steering: $z_{\mathrm{new}} = z + \alpha \cdot \hat{\delta}$")
ax.legend(fontsize=9)

fig.suptitle("Experiment 2 — concept direction & steering", fontweight="bold")
plt.tight_layout()
plt.show()

cos_sim = 1 - cosine_dist(delta_unit, true_dir)
print(f"Cosine similarity (estimated vs true direction): {cos_sim:.4f}")
print(f"|delta_hat| = {np.linalg.norm(delta_hat):.4f}")

np.random.seed(2) n = 40 # Ground-truth concept direction (normalised) true_dir = np.array([0.8, 0.3]) true_dir /= np.linalg.norm(true_dir) # Two clusters: neutral (class 0) and "smile" (class 1) z_neutral = np.random.randn(n, 2) * 0.35 z_smile = z_neutral + true_dir * 1.2 + np.random.randn(n, 2) * 0.15 # Estimated concept direction via mean difference delta_hat = z_smile.mean(axis=0) - z_neutral.mean(axis=0) delta_unit = delta_hat / np.linalg.norm(delta_hat) # Steer a single neutral point at several alpha values z_base = np.array([0.0, 0.0]) alphas = [-1.0, 0.0, 0.5, 1.0, 1.5, 2.0] steered = [z_base + a * delta_hat for a in alphas] fig, axes = plt.subplots(1, 2, figsize=(13, 5)) # Left: scatter + direction ax = axes[0] ax.scatter(*z_neutral.T, c="#4393c3", alpha=0.7, s=50, label="no smile (class 0)") ax.scatter(*z_smile.T, c="#d6604d", alpha=0.7, s=50, label="smile (class 1)") ax.annotate("", xy=z_neutral.mean(axis=0) + delta_hat, xytext=z_neutral.mean(axis=0), arrowprops=dict(arrowstyle="-|>", lw=2, color="black", mutation_scale=14)) ax.text(*(z_neutral.mean(axis=0) + delta_hat * 0.5 + np.array([-0.15, 0.1])), r"$\hat{\delta}$", fontsize=14) ax.set_xlabel("Latent dim 0") ax.set_ylabel("Latent dim 1") ax.set_title("Concept direction from mean difference") ax.legend(fontsize=9) # Right: steering trajectory ax = axes[1] ax.scatter(*z_neutral.T, c="#4393c3", alpha=0.3, s=30) ax.scatter(*z_smile.T, c="#d6604d", alpha=0.3, s=30) traj = np.array(steered) ax.plot(traj[:, 0], traj[:, 1], "o-", color="purple", lw=2, zorder=5, label="steered point") for pt, al in zip(steered, alphas): ax.annotate(f"α={al}", pt, textcoords="offset points", xytext=(5, 5), fontsize=8) ax.set_xlabel("Latent dim 0") ax.set_ylabel("Latent dim 1") ax.set_title(r"Steering: $z_{\mathrm{new}} = z + \alpha \cdot \hat{\delta}$") ax.legend(fontsize=9) fig.suptitle("Experiment 2 — concept direction & steering", fontweight="bold") plt.tight_layout() plt.show() cos_sim = 1 - cosine_dist(delta_unit, true_dir) print(f"Cosine similarity (estimated vs true direction): {cos_sim:.4f}") print(f"|delta_hat| = {np.linalg.norm(delta_hat):.4f}")

Cosine similarity (estimated vs true direction): 0.9999
|delta_hat| = 1.2281

What you see¶

Left panel: The two clusters are separated by the "smile" direction. The black arrow is $\hat{\delta}$, anchored at the mean of the neutral cluster — it points almost perfectly toward the smile cluster's mean.

Right panel: Steering trajectory. At $\alpha = 0$ we are at the base point; $\alpha = 1$ moves it to the "full smile" region; negative $\alpha$ moves it away from the smile concept. Beyond $\alpha \approx 1.5$ the steered point starts leaving the data distribution (out-of-distribution risk).

Key insight: The mean-difference estimator reliably recovers the concept direction from labelled examples — even with 40 noisy samples. The choice of $\alpha$ is a trade-off: too small = weak effect, too large = OOD.

Experiment 3: Linear Structure Test — Parallelism of Concept Directions¶

Question: How consistent is the concept direction $\delta_i$ across different example pairs? A high consistency (near-parallel $\delta$s) signals linear structure.

We compare two synthetic latent spaces:

• Linear space: concept encoded as a fixed additive offset + small noise.
• Nonlinear space: concept encoded as a rotation (nonlinear, context-dependent).

For each space we sample many "analogy pairs" $(a_i, b_i)$ and measure the pairwise cosine similarity of $\delta_i = z_{b_i} - z_{a_i}$.

In [4]:

np.random.seed(3)

n_pairs = 200
d = 10  # latent dimension

def pairwise_cos_sim(vecs: np.ndarray) -> np.ndarray:
    # returns upper-triangle cosine similarities
    norms = np.linalg.norm(vecs, axis=1, keepdims=True)
    vecs_n = vecs / (norms + 1e-9)
    gram = vecs_n @ vecs_n.T
    idx = np.triu_indices(len(vecs), k=1)
    return gram[idx]

# ── Linear space ──
concept_dir = np.random.randn(d)
concept_dir /= np.linalg.norm(concept_dir)

z_a_lin = np.random.randn(n_pairs, d)
# b = a + concept_dir + small noise
z_b_lin = z_a_lin + concept_dir[None, :] + np.random.randn(n_pairs, d) * 0.05
deltas_lin = z_b_lin - z_a_lin
cos_lin = pairwise_cos_sim(deltas_lin)

# ── Nonlinear space ──
# Each pair has a concept direction that depends on the base point (random rotation)
def random_rotation(d: int) -> np.ndarray:
    Q, _ = np.linalg.qr(np.random.randn(d, d))
    return Q

z_a_nl = np.random.randn(n_pairs, d)
z_b_nl = np.empty_like(z_a_nl)
for i in range(n_pairs):
    R = random_rotation(d)
    z_b_nl[i] = z_a_nl[i] + R @ concept_dir  # direction rotated per-sample
deltas_nl = z_b_nl - z_a_nl
cos_nl = pairwise_cos_sim(deltas_nl)

fig, axes = plt.subplots(1, 2, figsize=(12, 4), sharey=False)

kw = dict(bins=40, edgecolor="white", alpha=0.85)
axes[0].hist(cos_lin, color="#2166ac", **kw)
axes[0].axvline(cos_lin.mean(), color="red", lw=2, linestyle="--",
                label=f"mean = {cos_lin.mean():.3f}")
axes[0].set_xlabel("Pairwise cosine similarity of δ vectors")
axes[0].set_ylabel("Count")
axes[0].set_title("Linear space — high consistency")
axes[0].legend()
axes[0].set_xlim(-1, 1)

axes[1].hist(cos_nl, color="#d6604d", **kw)
axes[1].axvline(cos_nl.mean(), color="navy", lw=2, linestyle="--",
                label=f"mean = {cos_nl.mean():.3f}")
axes[1].set_xlabel("Pairwise cosine similarity of δ vectors")
axes[1].set_ylabel("Count")
axes[1].set_title("Nonlinear space — low consistency")
axes[1].legend()
axes[1].set_xlim(-1, 1)

fig.suptitle("Experiment 3 — linear structure test via δ-consistency", fontweight="bold")
plt.tight_layout()
plt.show()

print(f"Linear space   — mean pairwise cos-sim: {cos_lin.mean():.4f}  (std: {cos_lin.std():.4f})")
print(f"Nonlinear space— mean pairwise cos-sim: {cos_nl.mean():.4f}  (std: {cos_nl.std():.4f})")

np.random.seed(3) n_pairs = 200 d = 10 # latent dimension def pairwise_cos_sim(vecs: np.ndarray) -> np.ndarray: # returns upper-triangle cosine similarities norms = np.linalg.norm(vecs, axis=1, keepdims=True) vecs_n = vecs / (norms + 1e-9) gram = vecs_n @ vecs_n.T idx = np.triu_indices(len(vecs), k=1) return gram[idx] # ── Linear space ── concept_dir = np.random.randn(d) concept_dir /= np.linalg.norm(concept_dir) z_a_lin = np.random.randn(n_pairs, d) # b = a + concept_dir + small noise z_b_lin = z_a_lin + concept_dir[None, :] + np.random.randn(n_pairs, d) * 0.05 deltas_lin = z_b_lin - z_a_lin cos_lin = pairwise_cos_sim(deltas_lin) # ── Nonlinear space ── # Each pair has a concept direction that depends on the base point (random rotation) def random_rotation(d: int) -> np.ndarray: Q, _ = np.linalg.qr(np.random.randn(d, d)) return Q z_a_nl = np.random.randn(n_pairs, d) z_b_nl = np.empty_like(z_a_nl) for i in range(n_pairs): R = random_rotation(d) z_b_nl[i] = z_a_nl[i] + R @ concept_dir # direction rotated per-sample deltas_nl = z_b_nl - z_a_nl cos_nl = pairwise_cos_sim(deltas_nl) fig, axes = plt.subplots(1, 2, figsize=(12, 4), sharey=False) kw = dict(bins=40, edgecolor="white", alpha=0.85) axes[0].hist(cos_lin, color="#2166ac", **kw) axes[0].axvline(cos_lin.mean(), color="red", lw=2, linestyle="--", label=f"mean = {cos_lin.mean():.3f}") axes[0].set_xlabel("Pairwise cosine similarity of δ vectors") axes[0].set_ylabel("Count") axes[0].set_title("Linear space — high consistency") axes[0].legend() axes[0].set_xlim(-1, 1) axes[1].hist(cos_nl, color="#d6604d", **kw) axes[1].axvline(cos_nl.mean(), color="navy", lw=2, linestyle="--", label=f"mean = {cos_nl.mean():.3f}") axes[1].set_xlabel("Pairwise cosine similarity of δ vectors") axes[1].set_ylabel("Count") axes[1].set_title("Nonlinear space — low consistency") axes[1].legend() axes[1].set_xlim(-1, 1) fig.suptitle("Experiment 3 — linear structure test via δ-consistency", fontweight="bold") plt.tight_layout() plt.show() print(f"Linear space — mean pairwise cos-sim: {cos_lin.mean():.4f} (std: {cos_lin.std():.4f})") print(f"Nonlinear space— mean pairwise cos-sim: {cos_nl.mean():.4f} (std: {cos_nl.std():.4f})")

Linear space   — mean pairwise cos-sim: 0.9789  (std: 0.0098)
Nonlinear space— mean pairwise cos-sim: 0.0734  (std: 0.3083)

What you see¶

Linear space (left): All $\delta_i$ vectors cluster near cosine similarity $\approx 1$. They are nearly parallel because the concept is encoded as the same additive offset regardless of the base point — the hallmark of linear structure.

Nonlinear space (right): The $\delta_i$ vectors are almost uniformly distributed across $[-1, 1]$. The concept direction depends on the base point (context-dependent rotation), so different pairs point in completely different directions.

Key insight: A simple consistency check — compute $\delta$ over multiple example pairs and measure pairwise cosine similarity — is a practical diagnostic for linear structure. Mean cos-sim close to 1 → arithmetic is reliable; close to 0 → arithmetic is unreliable.

Experiment 4: Cross-Training-Run Breakdown¶

Question: Does arithmetic transfer between two models trained with different random seeds?

Two latent spaces that encode the same concepts will assign different absolute coordinates to them — the concept direction in model A points in an arbitrary direction relative to model B's axes. We simulate this by applying a random orthogonal transform to create a second "training run" and show that cross-space arithmetic fails.

In [5]:

np.random.seed(4)

d = 32
n = 100

# Model A: standard basis
concept_a = np.zeros(d); concept_a[0] = 1.0  # concept = first dimension
z_man_A  = np.random.randn(d) * 0.3
z_king_A = z_man_A + concept_a

# Model B: same concept but expressed in a *randomly rotated* coordinate system
Q, _ = np.linalg.qr(np.random.randn(d, d))  # random rotation = different random seed
concept_b = Q @ concept_a  # same concept, different coordinates

z_woman_B = np.random.randn(d) * 0.3         # encoded by model B
z_queen_B = z_woman_B + concept_b            # true queen in model B's coordinates

# Within-model arithmetic: delta from model A applied within model A
delta_A = z_king_A - z_man_A
z_queen_within = z_woman_B + concept_b       # apply B's concept to B's woman  (correct)
z_queen_cross  = z_woman_B + delta_A         # apply A's concept to B's woman  (wrong!)

err_within = np.linalg.norm(z_queen_within - z_queen_B)
err_cross  = np.linalg.norm(z_queen_cross  - z_queen_B)

# Repeat over many random seeds to get distributions
errs_within, errs_cross = [], []
for seed in range(500):
    rng = np.random.default_rng(seed)
    c = rng.standard_normal(d); c /= np.linalg.norm(c)

    z_a = rng.standard_normal(d) * 0.3
    z_b_same = z_a + c + rng.standard_normal(d) * 0.05      # within-model (slight noise)
    delta_same = z_b_same - z_a

    Q2, _ = np.linalg.qr(rng.standard_normal((d, d)))
    c_other = Q2 @ c                                          # rotated for other model
    z_c_other = rng.standard_normal(d) * 0.3
    z_d_true  = z_c_other + c_other                          # ground truth in other model

    errs_within.append(np.linalg.norm((z_c_other + delta_same) - z_d_true))  # cross
    errs_cross.append(np.linalg.norm((z_c_other + c_other)     - z_d_true))  # within (noise-only)

errs_within = np.array(errs_within)
errs_cross  = np.array(errs_cross)

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

kw = dict(bins=40, edgecolor="white", alpha=0.85)
axes[0].hist(errs_cross,  color="#2166ac", label="within-model (same coordinates)", **kw)
axes[0].hist(errs_within, color="#d6604d", label="cross-model  (different run)", **kw)
axes[0].set_xlabel("L2 error of arithmetic prediction")
axes[0].set_ylabel("Count")
axes[0].set_title("Error distribution: within vs cross model")
axes[0].legend(fontsize=9)

# Show as boxplot for clearer comparison
axes[1].boxplot([errs_cross, errs_within],
                labels=["Within-model", "Cross-model"],
                patch_artist=True,
                boxprops=dict(facecolor="#deebf7"),
                medianprops=dict(color="red", lw=2))
axes[1].set_ylabel("L2 error")
axes[1].set_title("Error comparison (500 random seed pairs)")

fig.suptitle("Experiment 4 — cross-training-run arithmetic failure", fontweight="bold")
plt.tight_layout()
plt.show()

print(f"Within-model error — mean: {errs_cross.mean():.4f}  std: {errs_cross.std():.4f}")
print(f"Cross-model  error — mean: {errs_within.mean():.4f}  std: {errs_within.std():.4f}")

np.random.seed(4) d = 32 n = 100 # Model A: standard basis concept_a = np.zeros(d); concept_a[0] = 1.0 # concept = first dimension z_man_A = np.random.randn(d) * 0.3 z_king_A = z_man_A + concept_a # Model B: same concept but expressed in a *randomly rotated* coordinate system Q, _ = np.linalg.qr(np.random.randn(d, d)) # random rotation = different random seed concept_b = Q @ concept_a # same concept, different coordinates z_woman_B = np.random.randn(d) * 0.3 # encoded by model B z_queen_B = z_woman_B + concept_b # true queen in model B's coordinates # Within-model arithmetic: delta from model A applied within model A delta_A = z_king_A - z_man_A z_queen_within = z_woman_B + concept_b # apply B's concept to B's woman (correct) z_queen_cross = z_woman_B + delta_A # apply A's concept to B's woman (wrong!) err_within = np.linalg.norm(z_queen_within - z_queen_B) err_cross = np.linalg.norm(z_queen_cross - z_queen_B) # Repeat over many random seeds to get distributions errs_within, errs_cross = [], [] for seed in range(500): rng = np.random.default_rng(seed) c = rng.standard_normal(d); c /= np.linalg.norm(c) z_a = rng.standard_normal(d) * 0.3 z_b_same = z_a + c + rng.standard_normal(d) * 0.05 # within-model (slight noise) delta_same = z_b_same - z_a Q2, _ = np.linalg.qr(rng.standard_normal((d, d))) c_other = Q2 @ c # rotated for other model z_c_other = rng.standard_normal(d) * 0.3 z_d_true = z_c_other + c_other # ground truth in other model errs_within.append(np.linalg.norm((z_c_other + delta_same) - z_d_true)) # cross errs_cross.append(np.linalg.norm((z_c_other + c_other) - z_d_true)) # within (noise-only) errs_within = np.array(errs_within) errs_cross = np.array(errs_cross) fig, axes = plt.subplots(1, 2, figsize=(12, 4)) kw = dict(bins=40, edgecolor="white", alpha=0.85) axes[0].hist(errs_cross, color="#2166ac", label="within-model (same coordinates)", **kw) axes[0].hist(errs_within, color="#d6604d", label="cross-model (different run)", **kw) axes[0].set_xlabel("L2 error of arithmetic prediction") axes[0].set_ylabel("Count") axes[0].set_title("Error distribution: within vs cross model") axes[0].legend(fontsize=9) # Show as boxplot for clearer comparison axes[1].boxplot([errs_cross, errs_within], labels=["Within-model", "Cross-model"], patch_artist=True, boxprops=dict(facecolor="#deebf7"), medianprops=dict(color="red", lw=2)) axes[1].set_ylabel("L2 error") axes[1].set_title("Error comparison (500 random seed pairs)") fig.suptitle("Experiment 4 — cross-training-run arithmetic failure", fontweight="bold") plt.tight_layout() plt.show() print(f"Within-model error — mean: {errs_cross.mean():.4f} std: {errs_cross.std():.4f}") print(f"Cross-model error — mean: {errs_within.mean():.4f} std: {errs_within.std():.4f}")

Within-model error — mean: 0.0000  std: 0.0000
Cross-model  error — mean: 1.5138  std: 0.1131

What you see¶

Within-model (blue): When we apply $\delta_A$ to another point in the same coordinate system, the error is tiny (only from small noise in the latent encoder).

Cross-model (red): When we apply $\delta_A$ from model A to a point from model B (with a random-rotation coordinate system), the error jumps dramatically — to the same order as the norm of the vectors themselves.

The boxplot makes the scale difference unmistakable.

Key insight: Latent vectors are absolute coordinates in a model-specific basis. A random orthogonal rotation (which has the same Euclidean geometry) creates a completely incompatible coordinate system. Cross-model arithmetic is equivalent to adding measurements in incompatible units — the result is meaningless.

Experiment 5: Concept Leakage from Non-Orthogonal Directions¶

Question: When two concept directions are correlated, how much does steering by concept A unintentionally shift the representation along concept B?

Steering by $\delta_1$ ideally moves the point only along the concept-1 axis. But if $\langle \delta_1, \delta_2 \rangle \neq 0$, the projection $\text{proj}_{\delta_2}(\delta_1) = \frac{\delta_1 \cdot \delta_2}{\|\delta_2\|^2}$ is non-zero, and any steering by $\alpha\,\delta_1$ leaks $\alpha \cdot \text{proj}$ into concept 2.

In [6]:

np.random.seed(5)

d = 128
alphas_test = np.linspace(0, 2, 41)
angles_deg  = [0, 15, 30, 45, 60, 75, 90]

def make_directions(d: int, angle_deg: float) -> tuple[np.ndarray, np.ndarray]:
    # delta1 = first standard basis vector (random)
    delta1 = np.zeros(d); delta1[0] = 1.0
    # delta2 = rotated by angle_deg in the 0-1 plane
    rad = np.radians(angle_deg)
    delta2 = np.zeros(d)
    delta2[0] = np.cos(rad)
    delta2[1] = np.sin(rad)
    return delta1, delta2

def leakage(delta1: np.ndarray, delta2: np.ndarray, alpha: float) -> float:
    # project alpha*delta1 onto delta2
    step = alpha * delta1
    return abs(np.dot(step, delta2) / np.linalg.norm(delta2))

# Compute leakage for each angle at alpha=1
leakage_at_alpha1 = []
for ang in angles_deg:
    d1, d2 = make_directions(d, ang)
    leakage_at_alpha1.append(leakage(d1, d2, 1.0))

# Compute leakage vs alpha for a few angles
fig, axes = plt.subplots(1, 2, figsize=(13, 5))

# Left: leakage at alpha=1 vs angle
ax = axes[0]
ax.plot(angles_deg, leakage_at_alpha1, "o-", color="#d6604d", lw=2, ms=8)
ax.set_xlabel("Angle between concept directions (degrees)")
ax.set_ylabel("Leakage (projection onto δ₂) at α = 1")
ax.set_title("Concept leakage vs. angle between directions")
ax.axhline(0, color="gray", lw=0.8, linestyle="--")
ax.set_xticks(angles_deg)

# Right: leakage vs alpha for several angles
ax = axes[1]
colors_map = plt.cm.RdBu_r(np.linspace(0.1, 0.9, len(angles_deg)))
for ang, col in zip(angles_deg, colors_map):
    d1, d2 = make_directions(d, ang)
    leaks = [leakage(d1, d2, a) for a in alphas_test]
    ax.plot(alphas_test, leaks, color=col, lw=1.8, label=f"{ang}°")

ax.set_xlabel("Steering strength α")
ax.set_ylabel("Leakage into concept 2")
ax.set_title("Leakage vs α for different inter-concept angles")
ax.legend(title="Angle", fontsize=8, ncol=2)

fig.suptitle("Experiment 5 — concept leakage from non-orthogonal directions", fontweight="bold")
plt.tight_layout()
plt.show()

print("Leakage (α=1) at each angle:")
for ang, lk in zip(angles_deg, leakage_at_alpha1):
    print(f"  {ang:3d}°  →  leakage = {lk:.4f}")

np.random.seed(5) d = 128 alphas_test = np.linspace(0, 2, 41) angles_deg = [0, 15, 30, 45, 60, 75, 90] def make_directions(d: int, angle_deg: float) -> tuple[np.ndarray, np.ndarray]: # delta1 = first standard basis vector (random) delta1 = np.zeros(d); delta1[0] = 1.0 # delta2 = rotated by angle_deg in the 0-1 plane rad = np.radians(angle_deg) delta2 = np.zeros(d) delta2[0] = np.cos(rad) delta2[1] = np.sin(rad) return delta1, delta2 def leakage(delta1: np.ndarray, delta2: np.ndarray, alpha: float) -> float: # project alpha*delta1 onto delta2 step = alpha * delta1 return abs(np.dot(step, delta2) / np.linalg.norm(delta2)) # Compute leakage for each angle at alpha=1 leakage_at_alpha1 = [] for ang in angles_deg: d1, d2 = make_directions(d, ang) leakage_at_alpha1.append(leakage(d1, d2, 1.0)) # Compute leakage vs alpha for a few angles fig, axes = plt.subplots(1, 2, figsize=(13, 5)) # Left: leakage at alpha=1 vs angle ax = axes[0] ax.plot(angles_deg, leakage_at_alpha1, "o-", color="#d6604d", lw=2, ms=8) ax.set_xlabel("Angle between concept directions (degrees)") ax.set_ylabel("Leakage (projection onto δ₂) at α = 1") ax.set_title("Concept leakage vs. angle between directions") ax.axhline(0, color="gray", lw=0.8, linestyle="--") ax.set_xticks(angles_deg) # Right: leakage vs alpha for several angles ax = axes[1] colors_map = plt.cm.RdBu_r(np.linspace(0.1, 0.9, len(angles_deg))) for ang, col in zip(angles_deg, colors_map): d1, d2 = make_directions(d, ang) leaks = [leakage(d1, d2, a) for a in alphas_test] ax.plot(alphas_test, leaks, color=col, lw=1.8, label=f"{ang}°") ax.set_xlabel("Steering strength α") ax.set_ylabel("Leakage into concept 2") ax.set_title("Leakage vs α for different inter-concept angles") ax.legend(title="Angle", fontsize=8, ncol=2) fig.suptitle("Experiment 5 — concept leakage from non-orthogonal directions", fontweight="bold") plt.tight_layout() plt.show() print("Leakage (α=1) at each angle:") for ang, lk in zip(angles_deg, leakage_at_alpha1): print(f" {ang:3d}° → leakage = {lk:.4f}")

Leakage (α=1) at each angle:
    0°  →  leakage = 1.0000
   15°  →  leakage = 0.9659
   30°  →  leakage = 0.8660
   45°  →  leakage = 0.7071
   60°  →  leakage = 0.5000
   75°  →  leakage = 0.2588
   90°  →  leakage = 0.0000

What you see¶

Left panel: Leakage is $\cos(\theta)$ — exactly the geometric projection of $\delta_1$ onto $\delta_2$. At $90°$ the directions are orthogonal and leakage is zero. At $0°$ (same direction) every unit of steering fully contaminates the second concept.

Right panel: Leakage grows linearly with $\alpha$ (the steering strength) — so a strong intervention pollutes the second concept proportionally more.

Key insight: For arithmetic to cleanly separate concepts, concept directions should ideally be orthogonal. In practice, check $\langle \delta_1, \delta_2 \rangle$ before stacking multiple steers. If two concepts are highly correlated (e.g., "royal" and "formal attire"), disentanglement is necessary to steer them independently.

Summary — Key Takeaways¶

Experiment	Key finding
1. Analogy arithmetic	The four points form a parallelogram: $z_d = z_c + (z_b - z_a)$ is exact when structure is perfectly linear.
2. Concept direction & steering	Mean-difference $\hat{\delta} = \bar{z}_{+} - \bar{z}_{-}$ reliably recovers concept directions. $\alpha > 1.5\|\delta\|$ risks OOD.
3. Linear structure test	Pairwise cosine-similarity of $\delta_i$ vectors diagnoses linearity. Near-1 = safe to use arithmetic.
4. Cross-model breakdown	Random-seed differences create incompatible coordinate systems. Cross-model L2 error is ~10× within-model error.
5. Concept leakage	Leakage = $\cos(\theta)$ between directions. Orthogonal directions ($90°$) give zero leakage; aligned directions give full cross-contamination.