UMAP Theory — Visual Experiments¶

Goal: Build intuition for how UMAP turns neighbourhood structure into a 2D map, and what its main knobs really do.

What we'll explore¶

#	Experiment	Question answered
1	n_neighbors trade-off	How does UMAP move from local detail to broader manifold structure?
2	min_dist packing	How tightly is UMAP allowed to compress clusters in the embedding?
3	UMAP vs t-SNE	What does it mean to say UMAP often keeps more coarse global structure?
4	Metric choice	Why can cosine and euclidean produce different stories from the same latent cloud?

---

Linked theory: research/05-umap-theory.md

In [1]:

import numpy as np
import matplotlib.pyplot as plt
import umap
from sklearn.manifold import TSNE
from sklearn.datasets import make_swiss_roll, make_blobs
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')

np.random.seed(42)
plt.rcParams.update({
    'figure.dpi': 110,
    'font.size': 11,
    'axes.titlesize': 12,
    'axes.labelsize': 11,
    'figure.facecolor': 'white',
    'axes.facecolor': '#f8f9fa',
    'axes.grid': True,
    'grid.alpha': 0.3,
})
print('Setup complete.')

import numpy as np import matplotlib.pyplot as plt import umap from sklearn.manifold import TSNE from sklearn.datasets import make_swiss_roll, make_blobs from sklearn.preprocessing import StandardScaler import warnings warnings.filterwarnings('ignore') np.random.seed(42) plt.rcParams.update({ 'figure.dpi': 110, 'font.size': 11, 'axes.titlesize': 12, 'axes.labelsize': 11, 'figure.facecolor': 'white', 'axes.facecolor': '#f8f9fa', 'axes.grid': True, 'grid.alpha': 0.3, }) print('Setup complete.')

/home/runner/work/latent-anything/latent-anything/latent-anything-theory/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Setup complete.

---

Experiment 1: n_neighbors controls the local/global trade-off¶

Question: UMAP starts from local neighbourhoods. What changes when we ask it to define 'local' using 5 neighbours versus 80 neighbours?

We use a noisy swiss-roll manifold. Points are coloured by their intrinsic roll coordinate, so preserving the smooth colour progression means preserving manifold order.

In [2]:

np.random.seed(7)
X_swiss, t_swiss = make_swiss_roll(n_samples=1400, noise=0.05, random_state=7)
X_swiss = StandardScaler().fit_transform(X_swiss)
neighbors_list = [5, 20, 80]
embeddings_neighbors = []
for n in neighbors_list:
    reducer = umap.UMAP(n_neighbors=n, min_dist=0.1, metric='euclidean', random_state=7)
    embeddings_neighbors.append(reducer.fit_transform(X_swiss))
print('Built', len(embeddings_neighbors), 'UMAP embeddings for different n_neighbors values.')

np.random.seed(7) X_swiss, t_swiss = make_swiss_roll(n_samples=1400, noise=0.05, random_state=7) X_swiss = StandardScaler().fit_transform(X_swiss) neighbors_list = [5, 20, 80] embeddings_neighbors = [] for n in neighbors_list: reducer = umap.UMAP(n_neighbors=n, min_dist=0.1, metric='euclidean', random_state=7) embeddings_neighbors.append(reducer.fit_transform(X_swiss)) print('Built', len(embeddings_neighbors), 'UMAP embeddings for different n_neighbors values.')

Built 3 UMAP embeddings for different n_neighbors values.

In [3]:

fig, axes = plt.subplots(1, 3, figsize=(15.5, 4.8))
fig.suptitle('UMAP local/global trade-off via n_neighbors', fontsize=13, fontweight='bold')
for ax, emb, n in zip(axes, embeddings_neighbors, neighbors_list):
    sc = ax.scatter(emb[:, 0], emb[:, 1], c=t_swiss, s=10, cmap='viridis')
    ax.set_title(f'n_neighbors = {n}')
    ax.set_xlabel('UMAP-1')
    ax.set_ylabel('UMAP-2')
cbar = fig.colorbar(sc, ax=axes.ravel().tolist(), shrink=0.9)
cbar.set_label('intrinsic roll coordinate')
plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 3, figsize=(15.5, 4.8)) fig.suptitle('UMAP local/global trade-off via n_neighbors', fontsize=13, fontweight='bold') for ax, emb, n in zip(axes, embeddings_neighbors, neighbors_list): sc = ax.scatter(emb[:, 0], emb[:, 1], c=t_swiss, s=10, cmap='viridis') ax.set_title(f'n_neighbors = {n}') ax.set_xlabel('UMAP-1') ax.set_ylabel('UMAP-2') cbar = fig.colorbar(sc, ax=axes.ravel().tolist(), shrink=0.9) cbar.set_label('intrinsic roll coordinate') plt.tight_layout() plt.show()

What you see¶

1. What the plot shows: With n_neighbors=5, the map emphasises tiny local fragments; with 20, the roll order becomes smoother; with 80, the embedding sacrifices some small bends to preserve the broad progression of the manifold.
2. Why it looks this way: n_neighbors decides how large a neighbourhood each point uses to estimate the manifold. Small values trust only very local geometry; large values average over wider regions and therefore stabilise coarse structure.
3. Key takeaway: When people say UMAP can preserve more global structure than t-SNE, this is usually the mechanism they mean: you can explicitly widen the neighbourhood scale with n_neighbors, but you pay by losing fine detail.

---

Experiment 2: min_dist controls packing and cluster compactness¶

Question: Once UMAP knows which points should stay nearby, how tightly is it allowed to squeeze them together in the low-dimensional map?

We use moderately overlapping blobs so the effect of cluster packing is easy to see.

In [4]:

np.random.seed(11)
X_blobs, y_blobs = make_blobs(
    n_samples=1300,
    centers=[(-4, -2), (-1, 2), (2, -1), (5, 3)],
    cluster_std=[0.9, 1.1, 0.8, 1.2],
    n_features=6,
    random_state=11,
)
X_blobs = StandardScaler().fit_transform(X_blobs)
min_dist_list = [0.0, 0.2, 0.8]
embeddings_mindist = []
for d in min_dist_list:
    reducer = umap.UMAP(n_neighbors=25, min_dist=d, metric='euclidean', random_state=11)
    embeddings_mindist.append(reducer.fit_transform(X_blobs))
print('Computed embeddings for min_dist values:', min_dist_list)

np.random.seed(11) X_blobs, y_blobs = make_blobs( n_samples=1300, centers=[(-4, -2), (-1, 2), (2, -1), (5, 3)], cluster_std=[0.9, 1.1, 0.8, 1.2], n_features=6, random_state=11, ) X_blobs = StandardScaler().fit_transform(X_blobs) min_dist_list = [0.0, 0.2, 0.8] embeddings_mindist = [] for d in min_dist_list: reducer = umap.UMAP(n_neighbors=25, min_dist=d, metric='euclidean', random_state=11) embeddings_mindist.append(reducer.fit_transform(X_blobs)) print('Computed embeddings for min_dist values:', min_dist_list)

Computed embeddings for min_dist values: [0.0, 0.2, 0.8]

In [5]:

fig, axes = plt.subplots(1, 3, figsize=(15.5, 4.8))
fig.suptitle('UMAP packing behaviour via min_dist', fontsize=13, fontweight='bold')
for ax, emb, d in zip(axes, embeddings_mindist, min_dist_list):
    ax.scatter(emb[:, 0], emb[:, 1], c=y_blobs, s=11, cmap='plasma')
    ax.set_title(f'min_dist = {d}')
    ax.set_xlabel('UMAP-1')
    ax.set_ylabel('UMAP-2')
plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 3, figsize=(15.5, 4.8)) fig.suptitle('UMAP packing behaviour via min_dist', fontsize=13, fontweight='bold') for ax, emb, d in zip(axes, embeddings_mindist, min_dist_list): ax.scatter(emb[:, 0], emb[:, 1], c=y_blobs, s=11, cmap='plasma') ax.set_title(f'min_dist = {d}') ax.set_xlabel('UMAP-1') ax.set_ylabel('UMAP-2') plt.tight_layout() plt.show()

What you see¶

1. What the plot shows: min_dist=0.0 produces tight, almost clumpy islands; 0.2 keeps clusters separated but less compressed; 0.8 spreads clusters out into softer shapes.
2. Why it looks this way: min_dist limits how close points may be in the embedding. Lower values let UMAP collapse dense regions into compact groups; larger values force more breathing room.
3. Key takeaway: min_dist changes the appearance of cluster sharpness. A very crisp UMAP map does not automatically mean the original latent space contained perfectly discrete clusters.

---

Experiment 3: UMAP vs t-SNE on a hierarchical toy dataset¶

Question: What does the oft-repeated claim 'UMAP preserves more global structure' actually look like in a controlled setting?

We build four clusters arranged as two super-groups in 12 dimensions. A useful 2D embedding should show both the individual clusters and the larger two-group structure.

In [6]:

np.random.seed(21)
centers = np.array([
    [-6, -1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
    [-2,  1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
    [ 3,  5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
    [ 7,  7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
])
X_hier, y_hier = make_blobs(n_samples=1600, centers=centers, cluster_std=0.9, random_state=21)
X_hier = StandardScaler().fit_transform(X_hier)
umap_hier = umap.UMAP(n_neighbors=50, min_dist=0.15, metric='euclidean', random_state=21).fit_transform(X_hier)
tsne_hier = TSNE(n_components=2, perplexity=35, init='pca', learning_rate='auto', random_state=21).fit_transform(X_hier)
print('Computed one UMAP and one t-SNE embedding for the same hierarchical dataset.')

np.random.seed(21) centers = np.array([ [-6, -1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [-2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 3, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [ 7, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], ]) X_hier, y_hier = make_blobs(n_samples=1600, centers=centers, cluster_std=0.9, random_state=21) X_hier = StandardScaler().fit_transform(X_hier) umap_hier = umap.UMAP(n_neighbors=50, min_dist=0.15, metric='euclidean', random_state=21).fit_transform(X_hier) tsne_hier = TSNE(n_components=2, perplexity=35, init='pca', learning_rate='auto', random_state=21).fit_transform(X_hier) print('Computed one UMAP and one t-SNE embedding for the same hierarchical dataset.')

Computed one UMAP and one t-SNE embedding for the same hierarchical dataset.

In [7]:

fig, axes = plt.subplots(1, 2, figsize=(11.5, 4.8))
fig.suptitle('Hierarchical clusters: UMAP versus t-SNE', fontsize=13, fontweight='bold')
for ax, emb, title in zip(axes, [umap_hier, tsne_hier], ['UMAP', 't-SNE']):
    ax.scatter(emb[:, 0], emb[:, 1], c=y_hier, cmap='tab10', s=10)
    ax.set_title(title)
    ax.set_xlabel('component 1')
    ax.set_ylabel('component 2')
plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(11.5, 4.8)) fig.suptitle('Hierarchical clusters: UMAP versus t-SNE', fontsize=13, fontweight='bold') for ax, emb, title in zip(axes, [umap_hier, tsne_hier], ['UMAP', 't-SNE']): ax.scatter(emb[:, 0], emb[:, 1], c=y_hier, cmap='tab10', s=10) ax.set_title(title) ax.set_xlabel('component 1') ax.set_ylabel('component 2') plt.tight_layout() plt.show()

What you see¶

1. What the plot shows: Both methods separate the four clusters. In this deterministic toy, UMAP more clearly retains the two-super-group layout, while t-SNE focuses on making each cluster locally clean and may equalise gaps between groups.
2. Why it looks this way: UMAP optimises a fuzzy neighbourhood graph whose effective scale is tunable through n_neighbors, so coarse neighbourhood relations can survive. t-SNE is excellent at local separation, but the distances between far-away clusters are less stable and less directly interpretable.
3. Key takeaway: 'UMAP preserves more global structure' should be read as a practical tendency, not a theorem. The right conclusion is comparative and dataset-specific: inspect whether super-cluster relationships survive, rather than trusting the map by default.

---

Experiment 4: Metric choice changes the story¶

Question: If two latent vectors point in the same direction but have different norms, should they be considered similar? The answer depends on the metric.

We generate direction-based data with random magnitudes. Cosine should group by direction; Euclidean will also care about norm.

In [8]:

np.random.seed(31)
dim = 20
proto = np.array([
    np.r_[np.ones(5), np.zeros(dim - 5)],
    np.r_[np.zeros(5), np.ones(5), np.zeros(dim - 10)],
    np.r_[np.zeros(10), np.ones(5), np.zeros(dim - 15)],
])
proto = proto / np.linalg.norm(proto, axis=1, keepdims=True)
X_metric = []
y_metric = []
for k, p in enumerate(proto):
    for _ in range(260):
        scale = np.random.uniform(0.4, 3.0)
        noise = 0.08 * np.random.randn(dim)
        X_metric.append(scale * p + noise)
        y_metric.append(k)
X_metric = np.array(X_metric)
y_metric = np.array(y_metric)
emb_euclid = umap.UMAP(n_neighbors=25, min_dist=0.15, metric='euclidean', random_state=31).fit_transform(X_metric)
emb_cosine = umap.UMAP(n_neighbors=25, min_dist=0.15, metric='cosine', random_state=31).fit_transform(X_metric)
print('Built metric-sensitive embeddings for euclidean and cosine distance.')

np.random.seed(31) dim = 20 proto = np.array([ np.r_[np.ones(5), np.zeros(dim - 5)], np.r_[np.zeros(5), np.ones(5), np.zeros(dim - 10)], np.r_[np.zeros(10), np.ones(5), np.zeros(dim - 15)], ]) proto = proto / np.linalg.norm(proto, axis=1, keepdims=True) X_metric = [] y_metric = [] for k, p in enumerate(proto): for _ in range(260): scale = np.random.uniform(0.4, 3.0) noise = 0.08 * np.random.randn(dim) X_metric.append(scale * p + noise) y_metric.append(k) X_metric = np.array(X_metric) y_metric = np.array(y_metric) emb_euclid = umap.UMAP(n_neighbors=25, min_dist=0.15, metric='euclidean', random_state=31).fit_transform(X_metric) emb_cosine = umap.UMAP(n_neighbors=25, min_dist=0.15, metric='cosine', random_state=31).fit_transform(X_metric) print('Built metric-sensitive embeddings for euclidean and cosine distance.')

Built metric-sensitive embeddings for euclidean and cosine distance.

In [9]:

fig, axes = plt.subplots(1, 2, figsize=(11.5, 4.8))
fig.suptitle('Metric choice changes neighbourhoods', fontsize=13, fontweight='bold')
for ax, emb, title in zip(axes, [emb_euclid, emb_cosine], ['metric = euclidean', 'metric = cosine']):
    ax.scatter(emb[:, 0], emb[:, 1], c=y_metric, cmap='coolwarm', s=11)
    ax.set_title(title)
    ax.set_xlabel('UMAP-1')
    ax.set_ylabel('UMAP-2')
plt.tight_layout()
plt.show()

fig, axes = plt.subplots(1, 2, figsize=(11.5, 4.8)) fig.suptitle('Metric choice changes neighbourhoods', fontsize=13, fontweight='bold') for ax, emb, title in zip(axes, [emb_euclid, emb_cosine], ['metric = euclidean', 'metric = cosine']): ax.scatter(emb[:, 0], emb[:, 1], c=y_metric, cmap='coolwarm', s=11) ax.set_title(title) ax.set_xlabel('UMAP-1') ax.set_ylabel('UMAP-2') plt.tight_layout() plt.show()

What you see¶

1. What the plot shows: With Euclidean distance, some points separate because norm differences matter; with cosine distance, groups align more cleanly by direction.
2. Why it looks this way: UMAP builds its neighbourhood graph before it optimises the embedding. If the graph uses Euclidean distance, vector length affects who counts as a neighbour. If it uses cosine, angular similarity dominates.
3. Key takeaway: A UMAP map is never 'metric-free'. In latent analysis, choosing the wrong metric can create or erase apparent clusters before visualisation even begins.

---

Summary¶

Experiment	Key insight
1. n_neighbors	This is the main local/global knob: small values preserve detail, large values stabilise coarse geometry.
2. min_dist	This changes visual compactness, so apparent cluster sharpness is partly a display choice.
3. UMAP vs t-SNE	UMAP often retains super-structure better, but the claim is comparative and dataset-specific, not absolute.
4. Metric choice	The neighbourhood graph depends on the metric, so the embedding inherits that assumption.

Connection to the broader theory¶

UMAP is the unsupervised counterpart to probing: instead of asking which attribute is decodable?, it asks what geometric neighbourhood structure becomes visible if we compress the latent space? In Layer A of Latent-Anything, UMAP is therefore an exploratory lens — great for spotting clusters, drift, and outliers before we reach for probes or causal interventions.