Attention as a learned geometry

Research status. This note uses geometry as a precise mathematical interpretation of representation relationships, not as a claim that a transformer contains a literal physical space.

Attention begins with a bilinear comparison

For an input representation matrix $X$ , a transformer constructs

Q=XW_Q, \qquad K=XW_K, \qquad V=XW_V

and computes scaled dot-product attention:

\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\!\left(\frac{QK^{\mathsf T}}{\sqrt{d_k}}\right)V

The matrix $QK^{\mathsf T}$ contains pairwise, learned compatibility scores. Unlike a fixed Euclidean distance, those comparisons depend on the trained projections and on the role assigned to each token as query or key.

A data-dependent relational geometry

If vectors are normalized, a dot product is related to angular similarity:

\widehat q_i^{\mathsf T}\widehat k_j=\cos\theta_{ij}

But attention is generally asymmetric because query and key projections differ. It is therefore safer to call the score matrix a learned relational geometry rather than a metric: it need not satisfy symmetry, positivity or the triangle inequality.

The softmax converts scores into row-wise weights

A_{ij} = \frac{\exp(s_{ij})}{\sum_m\exp(s_{im})}

so each output vector becomes a context-dependent mixture

y_i=\sum_j A_{ij}v_j

Transformer attention diagram showing Q, K and V projections, the attention-score matrix, softmax normalization, a weighted relational graph and contextual outputs.

Scaled query-key comparisons become normalized attention weights and then contextual mixtures of value vectors. The resulting relation is learned and directional, so it is not necessarily a mathematical metric.

What is actually learned

Different heads can construct different relational structures over the same sequence. One may emphasize syntax, another positional regularity, and another long-range identity. The model does not discover one universal semantic map; it learns multiple operators useful for its objective.

This viewpoint also clarifies why attention weights alone are not a complete explanation. Values, residual streams, nonlinear layers and later transformations determine how the weighted relation affects the computation.

A productive research question

The rigorous extension is to study which geometric properties emerge in attention operators: effective rank, anisotropy, spectral gaps, curvature-like quantities on induced graphs and stability under perturbation. These can be measured rather than inferred from visualization alone.

Primary starting point

Ashish Vaswani et al., Attention Is All You Need.

The useful thesis is limited but strong: attention repeatedly constructs task-dependent relational operators, and geometric language becomes valuable when its mathematical conditions are stated rather than merely suggested.