How High-Dimensional Mathematics Rules Our World

Benjamin Skuse

Intuitively, we can picture a one-dimensional entity, eternally restricted to walking an infinite line, or a two-dimensional being resigned to life on a flat plane. We do not have to picture three-dimensional beings, as that is how we experience our universe. Yet, the computational power driving the modern world thrives in abstract spaces of five, 10, or even thousands of dimensions. How does high-dimensional mathematics allow us to process and interpret information, and reveal hidden patterns that govern everything from our biology to artificial intelligence?

Dimensions Beyond Perception

A good starting point is to separate the dimensions of the universe with other definitions of dimensions. If we are considering the former, a central question is: are we living in a truly three-dimensional universe or is this just a quirk of our perception? Left to right, forward and back, up and down. These are the only ways in which we can move, and the reason we perceive a three-dimensional world. But then came Albert Einstein. His special and general relativity brought time into play. Though we can only perceive time as running forwards, and is therefore experienced in a very different way to the three spatial dimensions (and, indeed, is mathematically different), relativity bundled space and time together as spacetime.

Spacetime became mathematically concrete thanks to mathematician Hermann Minkowski, who demonstrated that special relativity could be elegantly described by treating space and time on an almost equal footing. This unified view introduced the concept of the Minkowski space, a four-dimensional space where time is treated as the fourth coordinate.

This work laid the foundation for treating spacetime as a single four-dimensional fabric (three spatial dimensions plus one temporal dimension), known as a manifold in mathematical parlance, that is not immutable but can be affected by mass or energy. Space literally stretches out, as does time, the closer you get to a massive object like a black hole. In other words, the dimensions of the four-dimensional manifold become geometrically distorted.

The revelations that relativity offered over a century ago exposed the public to the possibility that we live in a universe of more than three dimensions and provided a visceral way to understand that what we perceive is not necessarily all that exists. But further revelations were ahead.

Soon after Einstein had given the world general relativity, German mathematician Theodor Kaluza and Swedish physicist Oskar Klein realised something was missing: another spatial dimension. If the universe were five-dimensional instead of four-dimensional, the pair demonstrated that the five-dimensional version of Einstein’s equations would split into three sets of four-dimensional equations: Einstein’s original field equations for gravity; James Clerk Maxwell’s equations for electromagnetism; and a new equation for a scalar field.

This fourth spatial dimension could be curled up into a tiny, closed loop with a radius far smaller than any measurable distance, explaining why we cannot perceive it. But at the same time, it could have a monumental impact on the world around us; the secret sauce that could unify gravity and electromagnetism.

Kaluza–Klein theory, as it became known, was brilliant mathematically, but flawed physically. It failed to predict particle properties correctly and missed the other fundamental forces (strong and weak nuclear force) as well as many fundamental particles. However, it was the first serious attempt to show that the fundamental forces we experience might simply be manifestations of higher-dimensional geometry. And its legacy is felt today in its descendants that attempt to unify the fundamental forces and particles using multiple curled up spatial dimensions: string theory and M-theory; the latter first proposed by 1990 Fields Medallist Edward Witten.

Four-dimensional general relativity, five-dimensional Kaluza–Klein theory, and 10- and 11-dimensional string and M-theories have hinted that the geometric structure of our universe may be far more exotic than what we can perceive. But these are all at the lower end of the high-dimensional spectrum that mathematicians and statisticians deal with. Why would anyone want to venture into realms of such deep abstraction?

Defining Data

Understanding the purpose of high-dimensional mathematics becomes easier when we describe the position of a point in a given \(N\)-dimensional space. Our one-dimensional being walking its infinite line can be located by a single coordinate \((x)\), our flatlander is found with just two coordinates \((x,y)\), and you or I can be pinpointed with just three coordinates \((x,y,z)\) or four \((x,y,z,t)\) if considering where and when we are in spacetime.

Points in higher dimensions just have more coordinates. This becomes particularly useful when we are no longer considering the dimensions of the universe and define dimensionality in different ways. Instead of thinking of dimensions as the multiple facets of what could be reality, we might define dimensionality as how many attributes or variables are considered within a given space. For example, a financial modeller might want to track and predict the risk involved in a given asset. To do this, the asset can be considered a point in an \(N\)-dimensional risk space, where \(N\) refers to variables such as current price, volatility, interest rate, etc.

Another good example is census data. A government census database will contain hundreds of variables on people (age, sex, ethnicity, occupation, etc) and households (accommodation type, tenure, number of bedrooms, etc). A person then becomes a single point in the \(N\)-dimensional space that is the database, where \(N\) refers to the hundreds of different characteristics that were measured during the census.

In both cases, it is in spotting and analysing the lower-dimensional patterns and shapes embedded within the high-dimensional data where insights are found. The financial modeller can run algorithms to search the high-dimensional risk space for a lower-dimensional hyperplane that optimally separates safe investments from risky ones. Or a statistician might group the original variables into principal components that represent the main differences between communities in terms of socioeconomic status to build a deprivation index based on region.

Another (current) example of the importance of high-dimensional mathematics is large language models (LLMs), which build on decades of research in AI, machine learning and natural language processing by researchers such as Yoshua Bengio and Yann LeCun (both recipients of the 2018 ACM A.M. Turing Award). In fact, LLMs could not function without high-dimensional mathematics.

These models process text entirely through vector mathematics. Every ‘token’ (word, subword or punctuation) is converted into a high-dimensional vector, typically between 512 and 4096 dimensions, and the token’s position in the sequence (sentence) is encoded as an additional vector. Computations are then primarily driven by the self-attention mechanism, which calculates the semantic relationships between all tokens by measuring the dot product of their high-dimensional vectors, producing successive new sets of high-dimensional vectors that encode the context of the given sequence. The final output vector is projected into the vocabulary space, whose dimension is the number of possible tokens. And finally, a function is applied that selects a token and generates text.

The Shape of Complexity

Analysis and insight really start to get complicated when large datasets include data points that themselves are high-dimensional vectors. For example, in single-cell RNA sequencing, each single cell is represented by a vector whose dimensions correspond to the expression level of tens of thousands of genes. To make any sense of such a vast, sparse space requires a different approach.

Treating the space as a huge data cloud and ignoring the specific coordinates within the data offers the opportunity to take a step back and take in the ‘shape’ of the data. Known as topological data analysis, this approach characterises the global structure and connectivity of the high-dimensional data manifold. This type of analysis shares roots with modern topology – from 1982 Fields Medallist Shing-Tung Yau’s geometric analysis of curved spaces to 1986 Fields Medallist Michael Freedman’s insights into the structure of manifolds.

Topological data analysis identifies topological features (properties of the data’s shape that remain unchanged even when the data is stretched, compressed or continuously transformed) using persistent homology to quantify structures that persist across different scales. These structures include clusters, loops, and voids, the presence of which signals deep insights that would otherwise be missed.

A particularly impactful application of this analysis technique has been in cancer genomics. Topological data analysis has been used to identify hidden clusters of breast cancer patients with a specific prognosis that other methods have missed, allowing tailored treatment. It has also been wielded to identify genomic markers that can be used to predict treatment responses and estimate patient prognosis with high accuracy.

Infinite Full Circle

Going beyond this already exceedingly high dimensionality brings us back to where we started: the very nature of reality and the universe. Alongside the formulation of relativity, Einstein was also a key founder of another pillar of modern physics: quantum mechanics. Quantum mechanics describes the behaviour of matter and light at the atomic and subatomic scales. At this level, the mathematical description requires continuity across the spatial domain and is governed by the principle of superposition; that a physical system, such as a wave or a quantum particle, can exist in a combination of all its possible states simultaneously until the moment of measurement.

These factors lead to an infinite number of potential degrees of freedom, or dimensions. And this complexity necessitates the use of functional analysis, the branch of mathematics that treats functions as points or vectors in an infinite-dimensional space, and the Hilbert space, a special infinite-dimensional vector space where quantum states live. In fact, the infinite-dimensional Hilbert-space framework, formalised rigorously by celebrated polymath John von Neumann and later generalised through the work of famed mathematician Israel Gelfand and 1962 Fields Medallist Lars Hörmander, provides the mathematical backbone of quantum mechanics. Without this and functional analysis, it would be impossible to precisely define quantum states, quantify probabilities, and describe the continuous dynamics of the subatomic world.

As we have seen, the limitations of our everyday perception are the starting line for true insight. We are no longer confined to observing a three-dimensional world, but possess the mathematical language to build and navigate spaces of arbitrary dimensions, exposing new and deep understanding in a huge range of fields and applications.

The post How High-Dimensional Mathematics Rules Our World originally appeared on the HLFF SciLogs blog.