This is a hypothesis and not an established result.

To the most philosophical and technical, discovering mathematical laws for understanding and influencing the world seems like a plausible trajectory which can be used for better AI alignment, as encoding this mathematically can help in AI models better understanding human foundations, and values., to quote Richard Ngo’s ‘Towards a scale-free theory of intelligent agency’ . I believe that this model of intelligence even if constructed will itself depends on representational assumptions that constrain the forms of intelligence it can describe. I want to make this as a position of my theory of intelligence which includes the current paradigm of intelligent LLM agents and also why the ‘mathematical laws’ theory will also be incomplete due to it’s dependence on geometry.

In this post I would like to clarify my position and also make arguments for a ‘radical’ heading, my contributions are hence as follows:

creativity is often described as thinking out of box. Geometry restrict LLMs to explore and be creative.
How current LLM agents can be made creative which might require the ‘The Most Forbidden Technique’.

Current LLMs and Latent Space

The space of representations in LLMs which they form under optimisation helps them in forming sentences and gives them the ability to construct a conversation with great efficiency to the point that they can be extremely persuasive about their arguments, while much has been said about the latent space, and much research has been done on LLMs representations, neural geometry, this space is inherently a constraint to creativity, LLMs basic flow of constructing a sentence requires prefill and decoding as the two main steps, this way of constructing meaning or understanding meaning where every new words and any experience related to those words will be attracted to the manifold in the basin which seems most likely to hold the most similar representations while that can be extremely helpful in conversations and the vast ways in which LLMs can stastically produce coherent sentences, this geometrical space when optimised to predict the next word can only constrain the models into always constructing the sentence with respect to the past and the events of the past and their details, it cannot construct all-novel ideas. While some pushback is deserved here from people in light of recent events where GPT 5.6 being able to solve 100 year old problems, and various other events, discovery requires something much more than statistical correlations and the ability to manipulate language.Recent examples of LLMs making apparently novel mathematical discoveries provide an important challenge to the strongest version of this hypothesis. The question is whether such discoveries can be shown to require the construction of representations that were not reachable under the model’s existing geometry.

Why Geometry is a constraint to Intelligence?

Let’s took a look at an example. Let’s say there is a prompt P talking about a concept/idea X, when the user prompts the model, the model does the following:

a) The tokenizer T converts X into n tokens, which get converted to embeddings E

where

is the tokenizer, mapping a text X from the space of possible texts X to a finite sequence of tokens from vocabulary V.

The token sequence is then mapped to embeddings:

More formally, if

is the embedding function, then

So the complete transformation is:

or, composing the two:

b) Then the first geometric constraint appears when the discrete token sequence is embedded into a finite dimensional space.

c) The model subsequently performs it’s computations on these embeddings

Thus although there can be rich semantic structure in the original concept X, the model only grasps the correlations of that semantics, and not the originality axis of the concept which gets lost in transalation. My argument is that an agent to be creative it needs to look far beyond it’s own representations that it has formed, let’s take art as an example and the famous Will Smith line from movie “Can a Robot write a symphony?”: Although AI can write a symphony, and can create paintings which resemble the great painters Van Gogh, etc. It is still unable to create something which we have not seen, the essential wow factor that there can be, a pushback I expect on this is the fact that AI can create new things, this statement is often said when it creates a sum of different parts and that gets labelled as ‘novel’, a great Neil deGrasse Tyson quote that I once came across was that what if Isaac Newton was not born, will we have not discovered gravity? we would have as it is inherently present in the nature, it was not created, it was discovered, but what if Van Gogh was not born, would we have seen ‘Starry Night’ we would have not, we might have seen rendition of it, but it is not the same and that is where creativity lies, in novelty.

Concept of creativity and exploration

From the above two paragraphs, we can see that while LLMs can form intermediate representations they cannot influence their representational geometry which when optimised has very strong attractor basins, and therefore I would like to differentiate exploration in-manifold, from creative exploration. An intelligent agent can and will be able to search through possible different regions of the manifold and form coherent conversations and solutions. I hypothesise that creating novelty requires the ability to recogonise that the current sum of concepts in a concept C is not ideal, hence it requires the model to shift it’s representational manifold followed by exploration inside this new . Let’s look at this formally:

where f is the function which converts embedding E of token T of some prompt X of some novel concept C

Now, let’s suppose that novel concept C has a enormous space of possible interpretations, and contains one interpretation which is novel, this interpretation space is independent of the interpreter.

}

and lets say the universal set of interpretations of concept C which contains the novel interpretations is which in the space of interpretations independent of the interpreter, for example: humans or AI, lies on a manifold .

The model which is very good at connecting different pieces of information together has the same set of concepts C will then form a manifold which: ^[1]

where is a subset of the universal set but has missed out on the important novel interpretation required for the novelty to be acted on. then becomes the set of missing components where is a subset of / .

We will call a novel interpretation, the important question then is whether the model can produce from searching over the manifold or requires constructing a new representational structure . I denote this hypothetical representation-changing operation as :

but this is dependent on the fact that is a discovery like gravity which means that it is present in the space of representations which are already formed but when we take into account the starry night example the did not exist until Van Gogh painted it hence is not a subset of .

So, let be the space of possible products/ interpretations that can be generated from C.

Thus, an agent A has some generative process :

where is a subset of as is restricted to producing representations limited to ^[2]. Hence a creative agent will be able to do the following:

In short, a conventional agent receives a prompt/ problem P, the agent has a representation

the agent then converts search()->

a creative agent does the following in order to find a novelty:

Hence, the model’s creative space can be defined as where if the model does ordinary exploration to produce y. A representation- enhancing operation produces and therefore lies on . Above I informally described creativity as thinking outside the box where the box is

Conclusion

From the above, we can define creativity as when a model creates a useful representation which remained unreachable under the original manifold M. My claim is that we should distinguish between exploration within a learned representational geometry and the ability to deliberately change that geometry itself. Current LLMs are extraordinarily capable at the former: they can combine concepts, traverse complex associations and produce outputs that are novel to the observer. But this does not necessarily imply that they can recognise when their existing representation of a problem is inadequate and construct a fundamentally different one. If this hypothesis is correct, it has implications beyond creativity. A theory of intelligence that characterises increasingly powerful optimisation within a fixed representational structure may remain fundamentally incomplete if intelligence also includes the ability to modify the representations through which optimisation occurs. This may also create an important tension for AI safety: representational stability can make an agent more predictable and governable, while representational plasticity may be necessary for genuinely open-ended intelligence.

The central question I therefore leave open is whether LLMs can discover that the geometry in which they are reasoning is itself inadequate, and autonomously construct a new geometry in which the problem becomes solvable. If they can, understanding how they do so may be central to understanding intelligence. If they cannot, then scaling capability within an existing geometry may impose a deeper limitation on the kind of intelligence we are building.

^
For simplicity basis, I have assumed that the universal set of all interpretations of the concept C wil lie on a manifold . Hence, not a grounded claim that I am making.
^
in my opinion will be restricted to produce representations beyond the manifold which is formed through training and optimisation preassure.

Creativity Beyond the Manifold

Current LLMs and Latent Space

Why Geometry is a constraint to Intelligence?

Concept of creativity and exploration

Conclusion