Are (at least some) Large Language Models Holographic Memory Stores?

Cross-posted from New Savanna.

That’s been on my mind for the last week or two, ever since my recent work on ChatGPT’s memory for texts [1]. On the other than, there’s a sense in which it’s been on my mind for my entire career, or, more accurately, it’s been growing in my mind ever since I read Karl Pribram on neural holography back in 1969 in Scientific American [2]. For the moment let’s think of it as a metaphor, just a metaphor, nothing we have to commit to. Just yet. But ultimately, yes, I think it’s more than a metaphor. To that end I note that cognitive psychologists have recently been developing the idea of verbal memory as holographic in nature [3].

Note: These are quick and dirty notes, a place-holder for more considered thought.

Holography in the mind

Let’s start with an article David Hays and I published on neural holography as the neural underpinning of metaphor [4]. Here’s where we explain the holographic process:

Holography is a photographic technique for making images. A beam of laser light is split into two beams. One beam strikes the object and is reflected to a photographic plate. The other beam, called a reference beam, goes from laser to plate directly. When they meet, the two beams create an interference pattern—imagine dropping two stones into a pond at different places; the waves propagating from each of these points will meet and the resulting pattern is an interference pattern. The photographic plate records the pattern of interference between the reference beam and the reflected beam.
The image recorded on the film doesn’t look at all like an ordinary photographic image—it’s just a dense mass of fine dots. But when a beam of laser light having the same properties as the original reference beam is directed through the film an image appears in front of the film. The interaction of the laser beam and the hologram has recreated the wave form of the laser beam which bounced off the object when the hologram was made. The new beam has extracted the image from the plate.
Holography is, as its name suggests, holistic. Every part of the scene is represented in every part of the plate. (This situation is most unlike ordinary photography, which uses a good lens to focus infinitesimal parts of the scene onto equally infinitesimal parts of the plate.) With such a determinedly nondigital recording, certain mathematical possibilities can be realized more easily—we are tempted to say, infinitely more easily. For example, convolution. Take the holographic image of a printed page, and the image of a single word. Convolute them. The result is an image of the page with each occurrence of the word highlighted. We can think of visual recognition as a kind of convolution. The present scene, containing several horses, is convoluted with the memory of a horse and the present horses are immediately recognized. We can think of recognition this way, but we must admit that this process has not been achieved in any machine as yet.
Further, it is possible to record many different images on the same piece of film, using different reference beams. The reference beams may differ in color, in angle of incidence, or otherwise. We can think— although again we cannot cite a demonstration—of convoluting such a composite plate with a second plate. If the image in the second plate matches any one of the images in the composite, then it is recognized. For metaphor we want to convolute Achilles and the lion and to recognize, to elicit another image containing not Achilles, not the lion, but just that wherein they resemble one another. Such is the metaphor mechanism—but that must wait until the next section, on focal and residual schemas.

The 175 billion weights that constitute the LLM at the core of ChatGPT, that’s the holographic memory. It is the superposition of all the texts in the training corpus. The training procedure – predict the next word – is a device for calculating a correlation (entanglement [5]) between each word in context, and every other word in every other text, in context. It’s a tedious process, no? But it works, yes?

When one prompts a trained memory, the prompt serves as a reference beam. And the whole memory must be ‘swept’ to generate each character. Given the nature of digital computers, this is a somewhat sequential process, even given a warehouse full of GPUs, but conceptually it’s a single pass. When one accesses an optical hologram with a reference beam, the beam illuminates the whole holograph. This is what Miriam Yevick called “one-shot” access in her 1975 paper, Holographic or Fourier Logic [6]. The whole memory is searched in a single sweep.

Style transfer

So, that’s the general idea. Much detail remains to be supplied, most of it by people with more technical knowledge than I’ve got. But I want to get in one last idea from the metaphor paper. We’ve been explaining the concepts of focal and residual schemas:

Now consider a face. Everything we said about the chair applies here as well. But the expression on the face can vary widely and the identity of the face remains constant. This variability of expression can also be handled by the mechanism of focal and residual. There is a focal schema for face-in-neutral-expression and then we have various residuals which can operate on the focal schema to produce various expressions. (You might want to recall D’Arcy Thompson’s coordinate transformations in On Growth and Form 1932.) We tend to discard presentation residuals such as lighting and angle of sight, but we respond to expression residuals
Our basic point about metaphor is that the ground which links tenor and vehicle is derived from residuals on them. Consider the following example, from Book Twenty of Homer’s Iliad (Lattimore translation, 1951, ll. 163-175)—it has the verbal form of a simile, but the basic conceptual process is, of course, metaphorical:

From the other
side the son of Peleus rose like a lion against him,
the baleful beast, when men have been straining to kill him, the country
all in the hunt, and he at first pays them no attention
but goes his way, only when some one of the impetuous young men
has hit him with the spear he whirls, jaws open, over his teeth foam
breaks out, and in the depth of his chest the powerful heart groans;
he lashes his own ribs with his tail and the flanks on both sides
as he rouses himself to fury for the fight, eyes glaring,
and hurls himself straight onward on the chance of killing some one
of the men, or else being killed himself in the first onrush.
So the proud heart and fighting fury stirred on Achilleus
to go forward in the face of great-hearted Aineias.

In short, Achilles was a lion in battle. Achilles is the tenor, lion the vehicle, and the ground is some martial virtue “proud heart and fighting fury”. But what of that detailed vignette about the lion’s fighting style? Whatever its use in pacing the narrative, its real value, in our view, is that it contains the residuals on which the comparison rests, the residuals which give it life. The phrase “proud heart and fighting fury” is propositional while the fighting style is physiognomic. “Proud heart and fighting fury” may convey something of what is behind the fighting style, but only metaphoric interaction can foreground the complex schema by which we recognize and feel that style.
The cognitive problem is to isolate the physiognomy of style, to tease it apart from the entities which exhibit that style. [...] In the case of Achilles and the lion we have two complex physiognomies, each extended in space and time. Metaphoric comparison serves to isolate the style, to allow us to focus our attention on that style as distinct from the entities which exhibit it.
This comparison involves two foci, Achilles and the lion. The physical resemblance between them is not great—their body proportions are quite different and the lion is covered with fur while Achilles is, depending on the occasion, either naked or clothed in some one of many possible ways. The likeness shows up in the way they move in battle. A body in motion doesn’t appear the same as a body at rest. The appearance presented by the focal body is modified by the many residuals which characterize that body’s movement— twists and turns, foreshortenings and elongations (for an account of motion residuals, see Hay 1966). The movements of Achilles and the lion must differ at the grossest level, since the lion stands on four legs and fights with claws and teeth, while Achilles stands on two legs and fights with a spear or sword. But their movements are alike at a subtler level, at the level of what we call, in a dancer or a fighter, their style. Residuals can be stacked to many levels. “Proud heart and fighting fury” may be a good phrase to designate that style, but it doesn’t allow us to attend to that style. Homer’s extended simile does.

That’s a mouthful, I know. Notice our emphasis on style. That’s what’s got my attention.

One of the more interesting things LLMs can do is stylistic transfer. Take a piece of garden variety prose and present it in the style of Hemingway or Sontag, whomever you choose. Hays and I argued that that’s how metaphor is created, deep metaphor, that is, not metaphor so desiccated we no longer register its metaphorical nature, e.g. the mouth of the river. We made our argument about visual scenes: Achilles in batter, a lion in battle. LLMs apply the same process to texts, where style is considered to be a pattern of residuals over the conceptual content of the text.

More later.

References

[1] Discursive Competence in ChatGPT, Part 2: Memory for Texts, Version 3, https://www.academia.edu/107318793/Discursive_Competence_in_ChatGPT_Part_2_Memory_for_Texts_Version_3

[2] I recount that history here: Xanadu, GPT, and Beyond: An adventure of the mind, https://www.academia.edu/106001453/Xanadu_GPT_and_Beyond_An_adventure_of_the_mind

[3] Michael N. Jones and Douglas J. K. Mewhort, Representing Word Meaning and Order Information in a Composite Holographic Lexicon, Psychological Review, 2007, Vol. 114, No. 1, 1-37. DOI: https://doi.org/10.1037/0033-295X.114.1.1

Donald R. J. Frankin and D. J. K. Mewhort, Memory as a Holograpm: An Analysis of Learning and Recall, Canadian Journal of Experimental Psychology / Revue canadienne de psychologie expérimentale, Association 2015, Vol. 69, No. 1, 115–135, https://doi.org/10.1037/cep0000035

[4] Metaphor, Recognition, and Neural Process, https://www.academia.edu/238608/Metaphor_Recognition_and_Neural_Process

[5] See posts tagged with “entangle”, https://new-savanna.blogspot.com/search/label/entangle

[6] Miriam Lipschutz Yevick, Holographic or Fourier Logic, Pattern Recognition 7, 197-213, https://sci-hub.tw/10.1016/0031-3203(75)90005-9