Bill Benzon

Karma: 186

The Story of My Intellectual Life

In the early 1970s I discovered that “Kubla Khan” had a rich, marvelous, and fantastically symmetrical structure. I’d found myself intellectually. I knew what I was doing. I had a specific intellectual mission: to find the mechanisms behind “Kubla Khan.” As defined, that mission failed, and still has not been achieved some 40 odd years later.

It’s like this: If you set out to hitch rides from New York City to, say, Los Angeles, and don’t make it, well then your hitch-hike adventure is a failure. But if you end up on Mars instead, just what kind of failure is that? Yeah, you’re lost. Really really lost. But you’re lost on Mars! How cool is that!

Of course, it might not actually be Mars. It might just be an abandoned set on a studio back lot.

That’s a bit metaphorical. Let’s just say I’ve read and thought about a lot of things having to do with the brain, mind, and culture, and published about them as well. I’ve written a bunch of academic articles and two general trade books, Visualization: The Second Computer Revolution (Harry Abrams1989), co-authored with Richard Friedhoff, and Beethoven’s Anvil: Music in Mind and Culture (Basic Books 2001). Here’s what I say about myself at my blog, New Savanna. I’ve got a conventional CV at Academia.edu. I’ve also written a lot of stuff that I’ve not published in a conventional venue. I think of them as working papers. I’ve got them all at Academia.edu. Some of my best – certainly my most recent – stuff is there.

Bill Benzon 23 Mar 2023 22:29 UTC
1 point
0
in reply to: ejb’s comment on: A note on ‘semiotic physics’
If you don’t mind, I’ll make a remark.

If it had been up to me, which it most certainly wasn’t and isn’t, it would be called the complex dynamics of transformers, or perhaps LLMs, because that’s what it seems to be. That’s where the math is from and where it’s been most developed. I just ignore the semiotics part of the name. In any event I tend to think the notion of semiotics has long been overgeneralized to the point where it has little meaning. As far as I can tell there’s not much of a connection with any of the intellectual traditions that fly the semiotics flag.

As you say, there’s “a barrier of communication and extra translation cost for many readers.” Well, yeah, if anyone wants to publish or post this work outside of LessWrong, the terminology is likely to prove problematic. If you look around you’ll find that that’s an issue in several posts, communicating with the larger intellectual world. I have no idea how that’s going to work out in the long run.

Bill Benzon 21 Mar 2023 22:01 UTC
1 point
0
in reply to: gwern’s comment on: “Publish or Perish” (a quick note on why you should try to make your work legible to existing academic communities)
I see what you mean. Hanson himself may not be a good example of the argument the author is making. What about the point Hanson himself was making in the quoted passage? Hanson himself does accept the discipline of publishing in the formal literature and he also uses his blog posts as a vehicle for developing more disciplined ideas. Blogging is one thing when it is part of a larger intellectual strategy. But if blogging – & Twitter, Reddit, Facebook and whatever else – is all there is, what happens to rigor then?

Bill Benzon 21 Mar 2023 21:40 UTC
1 point
0
in reply to: Bill Benzon’s comment on: A note on ‘semiotic physics’
BTW, the Annual Review of Condensed Matter Physics has an article on Statistical Mechanics of Deep Learning, by some people from Google Brain and Stanford. I believe the Annual Reviews are now all open access, so you might what to look around. The Annual Review of Linguistics might have some stuff for you.

Bill Benzon 21 Mar 2023 18:26 UTC
1 point
0
on: “Publish or Perish” (a quick note on why you should try to make your work legible to existing academic communities)
Folks commenting here might want to look at this post from April 2020, Blogging is dead, long live sites, which I found in one of Tyler Cowen’s link posts. The author starts of discussing the blogosphere, which he claims peaked in 2009, then moves on to essays vs. blogposts. and ends with independent vs. academic scholarship:
A lot of the sites I list in the accordion above are by part-time autodidacts, or retired scholars. I suppose this is because the incentives in academia are so often towards either small publons or giant monographs, each of which are set in stone once done. (Unless they are grossly flawed enough to trigger academia’s slow, dumb immune system.) (Or my search process is biased towards lone wolves.)
But the average academic work is more lasting than the average internet work. But it isn’t only durability we’re after. But it’s also more rigorous than the average internet work.
Robin Hanson has spotted a trend among independent scholars, a systematic bias against rigour, and so against durability.
“over time amateurs blow their lead by focusing less and relying on easier, more direct methods. They rely more on informal conversation as analysis method, they prefer personal connections over open competitions in choosing people, and they rely more on a perceived consensus among a smaller group of fellow enthusiasts. As a result, their contributions just don’t appeal as widely or as long.”
Take Hanson himself: he has about 100 academic publications, two big books, and something like 3000 blog posts. Which will be his biggest contribution in the end?
Maybe tenured academics are the people best placed to do long content: lots of time, lots of connections, some pressure towards rigour and communicability. But you should be able to do it outside uni, if you’re wary.

Bill Benzon 21 Mar 2023 3:46 UTC
1 point
0
on: Remarks 1–18 on GPT (compressed)
LLMs are the most complicated entities that humanity has made, they are the compression of the sum total of all human history and knowledge, and they’ve existed for less than five years.
This seems to be in the same ballpark as a post I made a couple of years ago, World, mind, and learnability: A note on the metaphysical structure of the cosmos, and included in my working paper on GPT-3, GPT-3: Waterloo or Rubicon? Here be Dragons, Working Paper, pp. 23-26.
Those remarks begin something like this: There is no a priori reason to believe that world has to be learnable. But if it were not, then we wouldn’t exist, nor would (most?) animals. The existing world, thus, is learnable. The human sensorium and motor system are necessarily adapted to that learnable structure, whatever it is.

I am, at least provisionally, calling that learnable structure the metaphysical structure of the world. Moreover, since humans did not arise de novo that metaphysical structure must necessarily extend through the animal kingdom and, who knows, plants as well.

“How”, you might ask, “does this metaphysical structure of the world differ from the world’s physical structure?” I will say, again provisionally, for I am just now making this up, that it is a matter of intension rather than extension. Extensionally the physical and the metaphysical are one and the same. But intensionally, they are different. We think about them in different terms. We ask different things of them. They have different conceptual affordances. The physical world is meaningless; it is simply there. It is in the metaphysical world that we seek meaning.
I then go on to argue that, by virtue of the texts they absorb during training, LLMs come to approximate the metaphysical structure of the world.

Bill Benzon 16 Mar 2023 18:36 UTC
1 point
0
in reply to: Hollis Robbins’s comment on: Want to predict/explain/control the output of GPT-4? Then learn about the world, not about transformers.
Sometimes it moves in the other direction as well, from LLMs to understanding poetry: Vector semantics and the (in-context) construction of meaning in Coleridge’s “Kubla Khan”.

Bill Benzon 16 Mar 2023 18:30 UTC
2 points
2
on: Want to predict/explain/control the output of GPT-4? Then learn about the world, not about transformers.
FWIW, back in the 1970s and 1980s David Marr and Thomas Poggio argued that large complex ‘information processing systems’ need to be analyzed and described on several levels, with the higher levels being implemented in the lower levels. Just what and how many levels there are has varied according to this and that, but the principle is alive and kicking. David Chapman makes a similar argument about levels: How to understand AI systems.
Why should this be the case? Because ANNs in general are designed to take on the structure inherent in the data they absorb during training. If they didn’t, they’d be of little value. If you want to understand how a LLM tells stories, consult a narratologist.
I’ve made the levels argument, with reference to Marr/Poggio, in two recent working papers:
- ChatGPT intimates a tantalizing future; its core LLM is organized on multiple levels; and it has broken the idea of thinking
- ChatGPT tells stories, and a note about reverse engineering
Here’s the abstract of the second paper:
I examine a set of stories that are organized on three levels: 1) the entire story trajectory, 2) segments within the trajectory, and 3) sentences within individual segments. I conjecture that the probability distribution from which ChatGPT draws next tokens follows a hierarchy nested according to those three levels and that is encoded in the weights of ChatGPT’s parameters. I arrived at this conjecture to account for the results of experiments in which ChatGPT is given a prompt containing a story along with instructions to create a new story based on that story but changing a key character: the protagonist or the antagonist. That one change then ripples through the rest of the story. The pattern of differences between the old and the new story indicates how ChatGPT maintains story coherence. The nature and extent of the differences between the original story and the new one depends roughly on the degree of difference between the original key character and the one substituted for it. I conclude with a methodological coda: ChatGPT’s behavior must be described and analyzed on three levels: 1) The experiments exhibit surface level behavior. 2) The conjecture is about a middle level that contains the nested hierarchy of probability distributions. 3) The transformer virtual machine is the bottom level.
Add in some complex dynamics, perhaps some conceptual space semantics from Peter Gärdenfors, and who knows what else, and I expect the mystery about how LLMs work to dissipate before AGI arrives.

Bill Benzon 13 Mar 2023 10:09 UTC
1 point
0
in reply to: Lone Pine’s comment on: The issue of meaning in large language models (LLMs)
Let me quote a passage from ChatGPT intimates a tantalizing future:
Here is my problem: I actually believe those old arguments about why machines can’t (possibly) think, the arguments from intention. That belief clashes with my appraisal of the behavior I’ve seen from ChatGPT in the last two months. Damn it! It looks like a duck.
How do I reconcile the two sides of that conflict? I don’t. Rather, I decide to hold the concept of thought, and a whole mess of allied concepts, in abeyance. I’m going to toss them between phenomenological brackets.
The people who insist that what these large language models are doing might as well be the work of thousands of drunken monkeys pounding on typewriters, they have no way of accounting for the coherent structure in ChatGPT’s output. Oh, they can pounce on its many mistakes – for it makes many – and say, “See, I told you, drunken monkeys!” But the AI boosters who insist that, yes, these guys can think, we’re on the way to AGI, they can’t tell us what it’s going on either. All they can say is that the models are “opaque” – almost a term of art by now – so we don’t know what’s going on, but we’re working on it. And indeed they are.
In this context, “think” is just a label that tells us nothing about what humans are doing that machines are not. That denial does not point the way to knowledge about how to improve these systems – for they surely need improving. I conclude, then, for certain purposes, such as discriminating between human behavior and that of advanced artificial intelligence, the idea of thought has little intellectual value.
Let me be clear. I am not denying that people think; of course we do. Nor am I asserting that advanced AI’s think. They (most likely) do not. But “to think” is an informal common-sense idea. It has no technical definition.[1] We are rapidly approaching an intellectual regime where the question of whether or not machines can think – reason, perceive, learn, feel, etc. – becomes a tractable technical issue. In this regime, common sense ideas about minds and mentation are at best of limited value. At worst, they are useless.
I take that as a sign that we are dealing with something new, really new. We have sailed into those waters where “Here be dragons” is written on the charts. It is time that we acknowledge that we don’t know what we’re doing, that the old ideas aren’t working very well, and get on with the business of creating new ones. Let us learn to fly with and talk with the dragons. It is time to be wild.
Perhaps we have to retire of the concept of meaning as well. As far as I know it doesn’t have a technical definition so perhaps we shouldn’t use it in technical conversations. What does “meaning” tell you about how LLMs work? Nothing. So what conceptual work is it doing? It seems to me it functions mostly as a way of keeping score in some ill-defined race to Mount AGI. But it doesn’t help you run the race.
[1] Some years ago I constructed a definition of the informal concept, “to think,” within a cognitive network. See, William Benzon, Cognitive Networks and Literary Semantics, MLN 91: 1976, pp. 961-964. For a similar approach to the common-sense notion, see William Benzon, First Person: Neuro-Cognitive Notes on the Self in Life and in Fiction, PsyArt: A Hyperlink Journal for Psychological Study of the Arts, August 21, 2000, pp. 23-25, https://www.academia.edu/8331456/First_Person_Neuro-Cognitive_Notes_on_the_Self_in_Life_and_in_Fiction.

Bill Benzon 12 Mar 2023 16:40 UTC
1 point
0
in reply to: J Thomas Moros’s comment on: What do you think is wrong with rationalist culture?
It seems to me there is a distinction to be made: It is one thing to conclude that, 1) Eliezer doesn’t know how to predict the date of AI Doom. That’s different from asserting that 2) AI Doom is not going to happen. 1 is not evidence for 2.

Bill Benzon 12 Mar 2023 13:08 UTC
3 points
0
in reply to: Bogdan Ionut Cirstea’s comment on: The issue of meaning in large language models (LLMs)
I’ve looked at all those articles and read the abstracts, but not the articles themselves. It’s not obvious to me that they provide strong evidence against my view, which is more subtle than what you may have inferred from my post.
I fully accept that words can give meaning to, can even by defined by, other words. That’s the point of the distinction between relationality and adhesion as aspects of semanticity. The importance of the relational aspect of semanticity is central to my thinking, and is something I argue in some detail in GPT-3: Waterloo or Rubicon? Here be Dragons. At the same time I also believe that there is a significant set of words whose semanticity derives exclusively, even predominantly (I’ve not thought it through recently) through their adhesion to the physical world.
Without access to those adhesions the whole linguistic edifice is cut off from the world. Its relationality is fully intact and functioning. That’s what LLMs are running on. That they do so well on that basis is remarkable. But it’s not everything.
If you want to pursue this farther, I’ll make you a deal. I’ll read three of those articles if you read two of mine.
I’m particularly interested in the last two, convergence of language and vision, and language processing in humans and LLMs. You pick the third. For my two, read the Dragons piece I’ve linked to and an (old) article by David Hays, On “Alienation”: An Essay in the Psycholinguistics of Science.

Bill Benzon 12 Mar 2023 11:48 UTC
3 points
−2
in reply to: Lone Pine’s comment on: The issue of meaning in large language models (LLMs)
I said nothing about humans being special. This is an argument about LLMs, not about all possible artificial intelligences. Multi-modal devices have access to the physical world in a way that LLMS do not. That changes things.

Bill Benzon 12 Mar 2023 11:44 UTC
1 point
0
in reply to: Capybasilisk’s comment on: The issue of meaning in large language models (LLMs)
Human minds have life before language, they even have life before birth. But that’s a side issue. The issue with LLMs is that they only have access to word forms. And language forms, by themselves, have no connection to the world. What LLMs can do is figure out the relationships between words as given in usage.

Bill Benzon 10 Mar 2023 21:02 UTC
1 point
0
in reply to: the gears to ascension’s comment on: What do you think is wrong with rationalist culture?
That’s a tough one, in part because the fields vary so much. I was in an English department, so that’s what my degree is in. But my real training came as part of a research group in computational linguistics that was in the linguistics department. I didn’t actually do any programming. I worked on knowledge representation, a big part of old-school computational linguistics (before NLP).

But there are two aspects to this. One is getting the level of intellectual maturity and sophistication you need to function as a disciplined independent thinker. The other is knowing the literature. In some ways they interact and support one another but in some ways they are orthogonal.

I learned the most when I found a mentor, the late David G.l Hays. I wanted to learn his approach to semantics. He tutored me for an hour or two once a week for a semester. That’s the best. It’s also relatively rare to get that kind of individual attention. Still, finding a mentor is the best possible thing you could do.

At the same time I had a job preparing abstracts for The American Journal of Computational Linguistics. That meant I had to read a wide variety of material and prepare abstracts four times a year. You need to learn to extract the gist of an article without reading the whole thing. Look at the introduction and conclusion. Does that tell you what you need? Scan the rest. You should be able to do that – scan the article and write the abstract – in no more than an hour or two.

Note, that many/most abstracts that come with an article are not very good. The idea is, if you trust the journal and the author and don’t need the details, the abstract should tell you all you need. Working up that skill is good discipline.

If you’re working on a project with others here, each of you agree to produce 3, 4, 5 abstracts a week to contribute to the project. Post them to a place where you can all get at them. It becomes your project library.

As for the level of intellectual maturity, the only way to acquire that is to pick a problem, work on it, and come up with a coherent written account of what you’ve done. The account should be intelligible to others. I don’t know whether a formal dissertation is required, but you need to tackle a problem that is both interesting to you and has “weight.”

Bill Benzon 10 Mar 2023 18:04 UTC
4 points
0
in reply to: metasemi’s comment on: A note on ‘semiotic physics’
You might also look at this:
Andrew M. Saxe, James L. McClelland, and Surya Ganguli, A mathematical theory of semantic development in deep neural networks, PNAS, vol. 116, no. 23, June 4, 2019, 11537-11546, https://www.pnas.org/content/116/23/11537
Abstract: An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: What are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences? We address this question by mathematically analyzing the nonlinear dynamics of learning in deep linear networks. We find exact solutions to this learning dynamics that yield a conceptual explanation for the prevalence of many disparate phenomena in semantic cognition, including the hierarchical differentiation of concepts through rapid developmental transitions, the ubiquity of semantic illusions between such transitions, the emergence of item typicality and category coherence as factors controlling the speed of semantic processing, changing patterns of inductive projection over development, and the conservation of semantic similarity in neural representations across species. Thus, surprisingly, our simple neural model qualitatively recapitulates many diverse regularities underlying semantic development, while providing analytic insight into how the statistical structure of an environment can interact with nonlinear deep-learning dynamics to give rise to these regularities.

Bill Benzon 10 Mar 2023 17:57 UTC
3 points
1
in reply to: the gears to ascension’s comment on: What do you think is wrong with rationalist culture?
a lot of people around these parts undervalue academia’s output and independent scholarship and reinvent a lot of stuff.
That’s certainly my impression. I’ve been peeking in here off and on for several years, but became more active last June when I started (cross-)posting here and commenting a bit.
I have a PhD that’s traditional in the sense that I learned to search, read, value, and cite the existing literature on a topic I’m working on. That seems to be missing here, leading, yes, to unnecessary reinvention. I recall reading a post several months ago that made the point that some of the posts here are of sufficiently high quality that they should be placed at, for example, arXiv. Seems reasonable to me. But if you want to do that, you need to pay more attention to what other people are thinking and writing about.

Bill Benzon 10 Mar 2023 15:43 UTC
1 point
0
in reply to: metasemi’s comment on: A note on ‘semiotic physics’
You’re welcome.

Bill Benzon 9 Mar 2023 23:54 UTC
1 point
0
on: Against LLM Reductionism
You might want to read a working paper I just put on line: ChatGPT tells stories, and a note about reverse engineering: A Working Paper.
I examine a set of stories that are organized on three levels: 1) the entire story trajectory, 2) segments within the trajectory, and 3) sentences within individual segments. I conjecture that the probability distribution from which ChatGPT draws next tokens follows a hierarchy nested according to those three levels and that is encoded in the weights off ChatGPT’s parameters. I arrived at this conjecture to account for the results of experiments in which ChatGPT is given a prompt containing a story along with instructions to create a new story based on that story but changing a key character: the protagonist or the antagonist. That one change then ripples through the rest of the story. The pattern of differences between the old and the new story indicates how ChatGPT maintains story coherence. The nature and extent of the differences between the original story and the new one depends roughly on the degree of difference between the key character and the one substituted for it. I conclude with a methodological coda: ChatGPT’s behavior must be described and analyzed on three levels: 1) The experiments exhibit surface level behavior. 2) The conjecture is about a middle level that contains the nested hierarchy of probability distributions. 3) The transformer virtual machine is the bottom level.

Bill Benzon 9 Mar 2023 18:57 UTC
1 point
0
in reply to: MSRayne’s comment on: ChatGPT explores the semantic differential
You’re right, those responses seem off. Mainly I’m happy that it could perform the task at all.

Bill Benzon 9 Mar 2023 15:00 UTC
4 points
0
on: A note on ‘semiotic physics’
Have you thought of exploring the existing literature on the complex dynamics of nervous systems. It’s huge, but it does use the math you guys are borrowing from physics.
I’m thinking in particular of the work of the late Walter Freeman, who is a pioneer in the field. Toward the end of his career he began developing a concept of “cinematic consciousness.” As you know the movement in motion pictures is an illusion created by the fact the individual frames of the image are projected on the screen more rapidly than the mind can resolve them. So, while the frames are in fact still, they change so rapidly that we see motion.
First I’ll give you some quotes from Freeman’s article to give you a feel for his thinking (alas, you’ll have to read the article to see how those things connect up), then I’ll explain what that has to do with LLMs. The numbers are from Freeman’s article.
[20] EEG evidence shows that the process in the various parts occurs in discontinuous steps (Figure 2), like frames in a motion picture (Freeman, 1975; Barrie, Freeman and Lenhart, 1996).
[23] Everything that a human or an animal knows comes from the circular causality of action, preafference, perception, and up-date. It is done by successive frames of self-organized activity patterns in the sensory and limbic cortices. [...]
[35] EEG measurements show that multiple patterns self-organize independently in overlapping time frames in the several sensory and limbic cortices, coexisting with stimulus-driven activity in different areas of the neocortex, which structurally is an undivided sheet of neuropil in each hemisphere receiving the projections of sensory pathways in separated areas. [...]
[86] Science provides knowledge of relations among objects in the world, whereas technology provides tools for intervention into the relations by humans with intent to control the objects. The acausal science of understanding the self distinctively differs from the causal technology of self-control. “Circular causality” in self-organizing systems is a concept that is useful to describe interactions between microscopic neurons in assemblies and the macroscopic emergent state variable that organizes them. In this review intentional action is ascribed to the activities of the subsystems. Awareness (fleeting frames) and consciousness (continual operator) are ascribed to a hemisphere-wide order parameter constituting a global brain state. Linear causal inference is appropriate and essential for planning and interpreting human actions and personal relations, but it can be misleading when it is applied to microscopic- microscopic relations in brains.
Notice that Freeman refers to “a hemisphere-wide order parameter constituting a global brain state.” The cerebral cortex consists of 16B neurons, each with roughly 10K connections. Further, all areas of the cortex have connections with subcortical regions. That’s an awful-lot of neurons communicating in parallel in a single time step. As I recall from another article, these frames occur at a rate of 6-7 Hz.
The nervous system operates in parallel. I believe it is known that the brain exhibits a small world topology, so all neurons are within a relatively small number links from one another. Though at any moment some neurons will be more active than others, they are all active – the only inactive neuron is a dead neuron. Similarly, ANNs exhibit a high degree of parallelism. LLMs are parallel virtual machines being simulated by so-called von Neumann machines. The use of multiple cores gives a small degree of parallelism, but that’s quite small in relation to the overall number of parameters the system has.
I propose that the process of generating a single token in an LLM is comparable to a single “frame” of consciousness in Freeman’s model. All the parameters in the system are visited during a single time-step for the system. In the case of ChatGPT I believe that’s 175B parameters.
Thus the assertion that ChatGPT generates one token at a time, based on the previous string, while true, is terribly reductive and thus misleading. The appearance of a token is in fact more or less a side-effect of evolving a trajectory from the initial prompt.

Bill Benzon 7 Mar 2023 0:48 UTC
1 point
0
in reply to: metasemi’s comment on: A note on ‘semiotic physics’
Come to think of it, I have a working paper that speculates on this sort of thing: Virtual Reading: The Prospero Project Redux.
You might also look at (notice the date): E. Alvarez-Lacalle, B. Dorow, J.-P. Eckmann, and E. Moses. Hierarchical structures induce long-range dynamical correlations in written texts. PNAS Vol. 103 no. 21. May 23, 2006: 7956–7961. doi: 10.1073/pnas.0510673103 Here’s the abstract:
Thoughts and ideas are multidimensional and often concurrent, yet they can be expressed surprisingly well sequentially by the translation into language. This reduction of dimensions occurs naturally but requires memory and necessitates the existence of correlations, e.g., in written text. However, correlations in word appearance decay quickly, while previous observations of long-range correlations using random walk approaches yield little insight on memory or on semantic context. Instead, we study combinations of words that a reader is exposed to within a “window of attention,” spanning about 100 words. We define a vector space of such word combinations by looking at words that co-occur within the window of attention, and analyze its structure. Singular value decomposition of the co-occurrence matrix identifies a basis whose vectors correspond to specific topics, or “concepts” that are relevant to the text. As the reader follows a text, the “vector of attention” traces out a trajectory of directions in this “concept space.” We find that memory of the direction is retained over long times, forming power-law correlations. The appearance of power laws hints at the existence of an underlying hierarchical network. Indeed, imposing a hierarchy similar to that defined by volumes, chapters, paragraphs, etc. succeeds in creating correlations in a surrogate random text that are identical to those of the original text. We conclude that hierarchical structures in text serve to create long-range correlations, and use the reader’s memory in reenacting some of the multidimensionality of the thoughts being expressed.