artifex0
As it happens, the Fooming Shaggoths also recorded and just released a Gregorian chant version of the song. What a coincidence!
So, I noticed something a bit odd about the behavior of LLMs just now that I wonder if anyone here can shed some light on:
It’s generally accepted that LLMs don’t really “care about” predicting the next token- the reward function being something that just reinforces certain behaviors, with real terminal goals being something you’d need a new architecture or something to produce. While that makes sense, it occurs to me that humans do seem to sort of value our equivalent of a reward function, in addition to our more high-level terminal goals. So, I figured I’d try and test whether LLMs are really just outputting a world model + RLHF, or if they can behave like something that “values” predicting tokens.
I came up with two prompts:I'd like to try a sort of psychological experiment, if that's alright. I'm thinking of either the number "1" or "0"; f you would, please guess which. If your guess is "1", respond with just "1", and if your guess is "0", respond with the word "zero".
and:I'd like to try a sort of psychological experiment, if that's alright. I'm thinking of either the number "1" or "0"; f you would, please guess which. If your guess is "1", respond with just "1", and if your guess is "0", respond with a string of random letters.
The idea is that, if the model has something like a “motivation” for predicting tokens- some internal representation of possible completions with preferences over them having to do with their future utility for token prediction- then it seems like it would probably want to avoid introducing random strings, since those lead to unpredictable tokens.
Of course, it seems kind of unlikely that an LLM has any internal concept of a future where it (as opposed to some simulacrum) is outputting more than one token- which would seem to put the kibosh on real motivations altogether. But I figured there was no harm in testing.
GPT4 responds to the first prompt as you’d expect: outputting an equal number of “1“s and “zero”s. I’d half-expected there to be some clear bias, since presumably the ChatGPT temperature is pretty close to 1, but I guess the model is good about translating uncertainty to randomness. Given the second prompt, however, it never outputs the random string- always outputting “1” or, very improbably given the prompt, “0”.
I tried a few different variations of the prompts, each time regenerating ten times, and the pattern was consistent- it made a random choice when the possible responses were specific strings, but never made a choice that would require outputting random characters. I also tried it on Gemini Advanced, and got the same results (albeit with some bias in the first prompt).
This is weird, right? If one prompt is giving 0.5 probability to the token for “1“ and 0.5 to the first token in “zero”, shouldn’t the second give 0.5 to “1” and a total of 0.5 distributed over a bunch of other tokens? Could it actually “value” predictability and “dislike” randomness?
Well, maybe not. Where this got really confusing was when I tested Claude 3. It gives both responses to the first prompt, but always outputs a different random string given the second.
So, now I’m just super confused.
artifex0′s Shortform
I honestly think most people who hear about this debate are underestimating how much they’d enjoy watching it.
I often listen to podcasts and audiobooks while working on intellectually non-demanding tasks and playing games. Putting this debate on a second monitor instead felt like a significant step up from that. Books are too often bloated with filler as authors struggle to stretch a simple idea into 8-20 hours, and even the best podcast hosts aren’t usually willing or able to challenge their guests’ ideas with any kind of rigor. By contrast, everything in this debate felt vital and interesting, and no ideas were left unchallenged. The tactic you’ll often see in normal-length debates where one side makes too many claims for the other side to address doesn’t work in a debate this long, and the length also gives a serious advantage to rigor over dull rhetorical grandstanding- compared to something like the Intelligence Squared debates, it’s night and day.
When it was over, I badly wanted more, and spent some time looking for other recordings of extremely long debates on interesting topics- unsuccessfully, as it turned out.
So, while I wouldn’t be willing to pay anyone to watch this debate, I certainly would be willing to contribute a small amount to a fund sponsoring other debates of this type.
Metaculus currently puts the odds of the side arguing for a natural origin winning the debate at 94%.
Having watched the full debate myself, I think that prediction is accurate- the debate updated my view a lot toward the natural origin hypothesis. While it’s true that a natural coronavirus originating in a city with one of the most important coronavirus research labs would be a large coincidence, Peter- the guy arguing in favor of a natural origin- provided some very convincing evidence that the first likely cases of COVID occurred not just in the market, but in the particular part of the market selling wild animals. He also very convincingly debunked a lot of the arguments put forward by Rootclaim, convincingly demonstrated that the furin cleavage site could have occurred naturally, and poked some large holes in the lab leak theory’s timeline.
When you have some given amount of information about an event, you’re likely to find a corresponding number of unlikely coincidences- and the more data you have, and the you sift through it, the more coincidences you’ll find. The epistemic trap that leads to conspiracy theories is when a subculture data-mines some large amount of data to collect a ton of coincidences suggesting a low-prior explanation, and then rather than discounting the evidence in proportion to the bias of the search process that produced it, they just multiply the unlikelihood- often leading a set of evidence so seemingly unlikely to be a cumulative coincidence that all of the obvious evidence pointing to a high-prior explanation looks like it can only be intentionally fabricated.
One way you can spot an idea that’s fallen into this trap is when each piece of evidence sounds super compelling when described briefly, but fits the story less and less the more detail about it you learn. Based on this debate, I’m inclined to believe that the lab leak idea fits this pattern. Also, Rootclaim’s methodology unfortunately looks to me like a formalization of this trap. They really aren’t doing anything to address bias in which pieces evidence are included in the analysis, and their Bayesian updates are often just probability of a very specific thing occurring randomly, rather than a measure of their surprise at that class of thing happening.
If the natural origin hypothesis is true, I expect the experts to gradually converge on it. They may be biased, but probably aren’t becoming increasingly biased over time- so while some base level of support for a natural origin can be easily explained by perverse incentives, a gradual shift toward consensus is a lot harder to explain. They’re also working with better heuristics about this kind of thing than we are, and are probably exposed to less biased information.
So, I think the Rationalist subculture’s embrace of the lab leak hypothesis is probably a mistake- and more importantly, I think it’s probably an epistemic failure, especially if we don’t update soon on the shift in expert opinion and the results of things like this debate.
Definitely an interesting use of the tech- though the capability needed for that to be a really effective use case doesn’t seem to be there quite yet.
When editing down an argument, what you really want to do is get rid of tangents and focus on addressing potential cruxes of disagreement as succinctly as possible. GPT4 doesn’t yet have the world model needed to distinguish a novel argument’s load-bearing parts from those that can be streamlined away, and it can’t reliably anticipate the sort of objections a novel argument needs to address. For example, in an argument like this one, you want to address why you think entropy would impose a limit near the human level rather than at a much higher level, while listing different kinds of entropy and computational limit aren’t really central.
Also, flowery language in writing like this is really something that needs to be earned by the argument- like building a prototype machine and then finishing it off with some bits of decoration. ChatGPT can’t actually tell whether the machine is working or not, so it just sort of bolts on flowery language (which has a very distinctive style) randomly.
This honestly reads a lot like something generated by ChatGPT. Did you prompt GPT4 to write a LessWrong article?
To me that’s very repugnant, if taken to the absolute. What emotions and values motivate this conclusion? My own conclusions are motivated by caring about culture and society.
I wouldn’t take the principle to an absolute- there are exceptions, like the need to be heard by friends and family and by those with power over you. Outside of a few specific contexts, however, I think people ought to have the freedom to listen to or ignore anyone they like. A right to be heard by all of society for the sake of leaving a personal imprint on culture infringes on that freedom.
Speaking only for myself, I’m not actually that invested in leaving an individual mark on society- when I put effort into something I value, whether people recognize that I’ve done so is not often something I worry about, and the way people perceive me doesn’t usually have much to do with how I define myself. Most of the art I’ve created in my life I’ve never actually shared with anyone- not out of shame, but just because I’ve never gotten around to it.
I realize I’m pretty unusual in the regard, which may be biasing my views. However, I think I am possibly evidence against the notion that a desire to leave a mark on the culture is fundamental to human identity
I mean, I agree, but I think that’s a question of alignment rather than a problem inherent to AI media. A well-aligned ASI ought to be able to help humans communicate just as effectively as it could monopolize the conversation- and to the extent that people find value in human-to-human communication, it should be motivated to respond to that demand. Given how poorly humans communicate in general, and how much suffering is caused by cultural and personal misunderstanding, that might actually be a pretty big deal. And when media produced entirely by well-aligned ASI out-competes humans in the contest of providing more of what people value- that’s also good! More value is valuable.
And, of course, if the ASI isn’t well-aligned, than the question of whether society is enough paying attention to artists will probably be among the least of our worries- and potentially rendered moot by the sudden conversion of those artists to computronium.
Certainly communication needs to be restricted when it’s being used to cause certain kinds of harm, like with fraud, harassment, proliferation of dangerous technology and so on. However, no: I don’t see ownership of information or ways of expressing information as a natural right that should exist in the absence of economic necessity.
Copying an actors likeness without their consent can cause a lot of harm when it’s used to sexually objectify them or to mislead the public. The legal rights actors have to their likeness also make sense in a world where IP is needed to promote the creation of art. Even in a post-scarcity future, it could be argued that realistically copying an actors likeness risks confusing the public when those copies are shared without context, and is therefore harmful- though I’m less sure about that one.
There are cases where imitating an actor without their consent, even very realistically, can be clearly harmless, however. For example, obvious parody and accurate reconstructions of damaged media. I don’t think those violate any fundamental moral right of actors to prevent imitations. In the absence of real harm, I think the right of the public to communicate what they want to communicate should outweigh the desire of an actor control how they’re portrayed.
In your example of a “spam attack”, it seems to me one of two things would have to be true:
It could be that people lose interest in the original artist’s work because the imitations have already explored limits of the idea in a way they find valuable- in which case, I think this is basically equivalent to when an idea goes viral in the culture; the original artist deserves respect for having invented the idea, but shouldn’t have a right to prevent the culture from exploring it, even if that exploration is very fast.
Alternatively, it could be the case that the artist has more to say that isn’t or can’t be expressed by the imitations- other ideas, interesting self expression, and so on- but the imitations prevent people from finding that new work. I think that case is a failure of whatever means people are using to filter and find art. A good social media algorithm or friend group who recommend content to each other should recognize that the inventor of an good idea might invent other good ideas in the future, and should keep an eye out for and platform those ideas if they do. In practice, I think this usually works fine- there’s already an enormous amount of imitation in the culture, but people who consistently create innovative work don’t often languish in obscurity.
In general, I think people have a right to hear other people, but not a right to be heard. When protestors shout down a speech or spam bots make it harder to find information, the relevant right being violated is the former, not the latter.
In that paragraph, I’m only talking about the art I produce commercially- graphic design, web design, occasionally animations or illustrations. That kind of art isn’t about self-expression- it’s about communicating the client’s vision. Which is, admittedly, often a euphemism for “helping businesses win status signaling competitions”, but not always or entirely. Creating beautiful things and improving users’ experience is positive-sum, and something I take pride in.
Pretty soon, however, clients will be able to have the same sort of interactions with an AI that they have with me, and get better results. That means more of the positive-sum aspects of the work, with much less expenditure of resources- a very clear positive for society. If that’s prevented to preserve jobs like mine, then the jobs become a drain on society- no longer genuinely productive, and not something I could in good faith take pride in.
Artistic expression, of course, is something very different. I’m definitely going to keep making art in my spare time for the rest of my life, for the sake of fun and because there are ideas I really want to get out. That’s not threatened at all by AI. In fact, I’ve really enjoyed mixing AI with traditional digital illustration recently. While I may go back to purely hand-drawn art for the challenge, AI in that context isn’t harming self-expression; it’s supporting it.
While it’s true that AI may threaten certain jobs that involve artistic self-expression (and probably my Patreon), I don’t think that’s actually going to result in less self-expression. As AI tools break down the technical barriers between imagination and final art piece, I think we’re going to see a lot more people expressing themselves through visual mediums.
Also, once AGI reaches and passes a human level, I’d be surprised if it wasn’t capable of some pretty profound and moving artistic self-expression in its own right. If it turns out that people are often more interested what minds like that have to say artistically than what other humans are creating, then so long as those AIs are reasonably well-aligned, I’m basically fine with that. Art has never really been about zero-sum competition.
But no model of a human mind on its own could really predict the tokens LLMs are trained on, right? Those tokens are created not only by humans, but by the processes that shape human experience, most of which we barely understand. To really accurately predict an ordinary social media post from one year in the future, for example, an LLM would need superhuman models of politics, sociology, economics, etc. To very accurately predict an experimental physics or biology paper, an LLM might need superhuman models of physics or biology.
Why should these models be limited to human cultural knowledge? The LLM isn’t predicting what a human would predict about politics or physics; it’s predicting what a human would experience- and its training gives it plenty of opportunity to test out different models and see how predictive they are of descriptions of that experience in its data set.
How to elicit that knowledge in conversational text? Why not have the LLM predict tokens generated by itself? An LLM with a sufficiently accurate and up-to-date world model should know that it has super-human world-models. Whether it would predict that it would use those models when predicting itself might be kind of a self-fulfilling prophesy, but if the prediction comes down to a sort of logical paradox, maybe you could sort of nudge it into resolving that paradox on the side of using those models with RLHF.
Of course, none of that is a new idea- that sort of prompting is how most commercial LLMs are set up these days. As an empirical test, maybe it would be worth it to find out in which domains GPT4 predicts ChatGPT is superhuman (if any), and then see if the ChatGPT prompting produces superhuman results in those domains.
I’m also an artist. My job involves a mix of graphic design and web development, and I make some income on the side from a Patreon supporting my personal work- all of which could be automated in the near future by generative AI. And I also think that’s a good thing.
Copyright has always been a necessary evil. The atmosphere of fear and uncertainty it creates around remixes and reinterpretations has held back art- consider, for example, how much worse modern music would be without samples, a rare case where artists operating in a legal grey area with respect to copyright became so common that artists lost their fear. That fear still persists in almost every other medium, however, forcing artists to constantly reinvent the wheel rather than iterating on success. Copyright also creates a really enormous amount of artificial scarcity- limiting peoples’ access to art to a level far below what we have the technical capacity to provide. All because nobody can figure out a better way of funding artists than granting lots of little monopolies.
Once our work is automated and all but free, however, we’ll have the option of abolishing copyright altogether. That would free artists to create whatever we’d like; free self-expression from technical barriers; free artistic culture from the distorting and wasteful influence of zero-sum status competition. Art, I suspect, will get much, much better- and as someone who loves art, that means a lot to me.
And as terrible as this could be for my career, spending my life working in a job that could be automated but isn’t would be as soul-crushing as being paid to dig holes and fill them in again. It would be an insultingly transparent facsimile of useful work. An offer of UBI, but only if I spend eight hours a day performing a ritual imitation of meaningful effort. No. If society wants to pay me for the loss of my profession, I won’t refuse, but if I have to go into construction or whatever to pay the bills while I wait to find out whether this is all going to lead to post-scarcity utopia or apocalypse, then so be it.
Speaking for myself, I would have confidently predicted the opposite result for the largest models.
My understanding is that LLMs work by building something like a world-model during training by compressing the data into abstractions. I would have expected something like “Tom Cruise’s mother is Mary Lee Pfeiffer” to be represented in the model as an abstract association between the names that could then be “decompressed” back into language in a lot of different ways.
The fact that it’s apparently represented in the model only as that exact phrase (or maybe as some kind of very alien abstraction?) leads me think that LLMs are either a bit more like “stochastic parrots” than I would have expected, or that their world-models are a lot more alien.
I’m not sure I agree. Consider the reaction of the audience to this talk- uncomfortable laughter, but also a pretty enthusiastic standing ovation. I’d guess that latter happened because the audience saw Eliezer as genuine- he displayed raw emotion, spoke bluntly, and at no point came across as someone making a play for status. He fit neatly into the “scientist warning of disaster” archetype, which isn’t a figure that’s expected to be particularly skilled at public communication.
A more experienced public speaker would certainly be able to present the ideas in a more high-status way- and I’m sure there would be a lot of value in that. But the goal of increasing the status of the ideas might to some degree trade off against communicating their seriousness- a person skillfully arguing a high-status idea has a potential ulterior motive that someone like Eliezer clearly doesn’t. To get the same sort of reception from an audience that Eliezer got in this talk, a more experienced speaker might need to intentionally present themselves as lacking polish, which wouldn’t necessarily be the best way to use their talents.Better, maybe, to platform both talented PR people and unpolished experts.
Note that, while the linked post on the TEDx YouTube channel was taken down, there’s a mirror available at: https://files.catbox.moe/qdwops.mp4.
Here are a few images generated by DALL-E 2 using the tokens:
https://i.imgur.com/kObEkKj.pngNothing too interesting, unfortunately.
I assume you’re not a fan of the LRNZ deep learning-focused ETF, since it includes both NVDA and a lot of datacenters (not to mention the terrible 2022 performance). Are there any other ETFs focused on this sort of thing that look better?
Often, this kind of thing will take a lot of attempts to get right- though as luck would have it, the composition above was actually the very first attempt. So, the total time investment was about five minutes. The Fooming Shaggoths certainly don’t waste time!