I like the analogy of a LARP. Characters in a book don’t have reputation or human-like brain states that they honestly try to represent—but a good book can contain interesting, believable characters with consistent motivation, etc. I once participated in a well-organized fantasy LARP in graduate school. I was bad at it but it was a pretty interesting experience. In particular people who are good are able to act in character and express thoughts that “the character would be having” which are not identical to the logic and outlook of the player (I was bad at this, but other players could do it I think). In my case, I noticed that the character imports a bit of your values, which you sometimes break in-game if it feels appropriate. You also use your cognition to further the character’s cognition, while rationalizing their thinking in-game. It obviously feels different from real life: it’s explicitly a setting where you are allowed and encouraged to break your principles (like you are allowed to lie in a game of werewolf, etc.) and you understand that this is low-stakes, and so don’t engage the full mechanism of “trying as hard as possible” (to be a good person, to achieve good worlds, etc.). But also, there’s a sense in which a LARP seems “Turing-complete” for lack of a better word. For example in this LARP, the magical characters (not mine) collaboratively solved a logic puzzle to reverse engineer a partially known magic system and became able to cast powerful spells. I could also imagine modeling arbitrarily complex interactions and relationships in an extended LARP. There would probably always be some processing cost to add the extra modeling steps, but I can’t see how this would impose any hard constraints on some measure of “what is achievable” in such a setting.
I don’t see hard reasons for why e.g. a village of advanced LLMs could not have equal or greater capability than a group of smart humans playing a LARP. I’m not saying I see evidence they do—I just don’t know of convincing systematic obstructions. I agree that modern LLMs seem to not be able to do some things humans could do even in a LARP (some kind of theory of mind, explaining a consistent thinking trace that makes sense to a person upon reflection, etc.) but again a priori this might just be a skill issue.
So I wonder in the factorization “LLM can potentially get as good as humans in a LARP” + “sufficiently many smart humans in a long enough LARP are ‘Turing complete up to constant factors’ ” (in the sense of in principle being able to achieve, without breaking character, any intellectual outcome that non-LARP humans could do), which part would you disagree with?
I am both experienced enough in text-based RP and have interacted with Character.AI enough to confidently assert that LLMs are not categorically different in their output from a poor-memory RPer, despite sometimes clearly different underlying patterns.
I see another obstruction in attentionspan. I strongly suspect that whenever an LLM is tasked with writing the next token, attention mechanisms compress all potentially relevant information into less than a hundred thousand numbers, preventing the model from taking many nuances into account when writing the token. A human brain, on the other hand, takes into account billions of bits of information stored in neuron activations.
@TsviBT, we had a warning shot of LLMs becoming useful in research or writing a coherent short story in @Tomás B.’s experiment (the post on LW describing it was removed for an unknown reason).
I moved it into my drafts. I published it again for you. I figured it was unlikely to be referenced again and I tend to take stuff down I don’t want people reading as one of the first things on my author’s page.
I like the analogy of a LARP. Characters in a book don’t have reputation or human-like brain states that they honestly try to represent—but a good book can contain interesting, believable characters with consistent motivation, etc. I once participated in a well-organized fantasy LARP in graduate school. I was bad at it but it was a pretty interesting experience. In particular people who are good are able to act in character and express thoughts that “the character would be having” which are not identical to the logic and outlook of the player (I was bad at this, but other players could do it I think). In my case, I noticed that the character imports a bit of your values, which you sometimes break in-game if it feels appropriate. You also use your cognition to further the character’s cognition, while rationalizing their thinking in-game. It obviously feels different from real life: it’s explicitly a setting where you are allowed and encouraged to break your principles (like you are allowed to lie in a game of werewolf, etc.) and you understand that this is low-stakes, and so don’t engage the full mechanism of “trying as hard as possible” (to be a good person, to achieve good worlds, etc.). But also, there’s a sense in which a LARP seems “Turing-complete” for lack of a better word. For example in this LARP, the magical characters (not mine) collaboratively solved a logic puzzle to reverse engineer a partially known magic system and became able to cast powerful spells. I could also imagine modeling arbitrarily complex interactions and relationships in an extended LARP. There would probably always be some processing cost to add the extra modeling steps, but I can’t see how this would impose any hard constraints on some measure of “what is achievable” in such a setting.
I don’t see hard reasons for why e.g. a village of advanced LLMs could not have equal or greater capability than a group of smart humans playing a LARP. I’m not saying I see evidence they do—I just don’t know of convincing systematic obstructions. I agree that modern LLMs seem to not be able to do some things humans could do even in a LARP (some kind of theory of mind, explaining a consistent thinking trace that makes sense to a person upon reflection, etc.) but again a priori this might just be a skill issue.
So I wonder in the factorization “LLM can potentially get as good as humans in a LARP” + “sufficiently many smart humans in a long enough LARP are ‘Turing complete up to constant factors’ ” (in the sense of in principle being able to achieve, without breaking character, any intellectual outcome that non-LARP humans could do), which part would you disagree with?
“Potentially get as good as humans” I of course think in general, as I think we’re by default all dead within 100 years to AGI. If you mean actual current LLMs, I’m pretty sure no they cannot. See https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce and https://www.lesswrong.com/posts/5tqFT3bcTekvico4d/do-confident-short-timelines-make-sense
I would point you for example to low-sample-complexity learning that humans sometimes do, and claim that LLMs don’t do this in the relevant sense and that this is necessary for getting good. See also this thread: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce?commentId=dqbLkADbJQJi6bFtN
I am both experienced enough in text-based RP and have interacted with Character.AI enough to confidently assert that LLMs are not categorically different in their output from a poor-memory RPer, despite sometimes clearly different underlying patterns.
I see another obstruction in attention span. I strongly suspect that whenever an LLM is tasked with writing the next token, attention mechanisms compress all potentially relevant information into less than a hundred thousand numbers, preventing the model from taking many nuances into account when writing the token. A human brain, on the other hand, takes into account billions of bits of information stored in neuron activations.
@TsviBT, we had a warning shot of LLMs becoming useful in research or writing a coherent short story in @Tomás B.’s experiment (the post on LW describing it was removed for an unknown reason).
I moved it into my drafts. I published it again for you. I figured it was unlikely to be referenced again and I tend to take stuff down I don’t want people reading as one of the first things on my author’s page.
(FWIW, I’ve referenced that post 2-4 times since it was posted)