Snippets on Living In Reality
Social reality is quite literally another world, in the same sense that the Harry Potter universe is another world. Like the Harry Potter universe, social reality is a world portrayed primarily in text and in speech and in our imaginations. Like the Harry Potter universe, social reality doesn’t diverge completely from physical reality—they contain mostly the same cities, for instance. Like the Harry Potter universe, social reality matches physical reality “by default” wherever there’s no particular pressure pushing against that match—after all, it’s just easier to make up fewer details rather than more. But like the Harry Potter universe, social reality does diverge in many places, and then the “fandom” tries to make up as coherent an interpretation as they can. Indeed, it’s that drive-toward-coherence which pushes both social reality and the Harry Potter universe to be fictional worlds, as opposed to just topics or genres about which people say lots of conflicting things.
And like the Harry Potter universe, sometimes people in the physical world do physical things as a result of goings-on in social reality.
What makes physical reality different from social reality or the Harry Potter universe or any other fictional world? Well, if we change all of our representations of the Harry Potter universe—every copy of every Harry Potter book and movie, every human’s recollections, etc—then we’ve effectively changed the Harry Potter universe. There’s nothing else left, besides the symbolic representations.
But if we changed every record of the night sky, altered every photograph and diagram, altered every human’s recollections, etc… then the next night an awful lot of people would look up and be very, very confused, because the stars would just keep doing their thing even with all of our representations changed. Physical reality is that one special world which we don’t get to choose just by moving around the symbols and representations.
While getting shawarma, I overheard some girls talking about some kind of conference they were at, sounded like a women in CS kind of thing? Anyway, one of them seemed only able to talk about the social impressiveness markers of the speakers. It sounded like she just… didn’t perceive the object level contents of anything they were saying at all? (Which, to be clear, may have been intended—i.e. maybe the speakers themselves were just trying to show their social impressiveness markers and didn’t say anything real at the object level.) Anyway, it was… weird to see this apparent inability-to-perceive reality, like she could only see the social world.
I live in reality. I like it here. It is my home. Insofar as others seek to escape it, that just leaves more reality for me to claim; and by choosing to live more fully in reality, I gain greater claim to it.
Google maps lives in reality.
LLM output usually doesn’t live in reality, at least not for very long. Like social reality and the Harry Potter universe, the worlds which LLMs write about are similar to ours by default wherever nothing in particular is pushing them to be different. But any time the LLM makes up some detail, and then continues to produce text in a manner consistent/coherent with that made-up detail, it’s diverged from physical reality and has begun to paint a fictional world.
Likewise, much/most internet content doesn’t live in reality. Much of it is explicitly fiction. Much of the remainder is social reality, or a picture of the world painted by the egregore of some political faction, or stuff that somebody made up but then backed up with a whole fictional world model (like e.g. some conspiracy theories).
For someone who doesn’t usually perceive physical reality much, someone who mentally lives in e.g. social reality or one of the political worlds, do LLMs’ worlds seem real?
“Should world” or “moral reality” is another different world.
Virtual reality (VR) shows other worlds. Augmented reality (AR) is intended to live in our world. That’s the root cause of the difference in difficulty between the two technologies: part of living in our world is that we don’t get to choose most of it just by saying things or by writing a few lines of code. We can rewrite an entire virtual reality just by changing the code which specifies that. We can’t do that for physical reality; there are parts beyond the programmer’s control. That’s the main reason why making VR apps is so much easier than making AR apps, to the point where the former is basically-solved and the latter is basically-unsolved.
Maybe LLMs are impressive in the same way as VR: they can display elaborate constructed worlds, but can’t deal with the real world. That would be because, whenever they make up some detail, however minor, there is no particular force to push them back into agreement with physical reality; no tight feedback loop. They just wander further and further off from physical reality as they generate more tokens, maintaining coherence to a large degree, and thereby creating a more and more fictional world.
Is the “fake vs real thinking” distinction about the same thing? Like, thinking which uses mental representations of the real world vs mental representations of other worlds? Is that the True Name of “real thinking”? If so, then tracking “which world does this live in” at all times could maybe lock in real thinking much more robustly.
It sure does match the opening CS Lewis quote perfectly: “There comes a moment when the children who have been playing at burglars hush suddenly: was that a real footstep in the hall?”
How would more data and software live in reality? What would the Great Generalized Google Maps project include, in what order? What data structures would it use for more abstract things?
How would this very essay be represented in the great generalized maps project?
I’d say Google Maps only mostly lives in physical reality, it does also contain things like political borders and the names of restaurants and such, and there are various pressures on the reviews displayed, and location data in China is slightly fuzzed, and maybe there’s other things like that. This is a nitpick, though.
I really like the comparison with literally fictional worlds, and the mention of “should” worlds. (I think I live in a superposition of social reality and normative reality more than I endorse.)
I think it’s more that modeling/thinking about the real world requires constant effort of paying attention/updating/inferring. Not doing that and operating off of a dream mashup is the default. So it’s not that there are real vs. other-world representations, it’s that grounding yourself in the real world requires sophisticated inference-time cognitive work, rather than just retrieving cached computations and cobbling them together.
(And then LLMs could only do the cobbling-together thing, they’re always living in the dream mashup.)
Per above, we’d need tighter feedback loops/quicker updates, appropriate markings of when content/procedures become outdated, some ability to compare various elements of constructed realities against the ground truth whenever the ground truth becomes known, etc. (Consider if Google Maps were updating very slowly, and also had some layer on top of its object-level observations whose representations relied on chains of inferences from the ground-true data but without quick ground-true feedback. That’d gradually migrate it to a fictional world as well.)
The general-purpose solution is probably some system that’d incentivize people to flag divergences from reality… Prediction markets?
A combination of more pervasive and reliable provenance annotation (citations, signed raw observations/recordings, trust-weighted endorsement of provenance claims, …) and discourse mapping (connecting supporting and contradictory arguments and evidence, clustering claims semantically, navigation of the emergent graph) would move some way towards this. FLF is among organisations working toward boosting societal collective intelligence by developing these sorts of tech. Many strategy and design variables to resolve! And questions of distribution/demand. But the design space looks newly feasible [[1]] and ripe for exploration.
for example through massive cheap clerical labour from suitably scaffolded LM agents, and semantically-sensitive software (again via LMs).