My favorite April Fools’ Day thing ever: Ramsey Theory and the History of Pre-Christian England: An Example of Interdisciplinary Research.
Gram Stone
Karma: 322
No nitpicks from Tampa. Cf. WHEN MUST I START KICKING AND SCREAMING AT YOU THAT IT IS FUCKING HAPPENING
I’m working on a piece to enter into the discourse about poverty lines and vibecessions and how hard life is actually getting in America, and hope to have that done soon, but there’s a lot to get through.
I’m excited to hear this, and I wanted to plug economist Gary Stevenson, who I think has the best combination of models, communications, and platform out of everyone I’ve seen speak or write on this issue. He’s from the UK and has a weak focus there, but he talks about the US aplenty and recognizes that this is a global problem that will only get worse.
I found this kinda hard to grok but gonna throw in my two cents anyway. Seems like priests (sensu stricto) obtain power by creating a body of ‘consensually true’ ‘expertise’ and hijacking human prestige bias adaptations, not really corresponding to reality because all theistic hypotheses are false, but not totally arbitrary because the ‘knowledge’ is culturally inherited. When someone tries to establish a new consensus, we call it a schism. This seems like one meta level up from what kings do and I’m not sure is faithfully captured by the obscure vs. arcane distinction. This also doesn’t directly contradict my vague impression that wars of succession are way more common than schisms historically, each of these being special cases of the parade going somewhere else than where kings and priests tried to lead them, respectively. I think this fits pretty well with lawyers as modern warrior-priests too (priests sensu lato of course). Law isn’t quite the same as theology I think, in that it’s a little less contingent than theology but more than science/CS/math.
I really like how many hypotheses you generated here.
A few months ago my GM pulled me in kind of in the way where you think you might get fired, he wants you to feel like you could hear a pin drop you know? He’s in his early fifties. Then he showed me a huge order where he had to ship a whole bunch of stuff separately to different locations and he was trying to calculate the weights and shipping costs of all these separate packages going to separate locations, and the non-AI solutions were failing him, so he copypasta’d into ChatGPT and it showed all its work, and he’s like “Look at this!”, and I just told him that he was having an appropriate emotional reaction because he’s old and almost everyone younger than him is a boiling frog or as clueless as he was ten minutes ago. That kind of stuff gives me hope.
Particularly I wonder how you would shed all that heat in a vacuum? I keep seeing arguments that the cold of space would make data centers easier to cool, but this totally contradicts my observation that all the spacecraft I’ve read about have dedicated systems to shed excess heat produced by electronic systems, as opposed to, say, having dedicated heating systems for life support and never having to worry about electronic systems generating excess heat because the cold of space just takes care of it? E.g. Apollo as I understand it mostly relied on the waste heat of systems dedicated to other functions, the Apollo 13 crew only had to worry about cold because they eliminated virtually all sources of waste heat simultaneously to conserve electricity. I think Apollo’s only dedicated systems for cabin temperature control were actually cooling systems. And this totally wouldn’t apply in the data center scenario.
I feel like this quickly glosses over the hypothesis that gestural language evolved first, or that they evolved simultaneously with significantly more sophisticated gestural behavior evolving earlier. I believe gestural language is much older than 500 ka (up to, let’s say, 2 Ma), which is consistent with the fossil evidence on vocalization adaptations.
It’s undeniable that some of the cognitive changes that occurred during human evolution affected motivation; in fact, in my view, I think proto-curiosity and proto-patience would have been favored by selection quite early. On the other hand, in my view, sustainable, scalable joint attention and behaviorally modern imitation learning (e.g. overimitation) are more complex and would have required more than just motivational changes. In particular, I don’t believe that most of the linguistic capability gap between chimps and humans can be explained as ‘motivational hobbling.’
F5 in Old World monkeys is very likely homologous to Broca’s area in humans, and although the gross neuroanatomy of humans and nonhuman primates is highly conserved, there are notable differences between the fine neuroanatomy of F5 in macaques and Broca’s area. Chimp F5 has intermediate features, but the evidence here is limited since we don’t do single-cell recordings in great apes anymore.
My own explanation for why there does not appear to be a derived gross language organ in humans is that F5 and Broca’s area both generate and interpret hierarchical act strings as such. Such a scheme would have several continuous parameters responsive to selection, including hierarchy depth, hierarchy breadth, goal maintenance duration and goal switching speed. I think at various scales this system is general enough to generate and interpret (i.e. socially learn) act strings for flintknapping, gestural and vocal language, controlled fire use, etc. I think this explains why chimps can also learn to knap, but their tools are worse than habilis, and I think it also explains many of the specific linguistic limitations observed in apes using sign and lexigrams.
MIRI got Guinan to read IABIED you guys https://www.facebook.com/share/v/1BJZz9MHSM/
It seems like a bad sign that, even with maximally optimistic inputs, your model never falsely retrodicts intelligence explosions in the past.
Not as immediately obvious as reflexively avoiding a branch, but on my model, you ought to think of language (first gesture, then speech) as an exaptation of cognitive adaptations for flintknapping. F5 in Old World monkeys is an interface between abstract goal representations and concrete action sequences, and F5 is believed to be the homologue of Broca’s area in humans. Looks like F5 generates and interprets hierarchical act strings as such (i.e. regardless of modality). You can hill climb on hierarchy depth, breadth, goal maintenance duration, switching speed (goals are active until completed, losing track of a goal being premature goal inactivation), which lends credibility to this idea. I think of Broca’s area as F5 with more atomic action categories + big model smell.
I agree with this as an object-level observation on the usefulness of MAV3D itself, but also have a Lucas critique of the Bitter Lesson that ultimately leads me to different conclusions about what this really tells us.
I think of EURISKO, Deep Blue, and AlphaGo/Zero as slightly discordant historical examples that you could defy, but on my view they are subtle sources of evidence supporting microfoundations of cognitive returns on cognitive reinvestment that are inconsistent with Sutton’s interpretation of the observations that inspired him to compose The Bitter Lesson.EURISKO is almost a ghost story, but if the stories are true, then this doesn’t imply that N clever tricks would’ve allowed EURISKO to rule the world, or even that EURISKO is better classified as an AI as opposed to an intelligence augmentation tool handcrafted by Lenat to complement his particular cognitive quirks, but Lenat + EURISKO reached a surprising level of capability quite early. Eliezer seems to have focused on EURISKO as an early exemplar of cognitive returns on recursive self-improvement, but I don’t think this is the only interesting frame.
It’s suggestive that EURISKO was written in Interlisp, as the homoiconic nature of LISPs might have been a critical unhobbling. That is to say, because Lenat’s engineered heuristics were LISP code, by homoiconicity they were also LISP data, an advantage that Lenat fearlessly exploited via macros, and by extension, domain specific languages. It also appears that Lenat implemented an early, idiosyncratic version of genetic algorithms. EURISKO was pretty close to GOFAI, except perhaps for the Search, but these descriptions of its architecture strongly suggest some intuitive appreciation by Lenat of something akin to the Bitter Lesson, decades before the coining of that phrase. It looks like Lenat figured out how to do Search and Learning in something close to the GOFAI paradigm, and got surprisingly high cognitive returns on those investments, although perhaps I have just made them seem a little less surprising. Of course, in my view, Lenat must not have fully appreciated the Lesson, as he spent the rest of his career working on Cyc. But for a little while at least, Lenat walked a fine line between the version of Engineering that doesn’t work, and the version that (kind of) does.
I would compare this distinction to the presence of folk semantics in all natural languages, and the absence of folk syntax. Parts of semantics are introspectively accessible to humans, so introspection permits veridical folk descriptions of semantics (informal, useful descriptions of what is true, possible, etc.), but the generator of syntax is introspectively inaccessible to humans, so generating veridical folk descriptions of syntax is much harder, if not impossible, via the same mechanism we applied in the case of folk semantics, thus successful computational modeling of syntax for the most part requires Science/Bayes (e.g. Linguistics). In my view, EURISKO was indeed mostly invented/discovered with Science/Bayes rather than Introspection, but this was hard for Lenat to tease out post mortem, and then he went way too far in the Introspection direction, failing to appreciate that most if not all of his cognitive returns came from mostly implicit, successful Science/Bayes (like mathematicians), which from the inside is hard to distinguish from successful Introspection. But Lenat’s ostensible error does not explain away the cognitive returns observed in the case of EURISKO, if we have in fact observed any.
Deep Blue demonstrated significant cognitive returns from massively parallel alpha-beta pruning + engineered evaluation functions and opening/endgame heuristics. Arguably, these were functions as opposed to data, but if we maintain the LISP/FP mindset for a moment, functions and source code are data. I can squint at Deep Blue as an exemplar of ‘feature engineering’ ‘working’ i.e., large allocations of engineering effort on a ‘reasonable’ (to humans) timescale, in concert with unprecedentedly ambitious hardware allocation/specialization and parallelism, permitting cognitive returns on cognitive reinvestment to exceed a critical threshold of capability (i.e. beating Kasparov even once, possibly even on an off-day for Kasparov).
Crucially, not all engineered features are brittle (or even unscalable, with respect to a concrete capability target, which is my model of Deep Blue), and not all learned features are robust (or even scalable, again with respect to a concrete capability target, which is my model of how DQN didn’t solve (i.e. meet the human expert capability threshold in the domain of) Go before AlphaGo (Go board state Search in ‘latent space’ was not ‘tractable’ with those permutations of compute, data, and (Learning) algorithm)), which might explain a thing or two about the weird competence profile of LLMs as well.
All of this to say, I don’t think about cognitive returns in a way that demands a fundamentally sharp distinction between Learning and Engineering, even if it’s been qualitatively pretty sharp under most historical conditions, nor do I think about cognitive returns in a way that forbids significant but reasonable amounts of pretty mundane engineering effort pushing capabilities past a critical threshold, and crucially, if that threshold is lethal, then you can die ‘without’ Learning.
As I hope might become especially clear in the case of AlphaGo/AlphaZero, I think the architectural incorporation of optimally specific representations can also significantly contribute to the magnitude of cognitive returns, as observed, I claim, in the cases of Deep Blue and EURISKO, where board states and domain specific languages respectively constituted strong priors on optimal actions when appropriately bound to other architectural modules (notably, Search modules in each case), and were necessary for the definition of evaluation functions.
I think a naive interpretation of the Bitter Lesson seems at first glance to be especially well-supported by the observed differences in capability and generality between the Alpha-series of architectures. You combine the policy and value networks into one, two-headed network, stop doing rollouts, and throw away all the human data, and it’s better, more capable and more general, it can perform at superhuman level in multiple perfect information, zero-sum, adversarial games besides Go (implicitly given their Rules in the form of the states, actions, and transition model of an MDP of course), and beat earlier versions of itself. But we also did the opposite experiments (e.g. policy head only Leela Chess Zero at inference time) and again an architecture with a Learning module but no Engineered Search module produced significantly lower cognitive returns than a similar architecture that did have an Engineered Search module.
Then MuZero wiped the board by tractably Learning the Rules in the latent space, but only four years after AlphaGo had reached the first target capability, and it still used Engineered Search (MCTS). To my knowledge, we are trying to build things that learn better search algorithms than we can engineer ourselves, but we still aren’t there. I’m not even claiming this wouldn’t work eventually, or that an engineered AGI architecture will be more general and more capable than a learned AGI architecture, I just think someone will build the engineered architecture first and then kill everyone, before we can learn that architecture from scratch and then kill everyone. On my model, returns on Search haven’t been a fundamental constraint on capability growth since right before EURISKO. On the other hand, returns on Engineered Game Rules (state space, action space, transition model), Compute, Data, and Learning have all been constraints under various historical conditions.
So I guess my model says that ‘merely static representations’ of semantic volumetric histories will constitute the first optimally specific board states of Nature in history, and we will use them to define loss functions so that we can do supervised learning on human games (recorded volumetric episodes) and learn a transition model (predictive model of the time evolution of recorded volumetric episodes, or ‘next-moment prediction’) and an action space (generative model of recorded human actions), then we will combine this with Engineered Search and some other stuff, then solve Go (kill everyone). Four years later something more capable and more general will discover this civilization from first principles and then solve Go, Chess, and Shogi with one architecture (kill everyone), and this trend will have been smooth.
If this isn’t pretty much exactly what Abram had in mind when he wrote:
Very roughly speaking, the bottleneck here is world-models. Game tree search can probably work on real-world problems to the extent that NNs can provide good world-models for these problems. Of course, we haven’t seen large-scale tests of this sort of architecture yet (Claude Plays Pokemon is even less a test of how well this sort of thing works; reasoning models are not doing MCTS internally).
then I might have to conclude that he and I have come to similar conclusions for completely different reasons.
I feel like explicit versions of the microfoundations implied by naive interpretations of the Bitter Lesson would falsely retrodict that we couldn’t beat Kasparov even once without ubiquitous adoption of ReLUs and late oughts/early teens amounts of compute, that DQN was sufficient to solve Go, and that EURISKO is a ghost story.
I didn’t talk about the neurobiology of constructive episodic simulation in humans at all, but would be willing to do so, and I think my model of that is also consistent with my microfoundations.
I view this from a heuristics and biases perspective. Mental health effects of LLMs seem well-explained as mediated by processing fluency as described by Schwarz et al. So it’s in the halo effect, mere exposure, confirmation bias, availability bias, etc., cluster. If there is a meaningful uptick in mental illness that could be intervened upon, this view suggests some funny interventions like making LLM outputs harder to read with a blur or drop shadow, or chewing popcorn while interacting with LLMs. Also, Cognitive Reflection Test scores (which could change within lifetime, despite apparently being pretty stable) should negatively correlate with susceptibility to LLM-induced psychosis.
Are you claiming NeRFs represent a substantial chunk of the Xth algorithmic insight?
I’m claiming that any future model that generates semantically rich volumetric histories seems to me to be implementing a simpler version of humans’ constructive episodic simulation adaptations, of which episodic counterfactuals, episodic memories, imaginary scenes, imagining perspectives you haven’t actually experienced, episodic prospection, dreams, etc. are special cases.
So ‘antepenultimate algorithmic insight,’ and ‘one of just a few remaining puzzle pieces in a lethal neuromorphic architecture’ both strike me as relatively fair characterizations. I have this intuition that some progress can be characterized more as a recomposition of existing tricks, whereas some tricks are genuinely new under the sun, which makes me want to make this distinction between architecture and algorithms, even though in the common sense every architecture is an algorithm; this feels fuzzy, relative, and not super defensible, so I won’t insist on it. But to describe my view through the lens of this distinction, more capable generalizations of stuff like MAV3D would be a critical module (algorithmic-level) in a generally intelligent neuromorphic architecture. Yes, you need other modules for this architecture to efficiently search for episodic simulations in a way that effectively guides action, and for taking simulation as an action itself and learning when and how to episodically simulate, and so on, but I’m specifically trying not to describe a roadmap here.Or not an algorithmic part, but rather setting up a data source
As far as I know we’re nowhere near exploiting existing video corpora as much as we could for training things like MAV3D, and yes, it seems to me we would be well-positioned to build synthetic datasets for future generative volumetric video models from the outputs of earlier models trained on videos of real scenes, and perhaps from data on VR users controlling avatars in interactive volumetric scenes as well. It seems like this would be easy for Meta to do. I’m more sure that this generates data that can be used for making better versions of this particular module, and less sure that this would be useful for generating data for other modules that I think necessary, but I do have at least one hypothesis in that direction.
I think you’re discounting the background stuff
Yes, we should distinguish between the ability to generate counterfactuals at all versus being able to use that ability instrumentally, but I was kind of trying to preempt this frame with “still don’t think this is the The Xth Insight.”
I mean, NeRFs were the beginning, but we can already generate 3d Gaussian splats from text prompts or single images, do semantic occupancy prediction in 3d splats, construct 4d splats from monocular video, do real-time 4d splats with enough cameras, etc., and it seems to me that doing these things opens the way to creating synthetic datasets of semantic 4d splats, which it further seems you could use to train generative models that would constitute Constructive Episodic Simulators, in which case on my model, actually yes, something akin to human episodic imagination, if not ‘true’ counterfactuals, should come right out of the box. By themselves, of course these modules will sometimes produce volumetric video analogs of the hallucinations we see in LLMs, not necessarily be very agentic by default, etc., so I don’t think achieving this goal immediately kills everyone, but it seems like an essential part of something that could.At the very least I guess I’m predicting that we’re going to get some killer VR apps in the next few years featuring models that can generate volumetric video at interactive frame rates.
Without, Azathoth forbid, implying too much depth here, this reminded me of that episode of Avatar: The Last Airbender where Toph sneaks up on Zuko while he’s sleeping to talk privately because she thinks he might actually be trustworthy but no one else in their group agrees with her and then he startles awake and reflexively burns her feet (which she uses to ‘see’ via an Earthbending-powered analog of echolocation, so in context surprisingly harmful) and then she runs away and he runs after her and grabs her so they can Just Talk and she strikes him in the chest with a pillar of earth (obviously aiming for the celiac plexus, but hey, she’s blind) and runs away and in his solitude Zuko lamentably exclaims, “Why am I so bad at being good?” Simplified for easy learning but exactly the problem Grayspaces are designed to solve.
Gram Stone’s Shortform
So I guess my model says that AGI R&D is basically the opposite of human evolution in a certain sense, actually basically all of the cognitive architecture necessary to cough up a human was in place by the speciation of chimpanzees, if not macaques, it really was just a matter of scaling (including having the selection pressures and metabolic resources necessary to connect previously unconnected modules, and don’t underestimate the generality of a ‘module,’ on my model), but like, if this took several complicated pieces, then if you’re relying on a different dependency structure (possibly like modern AI research, which the weirdly anachronistic capabilities of LLMs strongly suggest) with tons of money, time, energy, and hardware, you could enjoy way more abundance than human evolution and make the last of the algorithmic improvements that evolution made and suddenly get a system at least as capable as the human algorithm with that much hardware, and take over the world. I’m saying you should imagine human evolution thus far as having made way more algorithmic progress than our civilization because it was strongly constrained by resource availability.
I don’t have code that would end the world if I ran it, nor would I admit it if I did, but I feel like I have a good enough account of civilizational inadequacy in this domain, and a good enough model of human cognitive evolution, and ‘cultural evolution’, to conclude that LLMs are a massive enough boon to research productivity for key individuals/organizations to be a serious threat? I guess I feel like bimodal distributions can be reasonable by some kind of qualitative reasoning like, “How likely is it that I am merely two insights from AGI, as opposed to one or many?”
If I had to share some things that I don’t think would quickly end the world by being shared, given that they’re already public, and given that, if me pointing out this difference is likely to quickly end the world, it’s significant evidence in favor of massive civilizational inadequacy in this domain, which I would like everyone else to believe if it’s true and could save everyone, I guess the thing I would share would be the consilience between neural radiance fields and the constructive episodic simulation adaptations of modern humans? Like this is world-model-type stuff. If you can generate world-models from 2d visual frames at all, it seems to me that you are massively along the path of constructive episodic simulation, which gives you all sorts of benefits, like perspective-taking, episodic memory, prospective planning, detailed counterfactuals, I don’t know where to stop and I still don’t think this is the The Xth Insight.
Yeah I’ve been having a problem with people thinking I’m LLM-crazy, I think. But it’s what I believe.
I might have predicted decades ago that superhuman chess AI would end chess, but the game is more popular than ever. I suppose that the same could be true of fiction. If not, why not?