I’m an independent researcher currently working on a sequence of posts about consciousness. You can send me anonymous feedback here: https://www.admonymous.co/rafaelharth.
Rafael Harth
Karma: 5,415
(My bad for not reading the footnotes!)
but my guess is that there’s some hypothetical philosophy pretraining corpus for which LLMs (post-RLAIF) could do philosophy at vaguely the same level that they’ve been doing math, which is “a very impressive level” in an everyday sense, and certainly a much higher level than how they do at philosophy today. As for “topic X”, no I don’t think LLMs can invent new concepts in the way that humans can, in math or philosophy, regardless of pretraining corpus, but that’s a different topic that I’m putting aside for this post, see footnote 1.
Right, you think that’s a qualitatively different category. To me, coming up with a new concept/field is just the central example of progress-on-stuff-that-can’t-be-easily-verified, and of a piece with the kind of thing you need to make progress on almost anything. So [doing good philosophy, but only in a known field] feels like an unnatural category (which isn’t a critique, I’m just explaining why this view is odd to me). Like where is the line between doing good philosophy and just summarizing the now-very-high-quality-literature?
So thinking about this, ummmmm so I think it would be able to answer all commonly asked questions correctly plus stuff that takes 1-2 reasoning steps… which probably does look like what you described, like it would look like pretty good philosophy to most people (I doubt many philosophy arguments need more than 1-2 steps), and certainly way way better than what they say now....… so it doesn’t look like we disagree much at all there.
… but I still think LLM limitations are very much exactly about verifiability. I would just say the reason LLMs are better at judging math is that math is easier to verify in an objective sense (and figuring out what exactly that means and how humans verify non-math stuff would be an extremely important step toward building AGI, so umm hopefully no one does it anytime soon)
“easy to verify” only matters for the RL part of LLM training pipelines
I think easy to verify matters for execution. I don’t see an argument in this post against this (which is fine if you just want to argue against some views).
LLMs are especially good at math because almost everything in the math literature is correct.
Okay but the reason math training data has low % of false statements is because math is easy to verify. So even if this were true, it would still be downstream of verifiability.
I suspect it’s also not true though. If we got rid of the bottom 95% of philosophy papers plus all papers on topic X, I doubt an LLM trained on that corpus could reproduce the good papers on topic X. (Do you think it could?)
I think you omitted a step in the argument where you would need to argue that “conceptually novel” work is necessary for an AI to cause human extinction. I lean “yes it is” (which would be good news if true) but haven’t thought about it very hard and don’t see it as super obvious.
This is not obvious? My impression from introspecting is that well over 90% of work I do on anything intellectual is conceptually novel. [1] Inventing new technology is definitely conceptually novel. Superhuman persuasion is definitely conceptually novel. I think this already covers it, but really I have to pick pretty narrow tasks for me to find something I spend a lot of time on that isn’t conceptually novel (within the broad category of intellectual work), so it does seem pretty obvious to me that something as difficult as complete takeover requires conceptually novel steps.
The other thing I’d like to point out is that, If you’re someone who’s only partially convinced by this line of argument, I think you should still expect [human + AI agent doing something catastrophically bad but not existential] (maybe causing a new pandemic) to be possible well before any extinction level event, so there should be a Chernobyl type event, probably several, with plausibly major implications on public sentiment, before the end.
- ↩︎
Not quite the right framing imo, but close enough.
- ↩︎
This is also why having LLMs “edit” your writing is often pernicious. LLM editing, unless managed extremely carefully, often involves rephrasings, added qualifiers, and swapped vocabulary in ways that meaningfully change the semantic content of your writing. Very often this is in unendorsed ways, but this can be hard to pick up on because the typical LLM writing style has a tendency to make people’s eyes slide off of it[5].
Hmm so I think this centrally depends on what you mean by editing? There’s
Give text to the LLM (with whatever instructions), then take its edited output
Tell the LLM to make suggestions about what to change in your text, then incorporate the suggestions to whatever extent you think makes sense—but do it entirely via manual editing, no copy-pasting
I’ve never done #1. I’ve never even considered doing #1, I think because the idea of publishing anything actually written by LLMs is just so emotionally yuck to me. But I do #2 all the time, to the point that not doing it seems like a weird decision for anything important. And I think #2 fundamentally avoids the problems you mentioned?
It is, although it was unintentional karma farming, I had no expectations that this would go above, idk, 7.
The issue is (was?) that it’s difficult to make a more high-effort post without singling out users, which I don’t want. This is why I didn’t say anything like this until now. But this is now solved with habryka greenlighting a private channel. If I’ll say anything else about this, I’ll do it there.
This begs the question of how to develop priors. I thought the benefit of Bayes is that it can converge on the best probabilities no matter your starting point, when you’ve been presented enough evidence, so long as you don’t assign anything a 0% or 100% prior.
This is true, and this still happens, the post didn’t say anything about convergence in the limit
This begs the question of how to develop priors.
For instance, you could be playing a videogame and you don’t know whether an enemy boss was programmed to cycle between both of its possible attacks randomly, or if it was programmed to be Switchy or Sticky. Then I think the “fallacy” presented by the OP would apply, wouldn’t it?
So two things about this
there is a theoretical answer on the priors problem, though it’s computationally intractable. It’s a huge rabbit hole, if you want to go down
(nice example!) the real answer here is that you don’t start accumulating evidence with the first boss hit, but well before that. Lots of things in the world give you information about how real people will most likely have programmed a boss in this case. Or more practically relevant, you’d consider your prior knowledge to affect your choice of the prior in this case. Pretty sure sticky here is pretty unlikely, and it’s either switchy or randomly.
Qualitative/intuitive impression, I didn’t apply any formalized metric.
Gotcha, I’ll at least consider writing a more comprehensive argument. Don’t mind including examples in a private context.
Is this a problem that mods can solve? I mean if I 100% convinced you, what would you do about it?
My judgement is obviously different, in that I want other people to freak out (well, I don’t actually want them to be anxious and fearful, but I can’t control that) in that I want people to realize what I think is happening and, if they agree, take short term actions that may buy us more time to do critical safety work.
Already said this, but want to repeat that I think the perspective is totally valid.
But in terms of the cold consequentialist calculus, I don’t know how you get to the “more alarming is a good idea” result. Maybe I’m biased because 2⁄2 cases I know well (myself and the friend I mentioned) low-key left the platform because the constant reminders are so crippling for mental health. I don’t have a survey on how bad other people feel. But my impression is that I see a post about AI acceleration more than half the time I look at the frontpage. Valentine wrote Here’s the Exit literally over three years ago! It was already so bad back then that people contemplated leaving the community over it. And it’s been going on ever since.
I genuinely believe that even if your utility function has zero terms in it other than maximizing useful AI interventions (whether policy or technical safety work, or anything else), you should want fewer posts like this. Everyone got the memo that it’s time to panic. I think the awareness-of-how-bad-it-is curve would have plateaued even if there were one fifth as many posts like this one, and the marginal effect of every other post is just to make people freak out more.
It’s definitely not a response to your post! I mean, yes I do think the post is an example of the pattern, but your posts in general are not, and it’s also not the most egregious example, the front page is not exactly filled with takes on this particular topic.
Im terms of being directly causally linked to me getting annoyed enough to write the short form, the other post I just complained about playes a much bigger role, and is a much more central example. But I’d still dispute that the shortform is a response to that. This is a complaint I’ve had for months and could have written at any point.
Don’t really want to name examples because it would single out individual authors. That’s why I didn’t include any in the first place.
Strongly agree that it correlates. It’s just not quite the same as most people agreeing with you. It prob measures something more like a lot of people strongly agreeing with you, with it mattering less how many people disagree with you. Like if on topic A, agreement/disagreement is 50⁄50, and on topic B it’s 90⁄10, but for A people feel super strongly and B people are mostly indifferent, then A might get rapid upvotes whereas B would just disappear from visibility immediately.
Anyway, much more importantly:
That said, I wrote this post mostly because I’m freaking out about the state of things and this is my best attempt to crystallize the source of that freak out. I’m not actually trying to make a rigorous argument about what counts as AGI, and I don’t even know if making a rigorous argument is worth it. You should probably read this post more as “Gordon is saying that he’s feeling the AGI hard, and you should do with the information about his judgement as a 25 year veteran of AGI discourse what you will.”
Big props for saying that! And this is actually one of the reasons why I perceive these posts as harmful, I think a lot of people are freaking out, this kind of post contributes to more people freaking out, and bc I think AGI isn’t very close, to me that just seems like a big net negative. But of course that depends entirely on your beliefs; if AGI were close, maybe it would be correct to freak out. This is what always makes it difficult to know what to say about posts like this; like to me the entire pre-apocalypse vibe is a huge negative for the overall utility of lesswrong, but I think people genuinely believe it, so unclear to what extent I should/can push back.
Fwiw my ire is much more directed at people upvoting this than at you for writing it. This seems to me like a pretty clear case: even if the narrative were 100% true, I don’t see how continuously broadcasting the vibe is a good idea. If I were an alignment researcher, I doubt it’d be good for my productivity. In fact I just talked to someone a few weeks ago on Discord who told me they don’t check LW much anymore for mental health reasons even though they still completely believe in the narrative and are trying to work on it. (They even deleted their account, which I thought was a bad idea.)
It’s getting increasingly easy to farm karma on LessWrong by making almost substantive-less posts that just articulate a position most people agree with. Imo this is a really huge problem, karma matters, it shapes what people see and it sets incentives.
As your good deed for this week, find something to strong-downvote that you agree with. I’d guess most people do that far too infrequently, maybe never.
Selection Bias. I completely disagree, [1] I just don’t know what to do with that so didn’t comment. “No I think you’re wrong”? This doesn’t seem like a really productive comment. “No I think AGI can’t do xyz yet?”. Maybe better but that will just open me up for another debate about this, and I’ve zero interest in debating it. Overall I didn’t see what kind of comment was worth writing, so I didn’t bother. This reaction is probably quite common, and at any rate, looking comments like this doesn’t tell you otherwise.
I’m only saying something now to push back on the “reception indicates agreement” point.
I’m not sure even upvotes indicate net agreement because people will have stronger inhibitions for downvoting. Many people who agree will just upvote because they agree. I doubt everyone who disagrees will downvote without formulating a critique. Though idk, iirc mods have encouraged liberal downvoting, maybe I’m wrong and people do it freely.
- ↩︎
For my view, I’ll just refer to Steven Byrnes’ formulation:
By “AGI” I mean here “a bundle of chips, algorithms, electricity, and/or teleoperated robots that can autonomously do the kinds of stuff that ambitious human adults can do—founding and running new companies, R&D, learning new skills, using arbitrary teleoperated robots after very little practice, etc.”
Yes I know, this does not exist yet! (Despite hype to the contrary.) Try asking an LLM to autonomously write a business plan, found a company, then run and grow it for years as CEO. Lol! It will crash and burn!
And I guess I’ll add that I don’t think LLMs are close to AGI, will lead to AGI, or anything like that, and yea I do also think claims to the contrary are net harmful for several reasons.
- ↩︎
Hmm, okay. So after reflecting on this a bunch, I think the things that still bug me about this post after reading your clarification aren’t about factual merits but about implications and tone. I’m not sure what the best practice in such a case is, maybe just not saying anything is best. I guess tell me if you prefer I didn’t write this response lol. But decided to say it this time.
I think this post is a) mostly attacking a strawman, in that most people who you think disagree with you actually do so for different reasons (although not everyone, I concede some people will disagree with the algorithm thing as you define it) and b) even insofar as they do exist, the net effect of this post will be substantially negative because it’s antagonizing, mocking, and unpersuasive.
E.g.:
The hilarious irony of psychedelics is:[4]
Objectively, psychedelics should be the most clear-cut evidence you could imagine for the idea that the brain is a machine that runs an algorithm, and that the mind is something that this algorithm does. After all, these tiny molecules, which just so happen to lock onto a widespread class of neuron receptors, create seismic shifts in consciousness, beliefs, perceptions, and so on.
…And yet, the people who actually take psychedelics are much likelier to stop believing that. Ironic.
This would feel right at home in r/sneerclub, which is odd to me because your posts usually have a very humble vibe. And yea I guess I don’t understand what your theory of mind is for how something good will result from anyone reading this.
But yea feel free not to reply, and can not express similar things in the future if you want.
What’s odd to me about this post is that it doesn’t define algorithm, so I don’t know what the claim is. [1]
The closest to a definition is the third-last section. Which I’d critique because (a) the definition should be at the beginning, but okay that’s just a nitpick about writing/structure, more importantly (b) I read it and I’m still not sure I know what the definition is, and (c) insofar as I understand it, I don’t think it maps onto conventional usage very well (see last section).
My best guess of what you mean based on that section is “anything that’s entirely about data processing in the widest sense is an algorithm”? But I don’t think that maps onto the common usage of the term very well. Also I don’t know how it applies to obscure cases. If you have a knotted wire and make the surface repulsive, it will ‘compute’ a way to disentangle, which is an ‘output’ in the sense that it corresponds to a set of topological transformations, but is this an algorithm? (Probably not because it actually does the disentangling itself, rather than just computing how you would disentangle, so therefore it’s not entirely about data processing?)
Edit: even the mechanical adder seems like an unclear case because it physically instantiates the solution, I guess it’s an algorithm because we don’t care about the physical instantiation, so in this case we can view it as only an informational output rather than a device that “does” anything? But this is not a crisp distinction; what if the mechanical adder were part of a larger system where the physical arrangement of marbles were utilized more?
If I did grasp the distinction correctly, then I don’t think this distinction is all that practically relevant. If the brain used some kind of quantum algorithm that had 100000x computational overhang if it were instantiated on a classical computer, [2] then your conclusion that you could replace the input/output mapping with a computer would still be true, so what does it prove? This seems to me like just a restatement of the Turing thesis, or I guess an application of the thesis to brains?
My practical concern RE (c) is that, imE, the concept of algorithm is generally seen as implying that the manner in which the brain does computation is similar to what a computer does, but that’s totally orthogonal to what you discuss here. So I can see people taking away the wrong thing.
I’m not particularly buying any of this. The central metaphor just doesn’t seem true (scratching an itch can be way more pleasurable than not having one, imE, and ditto with many other instances of receding unpleasantness) and I don’t think “[t]herefore what we usually take as pleasure is just scratching the sore of underlying suffering” follows from any of the stuff before (lots of types of pleasure for which this doesn’t seem true).
Have you read the most upvoted responses to your link?
yes. I don’t think any of them suggest that LessWrong is supporting or enthusiastic about OpenAI. (In particular, whether you should work there doesn’t have much relation to whether the company as a whole is a net negative.) I would describe the stance of top 2 comments on that post as mixed [1] and of LW’s stance in general as mixed-to-negative.
Lesswrong should have had much more enmity toward OpenAI,
Fwiw this is not a crux, I might agree that we should be more negative toward OpenAI than we are. I don’t think that’s an argument for laxer standards of critcism. Standards for rigor should lead toward higher quality criticism, not less harsh criticism. If you had attacked Greenpeace twice as much but had substantiated all your claims, I wouldn’t have downvoted the post. I’d guess that the net effectiveness of a community’s criticism of a person or org goes up with stricter norms.
- ↩︎
e.g., Ben pace also says, “An obvious reason to think OpenAI’s impact will be net negative is that they seem to be trying to reach AGI as fast as possible, and trying a route different from DeepMind and other competitors, so are in some world shortening the timeline until AI. (I’m aware that there are arguments about why a shorter timeline is better, but I’m not sold on them right now.)”
- ↩︎
Did they? I asked Claude (Opus) about this and it leaned mostly toward this not being true, but that they’re too young for this to be meaningful. It mentioned Fable being too expensive to be widely used in coding. Would/should investment (like into data centers) count as generating value?
This doesn’t seem cruxy for anything so feel free to ignore if it’s too complicated to answer.