I don’t know if any/many Jews/Israelis actually celebrated, and if they did then why would that be; but the conspiratorial accusation towards Jews/Israelis in particular is that they caused or at least celebrated the attack because it would drag the US into fighting Israel’s enemies in the ME.
Separately, I could imagine [anyone who’s generally been a continual target of Islamic extremism] not exactly celebrating but (unwisely) seeing the upside “well at least now you get it”.
Hm. I thought I saw somewhere else in this comment thread that mentions this, but now I can’t find it, so I’ll put this here.
Sometimes mind is like oobleck ( https://www.lesswrong.com/posts/7RFC74otGcZifXpec/the-possible-shared-craft-of-deliberate-lexicogenesis?commentId=BHkcKpdmX5qzoZ76q ).
In other words, you push on it, and you feel something solid. And you’re like “ah, there is a thingy there”. But sometimes what actually happened is that by pushing on it, you made it solid. (...Ah I was probably thinking of plex’s comment.)
This is also related to perception and predictive processing. You can go looking for something X in yourself, and everything you encounter in yourself you’re like ”… so, you’re X, right?”; and this expectation is also sort of a command. (Or there could be other things with a similar coarse phenomenology to that story. For example: I expect there’s X in me; so I do Y, which is appropriate to do if X is in me; now I’m doing Y, which would synergize with X; so now X is incentivized; so now I’ve made it more likely that my brain will start doing X as a suitable solution.) (Cf. “Are you triggered yet??” https://x.com/tsvibt/status/1953650163962241079 )
If you have too much of an attitude of “just looking is always fine / good”, you might not distinguish between actually just looking (insofar as that’s coherent) vs. going in and randomly reprogramming yourself.
I’ll add a cluster of these, but first I’ll preface with an explanation. (Cf. https://www.lesswrong.com/posts/n299hFwqBxqwJfZyN/adele-lopez-s-shortform?commentId=99bPbajjHiXinvDCx )
So, I’m not really a fan of predictive processing theories of mind. BUT, an interesting implication/suggestion from that perspective is like this:
Suppose you have never before doubted X.
Now you proceed to doubt X.
When you doubt X, it is as if you are going from a 100% belief in X to a noticeably less than 100% belief in X.
We are created in motion, with {values, stances, actions, plans, beliefs, propositions} never yet having been separated out from each other.
Here, X is both a belief and an action-stance.
Therefore when you doubt X, it is as if you are going from a 100% action-stance of X, to a noticeably less than 100% action-stance of X.
In other words, doubting whether something is true, is equivalent to partly deciding to not act in accordance with believing it is true. (Or some even fuzzier version of this.)
(See also the “Nihilism, existentialism, absurdism” bullet point here https://tsvibt.blogspot.com/2022/11/do-humans-derive-values-from-fictitious.html )
Ok, so that’s the explanation. Now an answer blob to
“What [skills / character traits / etc] might reduce risk of psychosis, or might indicate a lack of vulnerability to psychosis, while also being good?”
Basically the idea is: A reverence / awe / fear of doubt. Which isn’t to say “don’t doubt”, but more to say “consider doubting to be a journey; the stronger, newer, and more foundational the doubt, the longer and more difficult the journey”. Or something.
A more general thing in this answer-blob is a respect for cognitive labor; and an attitude of not “biting off more than you can chew”. Like, I think normies pretty often will, in response to some challenge on some ideational point, just say something to the effect of “huh, interesting, yeah IDK, that’s not the sort of thing I would try to think through, but sounds cool”. A LW-coded person doesn’t say that nearly as much / nearly as naturally. I’m not sure what the suggestion should be because it can’t be “don’t think things through in uncommon detail / depth” or “don’t take ideas seriously” or “don’t believe in your ability to think through difficult stuff”, but it would be like “thought is difficult, some thoughts are really big and difficult and would take a long time, sometimes code refactors get bogged down and whole projects die in development hell; be light and nimble with your cognitive investments”.
(Speaking of development hell, that might be a nice metaphier for some manic mental states.)
Cf. the passage from Descartes’s Discourse on Method, part three:
And finally, just as it is not enough, before beginning to rebuild the house where one is living, simply to pull it down, and to make provision for materials and architects or to train oneself in architecture, and also to have carefully drawn up the building plans for it; but it is also necessary to be provided with someplace else where one can live comfortably while working on it; so too, in order not to remain irresolute in my actions while reason required me to be so in my judgments, and in order not to cease to live as happily as possible during this time, I formulated a provisional code of morals, which consisted of but three or four maxims, which I very much want to share with you.
all rapid psychological change will tend to cause it to some extent
Cf. https://x.com/jessi_cata/status/1113557294095060992
Quoting it in full:
forcing psych changes too fast is demanding that your code monkeys complete the code refactor on a harsh deadline, leading to technical debt.
cascading systems failures
Maybe important to keep in mind that this sort of “break” can potentially take lots of different “functional forms”. (I mean, it could have different macro-level contours; like, how many things are breaking, how fast and how thoroughly they break, how much aftershock they cause, etc.) See: https://tsvibt.blogspot.com/2024/09/break.html
I don’t know how this plays out specifically in psychosis, but ascribing intentionality in general, and specifically ascribing adversariality, seems like an especially important dimension / phenomenon. (Cf. https://en.wikipedia.org/wiki/Ideas_and_delusions_of_reference )
Ascribing adversariality in particular might be especially prone to setting off a self-sustaining reaction.
Consider first that when you ascribe adversariality, things can get weird fast. Examples:
If Bob thinks Alice is secretly hostile towards Bob, trust breaks down. Propositional statements from Alice are interpreted as false, lies, or subtler manipulations with hidden intended effects.
This generally winds Bob up. Every little thing Alice says or does, if you take as given the (probably irrational) assumption of adversariality, would rationally give Bob good reason to spin up a bunch of computation looking for possible plans Alice is doing. This is first of all just really taxing for Bob, and distracting from more normal considerations. And second of all it’s a local bias, pointing Bob to think about negative outcomes; normally that’s fine, all attention-direction is a local bias, but since the situation (e.g. talking to Alice) is ongoing, Bob may not have time and resources to compute everything out so that he also thinks of, well maybe Alice’s behavior is just normal, or how can I test this sanely, or alternative hypotheses other than hostility from Alice, etc.
This cuts off flow of information from Alice to Bob.
This cuts off positive sum interactions between Alice and Bob; Bob second guesses every proposed truce, viewing it as a potential false peace.
Bob might start reversing the pushes that Alice is making, which could be rational on the supposition that Alice is being adversarial. But if Alice’s push wasn’t adversarial and you reverse it, then it might be self-harming. E.g. “She’s only telling me to try to get some sleep because she knows I’m on the verge of figuring out XYZ, I better definitely not sleep right now and keep working towards XYZ”.
Are they all good or all out to get me? If Bob thinks Alice is adversarial, and Alice is not adversarial, and Carmi and Danit are also not adversarial, then they look like Alice and so Bob might think they are adversarial.
And suppose, just suppose, that one person does do something kinda adversarial. Like suggest that maybe you really need to take some sort of stronger calming drug, or even see a doctor. Well, maybe that’s just one little adversariality—or maybe this is a crack in the veneer, the conspiracy showing through. Maybe everyone has been trying really hard to merely appear non-adversarial; in that case, the single crack is actually a huge piece of evidence. (Cf. https://sideways-view.com/2016/11/14/integrity-for-consequentialists/ ; https://en.wikipedia.org/wiki/Splitting_(psychology))
The derivative, or the local forces, become exaggerated in importance. If Bob perceives a small adversarial push from Alice, he feels under attack in general. He computes out: There is this push, and there will be the next and the next and the next; in aggregate this leads somewhere I really don’t want; so I must push back hard, now. So Bob is acting crazy, seemingly having large or grandiose responses to small things. (Cf. https://en.wikipedia.org/wiki/Splitting_(psychology) )
Methods of recourse are broken; Bob has no expectation of being able to JOOTS and be caught by the social fabric / by justice / by conversation and cooperative reflection. (I don’t remember where, maybe in some text about double binds, but there was something about: Someone is in psychosis, and when interviewed, they immediately give strange, nonsensical, or indirect answers to an interviewer; but not because they couldn’t give coherent answers—rather, because they were extremely distrustful of the interviewer and didn’t want to tip off the interviewer that they might be looking to divulge some terrible secret. Or something in that genre, I’m not remembering it.)
Now, consider second that as things are getting weird, there’s more grist for the mill. There’s more weird stuff happening, e.g. Bob is pushing people around him into contexts that they lack experience in, so they become flustered, angry, avoidant, blissfully unattuned, etc. With this weird stuff happening, there’s more for Bob to read into as being adversarial.
Third, consider that the ascription of adversariality doesn’t have to be Cartesian. “Aliens / demons / etc. are transmitting / forcing thoughts into my head”. Bob starts questioning / doubting stuff inside him as being adversarial, starts fighting with himself or cutting off parts of his mind.
Mhm. Fair enough. I mean I believe you about having good AUROC. I think I would too, except that I don’t really care whether someone used LLMs at all or even substantially; rather the thing I’d both care about and also expect to have fairly good discernment ability for, is “Is the contentful stuff coming from a human mind?”. E.g. the bullet point
Nuclear treaties succeeded by banning concrete precursors (zero-yield tests, 8kg plutonium, 25kg HEU, 500kg/300km delivery systems), not by banning “extinction-risk weapons.” AGI bans need analogous thresholds: capabilities like autonomous replication, scalable resource acquisition, or systematic deception (these are merely illustrative examples, not formal proposals).
sounds pretty clearly like human content, and it’s useful content. So at that point I don’t care much whether some glue sentences or phrasing is LLMed. I think both my own writing and other people’s writing, before LLMs were a thing, already frequently contained some admixture of lazy wording and phrasing, which amounts to basically being LLM slop. I think that’s fine, writing everything strictly using only living language would be a lot of work. And you just skim those parts / don’t think about them too hard, but they’re there if you needed more signposting or something. (Not presuming you necessarily disagree with any of this, just riffing.)
I just want to say that all the things you listed (except maybe the last one, not sure) are things that I use routinely—as well as other LLM-associated things, such as liberal em-dashes—and I endorse my doing so. I mean it’s fair to use them as yellow flags indicated LLM writing, but if we delete the words like “excessive” from your descriptions these behaviors don’t seem inherently bad. I do think LLM writing is bad, both because
the actual words are bad (e.g. frequent use of vague words which, like a stable-diffusion image, kinda make sense if your eyes are glazed over but are meaningless slop if you think about it more sharply), and also because
LLM writing where any words that are apparently-meaning-bearing were principally added by the LLM will fail to be testimony, which destroys most of their value as communication: https://www.lesswrong.com/posts/KXujJjnmP85u8eM6B/policy-for-llm-writing-on-lesswrong?commentId=MDtbuQZcaXoD7r4GA
Thanks for writing this!! It would be great for someone to fund some law experts to work with AGI alignment thinkers to work out a viable AGI ban.
In pretty full generality I’m sorta skeptical of adult interventions. So much important stuff happens in development, and the ability for that important stuff to happen is surely lost to a large degree (who knows how much). But yeah this seems sorta interesting, though prima facie I’d be worried about cancer and also worried about ruining existing stuff. @kman
If you can make “golems” that transparently, mechanically enforce the rules, you get even more screening off. E.g. a computer-programmed game.
Conversely, contexts where high-level actions do screen off intent are hard to construct and fragile, and sometimes valuable, therefore precious. For example, you can have a tournament arbiter who enforces the rules completely and accurately. Ze might show disdain or admiration or whatnot towards some players, but if the rules are well-constructed and ze enforces them, in principle the screening-off could be near complete. (Or not, e.g. if the non-verbals nudge players to not bring up rules objections.) On the other hand, if there’s even a small hole in the rules or probability of discretion in enforcement (as there usually is), the screening off is destroyed.
required intelligence thresholds (for the individual scientists/inventors involved) are only weak evidence about relative problem difficulty (for society, which seems to me the relevant sort of “difficulty” here).
This sounds right, yeah. If I had to guess, I would guess AGI alignment is both kinds of problem (Maxwell/Faraday equations, and rockets).
I don’t know why you believe it,
So I wrote:
I think the threshold of brainpower where you can start making meaningful progress on the technical problem of AGI alignment is significantly higher than the threshold where you can start making meaningful progress toward AGI.
Simply put, it’s a harder problem. More specifically, it’s got significantly worse feedback signals: it’s easier to tell when / on what tasks your performance is and is not going up, compared to telling when you’ve made a thing that will continue pursuing XYZ as it gets much smarter. You can also tell because progress in capabilities seems to accelerate given more resources, but that is (according to me) barely true or not true in alignment, so far.
My own experience (which I don’t expect you to update much on, but this is part of why I believe these things) is that I’m really smart and as far as I can tell, I’m too dumb to even really get started (cf. https://tsvibt.blogspot.com/2023/09/a-hermeneutic-net-for-agency.html). I’ve worked with people who are smarter than I am, and they also are AFAICT totally failing to address the problem. (To be clear, I definitely don’t think it’s “just about being smart”; but I do think there’s some threshold effect.) It’s hard to even stay focused on the problem for the years that it apparently takes to work through wrong preconceptions, bad ideas, etc., and you (or rather, I, and ~everyone I’ve directly worked with) apparently have to do that in order to understand the problem.
This mechanism seems weak to me.
We probably disagree a little. I’d bring up a few points. E.g. I’d point out that you and I reason about X-risk better than them in large part due to the fact that we pay attention to people who are smarter than us and who are good at / ahead of the curve on reasoning about X-risk. E.g. I’d point out that more intelligence leads to more intellectual slack (meaning, if you can think faster, a given argument becomes less costly in time (though maybe not in opportunity) to come to understand). E.g. I’d point out that wisdom (in this sense: https://www.lesswrong.com/posts/fzKfzXWEBaENJXDGP/what-is-wisdom-1) is bottlenecked on considering and integrating many possibilities, which is a difficult cognitive task.
But, I agree it’s not that strong a reason.
Realistically I do not think there is any level of genetic modification at which humans can match the pace of ASI.
Yeah I agree, but that’s not the relevant threshold here. More like, can humanity feasibly get smart enough soon enough to be making a bunch of faster progress in material conditions, such that justifications of the form “AGI would give us much faster progress in material conditions” lose most of their (perhaps only apparent) force? I think probably we can.
I just didn’t find the given reasons convincing.
All of them, or just these two?
Oh, IDK. Thermonuclear weapons and viral human infectiousness gain of function research? I just mean, there’s probably a few things that we want everyone to know not to do; the Noahide Laws, if you will.
What I meant by general domain is that it’s not overly weird in the mental moves that are relevant there, so training methods that can create something that wins IMO are probably not very different from training methods that can create things that solve many other kinds of problems.
I took you to be saying
math is a general domain
IMO is fairly hard math
LLMs did the IMO
therefore LLMs can do well in a general domain
therefore probably maybe LLMs are generally intelligent.
But maybe you instead meant
working out math problems applying known methods is a general domain
?
Anyway, “general domain” still does not make sense here. The step from 4 to 5 is not supported by this concept of “general domain” as you’re applying it here.
I’m unsure whether or not we disagree, because I think our disagreement would be quantitative (probabilities, perhaps relative priorities) rather than qualitative (which considerations are coherent, plausible, relevant, etc.). My basic direct answer would be: Because effectful wise action is more difficult than effectful unwise action, and requires more ideas / thought / reflection, relatively speaking; and because generally humans want to do good things.
Maybe more intelligent humans just race to AGI faster because of coordination problems.
True, maybe.
I think the threshold of brainpower where you can start making meaningful progress on the technical problem of AGI alignment is significantly higher than the threshold where you can start making meaningful progress toward AGI. So more people around / above that threshold is differentially good in terms of technical progress. (It could still be bad overall, e.g. if you think there’s a fixed-rate process of reflection that gets outrun faster in those worlds.)
I think having a lot more super smart people makes other aspects of the situation better. For example, the leaders of AGI capabilities research would be smarter—which is bad in that they make progress faster, but good in that they can consider arguments about X-risk better. Another example: it’s harder to give even a plausible justification for plunging into AGI if you already have a new wave of super smart people making much faster scientific progress in general, e.g. solving diseases.
I think / hope that current memecrafting means that young smart people growing up will have the background sense that AGI is special and is one of the ~three things you really do not mess around with ever. BUT, if in fact it becomes clear that young super bright kids are actually, currently, growing up much more enthusiastic about AGI, that cuts against this a lot, and would significantly cut against my motivation towards reprogenetics (well, it would make me prioritize much higher stuff that affects the memesphere of newborn geniuses).
Maybe more intelligent terrorists release better bio-weapons.
AFAIK this is largely sci-fi? I mean you can find a few examples, but very very few super smart people are trying to do huge amounts of damage like that; and it’s even way harder to get a large and/or very expert group together to try to do that; and without a large/expert group it’s really hard to get super-high-effect things working, especially in biology, like that.
Maybe after every reasonable boost for biological human brains we still aren’t smart enough to solve alignment, perhaps because it’s totally intractable or perhaps because we can’t really get that much smarter, and we muddle along for awhile trying to stop everyone from building AGI until something out of left field wipes us out.
Yeah, true, this is plausible; I still think our chances are significantly in those worlds, but your point stands.
in a somewhat general domain,
The way you’re using this concept is poisoning your mind. Generality of a domain does imply that if you can do all the stuff in that domain, then you are generally capable (and, depending, that could imply general intelligence; e.g. if you’ve ruled out GLUT-like things). But if you can do half of the things in the domain and not the other half, then you have to ask whether you’re exhibiting general competence in that domain, vs. competence in some sub-domain and incompetence in the general domain. Making this inference enthymemically is poisoning your mind.
For example, suppose that X is “self-play”. One important thing about self-play is that it’s an infinite source of data, provided in a sort of curriculum of increasing difficulty and complexity. Since we have the idea of self-play, and we have some examples of self-play that are successful (e.g. AlphaZero), aren’t we most of the way to having the full power of self-play? And isn’t the full power of self-play quite powerful, since it’s how evolution made AGI? I would say “doubtful”. The self-play that evolution uses (and the self-play that human children use) is much richer, containing more structural ideas, than the idea of having an agent play a game against a copy of itself.
Most instances of a category are not the most powerful, most general instances of that category. So just because we have, or will soon have, some useful instances of a category, doesn’t strongly imply that we can or will soon be able to harness most of the power of stuff in that category. I’m reminded of the politician’s syllogism: “We must do something. This is something. Therefore, we must do this.”.
(I will abstractly state that I feel negatively towards the group dynamics around some AI debates in the broader EA/LW/AI/X-derisking sphere, e.g. about timelines; so, affirming that I feel “knee-deep” in something, or I would if my primary activity were about that; and affirming that addressing this in a gradual-unraveling way could be helpful.)