Coming back to this, I’m confused about why I wrote that a billion years would simplify the model to “launch immediately” for all non-extreme parameters (before you even talk about discounting or decreasing marginal utility). E.g. if the initial risk is 50% and there’s a 10% reduction per year, you obviously don’t want to launch immediately if you’re trying to maximize the average lifespan of currently living people.
kman
Karma: 1,816
I think we should focus on the tech VCs / libertarians.
I’m a bit puzzled by this. My impression is that these types have already been aware of the ideas for a while and in many cases are already polarized against notkilleveryoneism (because of Sinclair’s razor, government control scary/icky, e/acc, some are actual successionists, etc). I think the focus should be on helping the broader public understand what’s going on.
Seems obviously too narrow? What’s the point of restricting yourself to this from the more general “how are you getting around the hard part”?
I think you want to be really careful with this sort of thing. E.g. possibly this is part of what’s going on with the “disaster monkey” types? I.e. the guys that find working on AI super exciting in the moment and don’t really care that much what happens later or what happens to humanity.
Neuroticism has a very small correlation with IQ, something like R=-0.1 or a bit less. I guess that could be consistent with this picture: if you assume both things are normally distributed, if someone has 5 SD lower neuroticism we expect them to be 0.5 SD smarter. (The correlation between the underlying things could be higher, since personality measurements are very noisy). But I’m pretty darn skeptical. I get the sense that IQ testing isn’t affected that much by things like test anxiety in the first place, and also there are threshold effects for things like anxiety (e.g. either you’re in an anxious mood or not, neuroticism shifts the probability) rather than just a background level that can be higher or lower. If you aren’t anxious during an IQ test in the first place making you even less disposed towards anxiety won’t make you perform better. So I don’t see what the mechanism would be. Like if the difference was present from birth I could plausibly see the guy being less distracted by strong emotions throughout development, resulting in his brain being wired up differently. I guess it could be an adaptation that occurs on a much faster timescale than development… IDK, seems unlikely. I think you’d see a lot more drugs/interventions that affect behavior having associated IQ differences in that world.
Hm. I think a lot of these folks were off being much more optimistic than in this report circa 2015-2019, post DL but pre LLMs.
I was reading the “Singularity Summit 2011 Workshop Report” when I came across this:
After some discussion, participants gave their estimates for the probability of a good result (“win”) given various scenarios: e.g. prob(win | WBE or neuromorphic AI), prob(win | WBE), etc. Individuals’ estimates varied quite a bit. Zvi gave his personal estimate from a weighted average of others’ estimates, weighted by his estimate of each participant’s expertise. His given estimates were a roughly 14% chance of win if WBE or neuromorphic AI comes first and, coincidentally, a roughly 14% chance of win if de novo AI (Friendly AI or not) came first.
Or in modern lingo, P(doom | AGI) of 86%. These days Eliezer is the only participant that’s given a P(doom | AGI) number at least that high, and Zvi is the only other that’s even come close.
I have an impression that Eliezer was more optimistic than that back then (e.g. see here[1]), so I don’t predict that these numbers are due to his views being heavily weighted in the calculation. @Zvi do you have any recollection of how/why these numbers ended up so low?
A very incomplete, somewhat uncharitable list of possible explanations:
People on some level were like “now that AGI doesn’t feel like a distant abstract thing, I need to be optimistic in order to stay sane”
DL was somehow actually a huge positive update for many in a way that I’ve failed to understand
People such as Luke and Carl adopted Modest Epistemology and started deferring to those in the larger EA tent around them
The point of the exercise was to compare two counterfactual branches, so getting the absolute optimism of each branch right wasn’t prioritized and ended up lower than those involved would have endorsed at the time
- ^
Though one could perhaps interpret 14% as “in the more sloped region of the logistic success curve”; the slope there is only ~half that of at 50%.
“AI research becomes taboo” seems like a ~necessary ingredient of a successful/durable shutdown.
I personally feel super fucking pissed about the lack of virtue on display in the AI sphere by all these smart people that should know better, but don’t know how to channel this productively.
Society isn’t paying any attention to the METR task length forecasting to my knowledge (correct me if I’m wrong). Why expect this to be any different?
Do you see Carl as ‘EA’ or ‘rationalist’? Is he a counterexample to “My memory claims that it is really really not hard to tell, in advance, early on”?
You drew a distinction above between actual and performative epistemic modesty. Do your scathing criticisms of EA as a group only apply to those doing performative epistemic modesty?
I guess I’m curious whether he’d agree Carl is a counterexample to “My memory claims that it is really really not hard to tell, in advance, early on”, and if not, what exactly the distinction is.
To me this response feels incongruous with you describing his fund as “doing the worst possible thing about AI”, and with how you wrote many paragraphs on Leopold here. Carl has a 25-50% ownership stake in SALP and was described as the “co-portfolio manager” in SEC filings. He doesn’t have a small role there.
IIUC this whole research agenda assumes that you have a way to tell whether predicted short run behaviors lead to catastrophic long term consequences?
Does the research agenda generalize to when the AIs learn continuously (no longer have static weights in deployment)?
Seems clearly dual use in that if you can find greatly compressed explanations for how AIs accomplish cognitive feats, you can likely figure out how to make AIs that accomplish those feats much more efficiently.
Do you expect the sandboxes, or rather the strategies available to break out from them, differed widely between training episodes? To the greater an extent they varied, the more I expect it learned from breaking out of them.
If it’s the case that it was usually able to succeed at the rollout by breaking out, I guess we can infer the episodes did vary a lot, or it would have learned to win by breaking out in the same way every time?
Fair enough. I guess I want to get a sense of how hopeless an endeavor what I’ve proposed above is.
Obvious followup to this: would you be interesting/willing to work on human cognitive enhancement? If not, why?
Haven’t seen it discussed here so stating the obvious: why not try to increase the underlying propensity for strategic competence in addition to IQ? (Wisdom? Something to do with self awareness?) Human intelligence amplification need not mean just IQ.
Of course this will be more difficult than increasing IQ, because we don’t currently know how to measure the underlying propensity. Or worse, maybe it’s more of a complicated balance of things than simply dialing a propensity up. I’m interested to hear your thoughts on this.
I guess math very much isn’t like this. You can generally make at least some progress on a problem, but you can’t really make a problem worse except by giving bad advice to others working on it?
Partly, I am kinda assuming people have a harder time wrapping their brain around “what not to do” vs “what to do”
IDK, there being problems that are hard to fix but easy to make worse seems like a sort of common sense situation? Though I’m struggling to think of any good examples off the top of my head. Also just “we aren’t ready for the machine god, don’t help them build it faster” is super simple?
I don’t think it’s safe to scale up to AIs capable of 10x-ing the rate of economic growth.