Yes, this is an example of how you can do things wrong if you don’t pay attention—but it’s also an example of when you should absolutely be accounting for that uncertainty, because if the value really is distributed as a function of the power of ten, ignoring that fact and only looking at the mean is going to be a really, really bad idea.
Davidmanheim
Karma: 6,304
Someone else doing something I think is ineffective doesn’t imply that it’s effective to do it! And yes, the fact that one side of a debate does the ineffective thing makes it more likely the other side will as well, but that’s not any sort of vindication! To quote myself:
Manheim’s Law of Positive-Sum Badness:
In polarized disputes, evidence that one side is stupid, malicious, or evil increases the probability that the opposing side is too.
I agree that if that were the entire argument, it would not be compelling. But the global experts who object aren’t saying it would not work, they are either saying it can’t happen, or that having a concentrated power developing AI (which is not at all the class of proposal being discussed would do) has downsides.
Of course, if you have better ideas, I’d be interested in hearing what they are, and an explanation about why they make it less likely we all die.
It’s not that they don’t count, it’s that they won’t go anywhere, and so aren’t going to dictate the actual contents of any actual future treaty. As I said later in the post:
Other proposals are actually trying to solve the problem directly. For example, the intentional equivalent of the ill-fated 1946-proposed Baruch Plan… Because directly solving the problem by fiat, imposed globally, before negotiations between parties start, is very hard and not usually effective.
If you don’t see a difference between “totally rules out” and “highly unlikely”, you REALLY need to go read the sequences.
I don’t see clear evidence here that engaging with the community on twitter has updated him much.
I didn’t say he engaged with the community on twitter. If you need direct evidence, show up at Lighthaven events he’s attending. Or go read posts where community members talk about Dean engaging with them—they disagree, sure, but looking at his updates over the past 2 years are how many bits of evidence towards our view against his prior views? (Again, read the sequences.)
First, if you want to understand the goals, look at the links at the beginning, or read IABED.
Second, the entire post was explaining why it’s not true that, as you suggest, details “should be preemptively put forward, because the countries need to know what they are negotiating for before sitting to the table.”
I don’t think that either of those guarantees that most of humanity dies, much less everyone. Especially the latter, given what is actually possible.
...which he updated heavily from since, after engaging honestly and reasonably with people from the community!
Also:
it is pretty apparent to me that much less than superintelligence is sufficient to kill us.
That seems incredibly not obvious, and I’d call it a straw man of your position if you hadn’t literally said it.
Do you mean “something less than a very strong superintelligence is sufficient” or do you mean “sufficient to do something that probably could kill us, if humans don’t pay much attention, and not with anything like certainty”?
My understanding of Dean’s position is that he totally rules out the possibility of AI wiping out humanity mainly based on this “superintelligence is not omnipotent” argument.
He said something like that in the past, but has updated greatly, and since then said AI causing human extinction is only “highly unlikely”, then even more recently said that “ai present catastrophic risks” and “alignment may become a more central issue for me again depending on how well alignment seems to work for smarter-than-human widely deployed ai”.
But again, overall I agree with your points—I just think it’s better not to be insulting about it, and give people like Dean who are engaging in good faith the benefit of the doubt.
First, Squiggle absolutely can make this easier.
Second, I think doing this well is simply a skill people need to develop if they want to reason about uncertainties well—they spend several years in school learning to multiply properly, then don’t even learn what the word convolution means, so it’s no wonder they aren’t correctly working with distributions.Using distributions is dangerous; if you get the tails wrong it can wreck you
Strongly disagree—people really should want to reason about uncertainties well, because almost no-one is a risk neutral decisionmaker. And trying to be one fails badly, because not looking at uncertainties works badly. Not trying to build robust strategies given uncertainties can ALSO wreck you—as we saw when a certain billionaire funder lost everything thinking he should keep playing in St. Petersberg Paradox. (And then committed fraud to stay solvent a bit longer, which, to be clear, wasn’t about not knowing how to convolve probability distributions.)
I think the points here are good, but it would be much better as a post if it was more respectful of Ball’s position, attempting to understand it instead of just attacking it. (Especially the conclusion.)
I agree that he’s not thinking about superintelligence, and I think the actual argument is about how much intelligence, even superintelligence, translates into ability to do useful work. Being really smart and working really hard simply isn’t enough to do things that are actually implausibly difficult. And if so the question is whether things that cause existential risk are implausibly difficult. (In Biorisk, the answer may be yes, though it’s very unclear. But for exfiltration, persuasion, and scheming, the answer is pretty clearly no.)
I’m not arguing that this is easy, I was not trying to make a formal argument that would explain half of what EA is trying to figure out in a single post, and I agree that we won’t resolve all of this, but there is room for converging on some of this.
1) Agree, the data isn’t great. As I always say, the hard apart of decision making under uncertainty is the uncertainty. BUT that doesn’t mean you don’t have good ways of making those decisions anyways.
2&3) You don’t need to start from zero, and you absolutely can hire people; OpenAI hired away Jacob Trefethen from OpenPhil/Coeff, and they could similarly hire folks from Gates Foundation, USAID, etc. The people they hire have track records, and you don’t need nearly as much taste to look at what they actually donated to, or the analyses of lives saved from those programs. And the biggest part of the reason OpenPhil is giving money slowly is because Dustin doesn’t want to spend the money faster; if he did, they could lower the bar for programs, and/or give more to GiveWell, which really is open to getting much more funding..
4) Absolutely. And giving in areas that aren’t well established, like AI safety, or where there are huge information asymmetries, like most academic research, is hard. But doing so in areas like global poverty or public health or infectious disease work is much easier, and there are mature ecosystems for evaluating impact. But that said, yes, there will be fraud and waste, and you should work to minimize it—but the burden of keeping fraud or waste to 0.1% is more than two orders of magnitude larger than what is needed in keeping it below 10%, and given the stakes and timelines, that’s a tradeoff to make.
As to your final comment, I agree there is not an off-the-shelf “donate $25 billion in way that is pretty-good” action, and yes, it requires a lot of skilled, intense attention. But that is a thing you can, in fact, buy, or hire, and as we know, money is the unit of caring—not just caring about the thing, but caring about the ability to do the thing.
But there absolutely is a “donate at least $2-3 billion in way that is pretty-good” action—specifically, fully fund Givewell’s request, and fully fund GiveDirectly’s room for funding. Is this bulletproof? No. Are there perfect actions? No. But delaying is incredibly costly, and yes, if you commit to giving over a hundred billion dollars, you do have a moral, ethical, and legal duty to get off your ass and do it.
(And there WAS a “donate $25 billion in way that is clearly very good if not necessarily as effective as we normally expect in EA” action available a year ago, when they committed the money, which would have been to fund all the global health projects that got dropped by USAID for, say, a 2-year ramp down period to ensure that they don’t get broken or other funders or programs get step in.)
You were responding to a post that said specific things, not entering a discussion where you present related ideas which you want to talk about. If you want to talk about the news the post was responding to, or to talk about their general lack of mission alignment, you can do that in a separate post—or at the very least say that you have a separate related point from the topic of article.
I don’t disagree—but none of this is related to the argument made here about charitable giving by the foundation, which is conditional on allowing that the planned surrender of control of the for-profit was an OK decision and accepting their other stated views about charitable mission, AI timelines, etc.
If you read the full post on the EA Forum, I think I addressed this clearly. They could pick a half dozen ready-to-go paths, obviously including Givewell, with stated room for billions in funding, Givedirectly, with stated room for well over $1b, and Coeff, which is now explicitly set up to support large cause-area-specific commitments. And they can do all of this while ramping up internal capacity.
And all of that is without drastically lowering the bar for funding to, say, $10k/life saved instead of $5k—which I’d bet would mean those orgs have room for closer to $50b/year in funding. Or simply re-reviewing the > $20b in funding cut by USAID and picking their favorite 50%.So I disagree, and claim it’s not nearly as hard to do this if they care about getting things done ASAP, since they’re actually rich enough to give ten billion this year, and double that next year, and do it again the year after, and double yet again the year after that—while still having capital left even if somehow the equity completely fails to appreciate in value—and they have a magically aligned ASI they’ve created before 2030 to fix everything else, or we’re all dead and at least they made things slightly better for people before that.
This is fantastic, tons of things I agree with strongly.
That said, my big undressed question is about scale; obviously it’s easier to fund one $1m project than 5 $200k projects, but the smaller projects are often higher leverage. And that goes for smaller things too.
So taking this much further, in my experience lots of really great early stage opportunities are $5k or $10k grants (help someone write a paper, or fund a small experiment to check if a new idea works,) which can have as much expected impact as a marginal $200k on different opportunities; how do you manage these, both in terms of filtering and finding them, and managing the relatively very high overhead costs for them? (Or do you not find that this is true, or do you have a minimum?)
I’d guess that humans have a short term memory of much more than ten bits, though. My intuition is that human short term memory would be much more comparable to neuralese vectors than to single tokens or 4 mental “items”.
Maybe, but I’d guess that’s a difference of less than an order of magnitude—and it seems like the relevant question isn’t only bits passed between circuits, since LLMs, even without reasoning, are autoregressive, so they can reason sequentially over multiple tokens. (And with reasoning, that’s obviously even more true.)
insofar as humans are agentic, they apply some general purpose search instead of semi-randomly picking their heuristics?
To the extent that LLM agents are agents, they definitely do this too! And if we’re talking about single-forward-pass reasoning, very few humans intentionally train their system 1 to do something better than semi-randomly follow patterns that worked before. (If you don’t know what I’m referring to, see the discussion of firefighters not actually making decisions and the resolution of the debate about system 1 / system 2 in Thinking Fast and Slow.)
I basically agree with the descriptive model, but don’t see that the conclusions follow.
For example:
The token bottleneck is real.
Sure, and so are limits like short term memory for humans. Doesn’t stop us.
And the same applies to only using shallow heuristics—humans mostly do the same thing.
If you’re entirely uncertainty insensitive and you’re looking at a distribution over final outcomes of the world rather than a specific compound variable, sure. But that’s almost never the case. I’m not saying that you should be naively taking the expectation of poorly thought through distributions, but it’s insane to oppose sensitivity analysis.
(Also, unlike in your example where you impose uniform uncertainties on the powers of the multiplied variables instead of on the variables themselves, it’s actually impossible to have variables that themselves have reasonably quantified distributions with means that match the expectation and then get a different expectation because you multiply them together. But I agree that what people often actually do is make stupid mistakes that break this.)