(I didn’t actually downvote but I probably would have if it wasn’t already downvoted). But you were downvoted because of decoupling norms being common among rationalists, where arguments are typically evaluated on their own merits. I agreed with Zac’s point that ControlAI is overestimating their chances and he openly says on his bio that he works at Anthropic.
Vivek S
Karma: 36
I just noticed a possible slight discrepancy in your numbers. In your post, you say:
Overall, I estimate that a marginal $3,500 donation increases Bores’s chances of winning by a little over 0.01%.
You append footnote 8 to that:
This is slightly larger than the estimate in my original post. That’s because my original estimate put some probability on the race ending up not being close, but the race has continued to look close.
But this estimate is lower than that of your previous post. From your previous post:
My best guess is that a marginal $85,000 donated on Monday, October 20th raises [Bores’] chances of winning by 1%
$85,000 for a 1% increase would mean $3,500 gives you a little bit over a 0.04% increase.
Yeah, I should have clarified—I meant donating $1M to Public First to support Bores (assuming Public First agree).
This seems to be an unusual opportunity where Coefficient Giving[1] can donate and not incur any reputational costs[2]. because of LTF / OpenAI / Palantir’s unpopularity.
For example if CG’s c4 arm donated $1M to Bores, then I would expect good first-order effects for reasons in the above posts but also positive second-order reputational impacts:
AI populists often attack AI Safety / CG on the grounds that they’re just increasing the AI hype and are secretly AI industry shills. It would be very easy to that point and say that the biggest funder donated significant money to the anti-LTF candidate in a way it almost never does.
Concrete example: according to this politico article about these very tensions, More Perfect Union’s Faiz Shakir was very (unfairly) critical of AI safety[3] but he said he liked Bores in the same article.
AI may later trigger huge backlash if the unemployment rate spikes. In that case, it is particularly valuable for the AI Safety movement to credibly signal not being AI shills so the work of Redwood/Apollo/METR ends up important.
Also, truth isn’t on LTF’s side since their anti-Bores ads are very disingenous [4](moreso than even other political attack ads). Having truth on your side generally has good second-order effects.
I’m not familiar with the actual views of prominent CG-people beyond public statements so I’m unsure if a CG donation is possible. If it is possible, though, it seems very valuable.
EDIT: After thinking it over, I feel more unsure about the second-order impacts of this now, mostly because CG’s involvement may cause AI populists to sour on Bores (though the direct impacts seem good). I still think having a clear thing to point to populists that AI Safety (especially CG) is very anti AI-accelerationism is good on the long run. But I’m unsure about how to accomplish it.
- ^
Same logic applies to other prominent and wealthy people in AIS but note that according to Claude, other AIS funders like SFF and Manifund don’t have a c4 vehicle and cannot donate from the fund itself—and that more recent reporting indicates Anthropic hasn’t actually donated to Bores.
- ^
My (possibly mistaken) understanding is that CG typically doesn’t donate from its c4′s to political candidates largely because reputational costs may harm EA cause areas
- ^
To quote: ” ‘The [institutional AI Safety coalition] is comfortable with the politics around this centrism, of coziness around AI development,’ said Faiz Shakir, the executive director of the progressive nonprofit media organization More Perfect Union and a senior adviser to Sanders. ‘They’re deeply concerned, I think, about those of us who have far more committed views around pausing their AI development, and they’re spending a lot of dollars to go after us.’ ”
- ^
See the first 2 minutes of this
Oops yeah, I believe you’re right. I got confused and I thought we had specified a) incomp b) but in reality we had only specified alignment research after a positive update is incomparable to b*).
Thanks for the reply! I think you’re right to push back on Lukas’ point about incomparability in practice. But I (obviously) think there’s a question to be had about imprecise intervals in theory.
My wording was a bit confusing, but I meant to say that (a) is incomparable to (b) — and that a) is incomparable to (c) from one standpoint. In my formulation of the problem, (b) is exactly equal to (c) by simple cluelessness (unlikely in practice but plausible in theory).
I thought of two ways you can try to hold on to complex cluelessness here and why they both seem to struggle. How would you respond?
Would you bite the bullet here and say that (a) and (c) (or, equivalently, (a) and (b)) are incomparable?
If so, we’re clueless about whether we should redirect alignment money when you have a negative update to worlds with a positive update — which I find very unintuitive.
If you try bracketing away this cluelessness, then I’m also skeptical.
It seems arbitrary whether to bracket before the update on alignment research or after the update, since there’s complex cluelessness in both cases.
Little bit late, but I’ve been reading about cluelessness and I think Lukas’ example is a very good intuition pump against radical cluelessness and this counterpoint doesn’t seem to work. The discussion moved elsewhere afterwards but this example strikes me as strong. In particular, I don’t think it matters whether research typically has the side effect in practice, because it is plausible (at least in theory) to have a world where it doesn’t.
To see this, take Lukas’ above example except assume that “research” will be done without a side effect. For example, “research” could mean taking 5 seconds to look up the recommendation of a newly released report. This report analyzes new information you did not previously know about and summarizes into a binary positive or a negative consideration. Assume that the new information this report gives is somewhat valuable, but you will continue to be radically clueless about alignment research after reading it. Also assume that any knock-on effects from the 5 seconds taken to look up this report cancel out (simple cluelessness) so b) and c) are equivalent (not just incomparable).
Worlds a) and b) and worlds a) and c) are incomparable from one standpoint because you are still radically clueless about alignment research being good. But from another standpoint should cluefully say that a > (b = c) because it is good to move $1000 from worlds where you get a negative update for alignment research to worlds where you get a positive update for alignment research. Due to this contradiction, I’m inclined to question the theoretical foundations of the story of imprecise intervals and radical incomparability.
Interesting points to mull over, thanks for linking! I agree with the general vibe here as well as most of the core ideas such as the benefits of non-market research and that governance work seems especially valuable. But I think Nielsen is too dismissive of looking to positively influence the power of influencing the direction of work.
I disagree with the first two sentences quoted. My guess is that there are other good ways to contribute beyond building governance and defensive capacity
Improving the value of the work of existing actors that will become powerful. Examples:
Improving the epistemics of AI companies. Even selfishly, AI CEOs are motivated against human extinction and company CEOs seem to significantly underestimate x-risk.
Being one of the ten people on the inside in a frontier lab. I think this is reasonably plausible if you have good epistemics and some non-trivial amount of intrinsic concern for impartial altruism[1]
Contributing to more valuable safety research than would have happened otherwise
Improving the epistemics of the US Government. Nielsen would probably agree here but the US government might also amass a lot of power post-AGI
Giving power to better / more responsible actors. Examples:
Doing good AI company scorecards / evaluations. This would also helps with a race to the top so it can help the work of existing actors too.
Export controls towards China. It’s overall better that US is first to AGI than China (assuming the weights aren’t subsequently stolen by China).
I think it very much depends on the person and their competitive advantages / disadvantages to pick working on decentralization or improving the work of existing actors or giving power elsewhere. However, I would guess that decentralizing power like Nielsen suggests is best for someone who is equally good at all possibilities.
I disagree here because of the above points. If the researcher maintains good epistemics, then I would guess that people doing technical alignment research at Anthropic / OpenAI are good for the world largely because they can improve existing actors[2].
I would guess most people seriously engaging with Nielsen have both
My p(x-risk) < 50%. If I update to a higher p(x-risk), I may change my mind here.