That being said, this strategy relies on approaches that are fruitful for us and fruitful to AI assisted, accelerated, or done research to be the same approaches. (again reasonable, but not certain).
What is being excluded by this qualification?
Davidmanheim
Karma: 5,171
I strongly agree, and as I’ve argued before, long timelines to ASI are possible even if we have proto-AGI soon, and aligning AGI doesn’t necessarily help solve ASI risks. It seems like people are being myopic, assuming their modal outcome is effectively certain, and/or not clearly holding multiple hypotheses about trajectories in their minds, so they are undervaluing conditionally high value research directions.
Maybe we could look a 4-star generals, of which there are under 40 total in the US? Not quite as selective, but a more similar process. (Or perhaps around as selective given the number of US Catholics, vs. US citizens.)
You could compare to other strongly meritocratic organizations (US Senate? Fortune 500 C-level employees?) to see whether the church is very different.
The boring sense that is enough to say that it increases in intelligence, which was the entire point.
“infer a virtue-ethical utility function from a virtue-ethical policy”
The assumption of virtue ethics isn’t that virtue is unknown and must be discovered—it’s that it’s known and must be pursued. If the virtuous action, as you posit, is to consume ice cream, intelligence would allow an agent to acquire more ice cream, eat more over time by not making themselves sick, etc.
But any such decision algorithm, for a virtue ethicist, is routing through continued re-evaluation of whether the acts are virtuous, in the current context, not embracing some farcical LDT version of needing to pursue ice cream at all costs. There is an implicit utility function which values intelligence, but it’s not then inferring back what virtue is, as you seem to claim. Your assumption, which is evidently that the entire thing turns into a compressed and decontextualized utility function (“algorithm”) is ignoring the entire hypothetical.
OK, so your argument against my claim is that a stupid and biased decision procedure wouldn’t know that intelligence would make it more effective at being virtuous. And sure, that seems true, and I was wrong to assert unconditionally that “for virtue ethics, the derivative of that utility with respect to intelligence is positive.”
I should have instead clarified that I meant that any not idiotic virtue ethics decision procedure would have a positive first derivative in intelligence—because as your claim seems to admit, a less stupid decision procedure would not make that mistake, and would then value intelligence as it bootstrapped its way to greater intelligence.
I understand what an argument is, but I don’t understand why you think that converting policies to.utility functions needs to assume no systematic errors, or why, if true, that would make it incompatible with varying intelligence.
I don’t understand your argument here.
Yes, virtue ethics implies a utility function, because anything that outputs decisions implies a utility function. In this case, I’m noting that for virtue ethics, the derivative of that utility with respect to intelligence is positive.
My response was about your original PS, which was about this, not taboos.
I think the arguments you made there, and here, are confused, mixing up unrelated claims. The idea that some tasks will necessarily remain harder for AI than humans in the future is simply hopium.
Saying AI won’t be more efficient is obviously falsified for narrow tasks like adding numbers, and for general tasks like writing short stories, as LLMs currently do, the brain uses 20w/hour, and that’s about 30k tokens from GPT4o, i.e. it is done far more efficiently than a human.
And more generally, the argument that AI can’t be more efficient than the brain seems to follow exactly the same structure as the claim that AI can’t be smarter than humans, or the impossibility result here.
You should read the comments to that post.
I think this is confused about how virtue ethics works. Virtue ethics is centered on the virtues of the moral agent, but it certainly does not say not to predict consequences of actions. In fact, one aspect of virtue, in the Aristotelian system, is “practical wisdom,” i.e. intelligence which is critical for navigating choices—because practical wisdom includes an understanding of what consequences will follow actions.
It’s more accurate to say that intelligence is channeled differently — not toward optimizing outcomes, but toward choosing in a way consistent with one’s virtues. And even if virtues are thought of as policies, as in the “loyal friend” example, the policies for being a good friend require interpretation and context-sensitive application. Intelligence is crucial for that.
We believe it is more practical and sensible instead to commit to reconsidering the whole list of Capability Thresholds whenever we upgrade our safeguards.
I’m concerned because this change seems like starting down a slippery slope where it’s easy to change the rules which they previously said would apply, by making smaller changes instead.
This one shows the paperclip more clearly for me; 🙂↔️
Thank you, definitely agree about linking those as relevant.
It’s unclear to me how much economically-relevant activity is generated by low order correlation-type reasoning
I think one useful question is whether babbling can work to prune, and it seems the answer from reasoning models is yes.
I think we basically agree, but I think the Overton window needs to be expanded, and Pause is (unfortunately) already outside that window. So I differentiate between the overall direction, which I support strongly, and the concrete proposals and the organizations involved.
Very briefly, the fact that “The political position AI safety has mostly taken” is a single stance is evidence that there’s no room for even other creative solutions, so we’ve failed hard at expanding that Overton window. And unless you are strongly confident in that as the only possibly useful strategy, that is a horribly bad position for the world to be in as AI continues to accelerate and likely eliminate other potential policy options.
It would be more interesting to me to see this comparison repeated for models with a higher level of training data memorization / leakage like Mixtral or Llama, per this paper’s results; https://arxiv.org/abs/2502.12896v1
Similarly, looking at summarization performed by a model from a different vendor would arguably provide a stronger test for OpenAI models.
Or that the filter is far behind us—specifically, Eukaryotes only evolved once. And in the chain-model by Sandberg et al, pre-intelligence filters are the vast majority of the probability mass, so it seems to me that eliminating intelligence as a filter shifts the remaining probability mass for a filter backwards in time in expectation.