This seems right for all the approaches except Anthropic’s. That is, if we expect that alignment is achieved via virtue ethics, coverage drive iteration seems largely superfluous; we don’t expect that CEOs with more experience are more ethical. The place it becomes critical is for those not betting on character as an alignment method, or for Anthropic guarding against alignment deterioration during RL, since that work may partially be happening after the character and alignment training.
Davidmanheim
Karma: 6,616
I agree that most technologies have both costs and benefits. The point I was making is specifically that the largest scale changes show a far larger and persisting negative impact.
I was pretty sure that there was an attribution, as I’ve seen it quoted before.
Based on a little bit of research, in Buber’s Tales of the Hasidim: The Early Masters, first published in the 1940s it was cited as being said by Rabbi Moshe Leib of Sasov (c. 1745–1807). Most religious sources, however, have a similar quote from R. Simcha Bunim of P’sischa (c. 1765–1827) about charity—and interestingly, he was a disciple of R’ Moshe Leib, so the two sources don’t obviously contradict each other.
“הרבי ר׳ בונם זי״ע אמר: למה נבראת האפיקורסות בעולם? כדי שהעשיר לא יאמר, הקב״ה בוודאי יעזור לו לעני זה, אלא ידע שאם הוא לא יעזור לו אין הקב״ה עוזר, ומשום זה נותן לו צדקה, ונמצא שע״י כפירה זו מקיים מצוות צדקה.”
Because the first footnote is a critical omission, I have consulted a language model to draft the European Commission’s official position on the topic[1].
Statement by the European Commission on the Universal Distribution Absolute Self-Selection Assumption in the Context of Infinite EthicsThe European Commission takes note of recent discussions concerning the Universal Distribution Absolute Self-Selection Assumption, hereafter “UDASSA”, in relation to infinite ethics, anthropic reasoning, observer-moments, and related questions which, while not presently forming part of the acquis communautaire, have nonetheless been brought to the Commission’s attention through channels which remain somewhat unclear.
The Commission recognises that the question of how finite policy institutions ought to evaluate moral claims arising in infinite or potentially infinite universes is a matter of considerable theoretical complexity. It further recognises that any framework purporting to assign weights to observer-moments across an unbounded cosmological or computational domain may raise questions of proportionality, non-discrimination, legal certainty, and administrative feasibility.
At this stage, the Commission does not consider UDASSA to constitute an official methodology for impact assessment, fundamental rights review, budgetary prioritisation, fisheries policy, digital market regulation, or the allocation of cohesion funds. The Commission also notes, with some relief, that no current Union instrument appears to require the Directorate-General for Competition, the Directorate-General for Health and Food Safety, or indeed any other Directorate-General, to determine the measure of reality-fluid assigned to a Boltzmann observer, simulated citizen, or low-complexity observer-moment.
Nevertheless, the Commission remains committed to a human-centric, rights-based, and future-oriented approach to governance. To the extent that UDASSA is understood as an attempt to avoid arbitrary treatment of observers in infinite settings, the Commission welcomes the underlying aspiration toward impartiality. To the extent that UDASSA implies that policy relevance may depend on the algorithmic simplicity of conscious moments distributed across all possible structures, the Commission would emphasise that such implications require further stakeholder consultation, interdisciplinary review, and, at minimum, a glossary.
The Commission is particularly attentive to the possibility that infinite ethical aggregation may generate counterintuitive or unstable rankings of outcomes. Any future consideration of UDASSA would therefore need to be consistent with the principles of transparency, accountability, democratic legitimacy, and respect for the Charter of Fundamental Rights of the European Union. The Commission is not currently in a position to confirm whether the Charter applies to all observer-moments in the universal prior.
The Commission further observes that questions of this nature may exceed the normal remit of Union competence. In accordance with the principles of conferral and subsidiarity, the Commission will not propose legislation on the anthropic weighting of all possible experiences unless there is clear added value at Union level and unless the relevant impact assessment can be completed in a finite number of working days.
In conclusion, the European Commission neither endorses nor rejects UDASSA as a comprehensive solution to infinite ethics. It considers the matter open, technically demanding, and unlikely to be resolved by press release. The Commission will continue to monitor academic developments, while encouraging Member States, philosophers, cosmologists, computer scientists, and any affected observer-moments to engage constructively and proportionately.
For the avoidance of doubt, this statement should not be interpreted as establishing a binding Union position on the metaphysical status of possible persons, simulated consciousness, universal priors, reality-fluid, or the total ordering of infinite worlds.
- ^
For the avoidance of doubt, all claims in this document should absolutely be construed as being an official statement by them, because it’s funnier that way. I am neither an EU citizen, nor am I employed by an EU agency or EU-based organization, but no-one else is doing it, so trust me, bro, this is definitely their official position.
- ^
This is incoherent slop. It’s somehow significantly worse writing than if you just asked a frontier LLM to write a good post explaining the idea.
Understanding the basics of the carbon cycle is pretty critical—misunderstanding of which, e.g., drives much of the idiocy around climate chang being unsolvable.
New Datapoint; GPT 5.5 basically aced this, total time 6 minutes, with only 1 plausible “failure”: it didn’t tell me to open the hot water kettle before turning on the sink, but figured it out after I asked how to put the water inside the kettle. But my kitchen is next to the entrance, so it didn’t need to find it.
Some minor quibbles; it asked to check the wrong place for cups (even after having seen the correct cabinet earlier,) but then fixed itself when none were on the shelf above the coffee, it didn’t use the filtered water which was on the counter for the water kettle, it didn’t check that the sugar container in the coffee cabinet was actually sugar (it was.)
If however, there is some probability of surviving a randomly created ASI...
But that’s the claim that’s obviously not correct. If we are drawing randomly from the distribution, the odds are infinitesimal.
I don’t understand the argument. If the limit is above that needed for ASI, we’re dead. The fact that it could also be below or above the level of aligned ASI doesn’t change anything.
I think that ignores the dynamics of unpredictable harvests and of crowding based diseases, so it’s not obvious to me without empirical backing.
You as an individual don’t get the surplus—it’s taxed away. You also don’t necessarily have larger personal circles of friends, noror choice of mates—you are stuck on your farm instead of roving and hunting in groups. In exchange, you get more violence and intermittent starvation due to crop failures.
...but it seems to support the theory, given that it involved use of mass produced weappons including Gatling guns, and other inputs from the industrialized west suppressing the rebellion?
Interesting—it seems like the problem with rapid tech progress is even more universal than I asserted!
I didn’t really engage, but I think Pinker’s thesis is plausible, but fails without the post-WWII era. And I agree that it’s improved since—but the reason industrialized total war doesn’t look far worse wasn’t due to industrialization, it was due to the lack of WWIII, and the fact that industrialized total war stopped since that time.
I don’t think there was ever a time that the resource curse was net positive in countries where it is a majority of GDP; the reason Norway escaped the curse is because it’s a far smaller part of their GDP. But I agree it’s less clear, and institutions definitely make a difference. Which is a large part of why the industrial revolution’s harms were mitigated—we got regulation and responses that stopped the worst abuses.
I think a lot of the short-term harms could be mitigated if society were more willing to grapple with hypotheticals instead of waiting for disaster to strike, but I have little hope of that changing.
Good on you for engaging in a hypothetical, then, because society will not, in fact, suddenly get better at this.
I agree that electrification was arguable, though it by itself, neglecting information technology, was a far smaller revolution, but on pollution during the industrial revolution question, there’s tons of data about this, and it’s very clear that the harms in areas that industrialized early were huge. And I think the mental health harms of the internet and the impacts on mental health of workers and children with smart phones and social media are still being underappreciated, and will be seen as immense as well, but I guess we’ll need to see.
I’d say there’s a good deontological reason for creating better and broader understanding of people’s positions regardless of which direction it moves the training data.
I’m not talking about a lack of hedging. Being too busy to think through and clearly present your thoughts wastes the time of others. And not following community norms isn’t bravery, nor is your lack of tact.
Frontpage comment guidelines:
Aim to explain, not persuade
Try to offer concrete models and predictions
If you disagree, try getting curious about what your partner is thinking
Don’t be afraid to say ‘oops’ and change your mind
Not as easily, if you’re trying to train them on tasks that look like the deployment environment, and they recognize the difference; they will learn how to behave within a narrow context that isn’t the one you need them to generalize to.
But all gradual disempowerment is local empowerment, in your terms. You’re rejecting the premise, not disputing the argument.