Studying BTech+MTech at IIT Delhi. Check out my comment (and post) history for topics of interest. Please prefer seeing recent comments over older ones as my thoughts have been updating very frequently recently.
Discovered EA in September 2021, was involved in cryptocurrency before that, graduating 2023
profile: samueldashadrach.github.io/EA/
Last updated: June 2022
Interesting comment!
I also think this depends on how much work is being put into a problem. When there is significant scarcity of researchers then it might make sense for them to reason hard about whether what they’re doing actually helps. When there are many researchers the community as a whole can diversify its bets and people can pursue more lines of thought even though they’re not as sure how this ultimately helps with solving the problem yet.
I agree as AI safety field grows we should probably emphasize more decorrelation between research agendas, I wonder at what group size does that happen though!
Thanks for this comment! Working on fundamentals too has a theory of change though. Are you saying the theory of change is less clear when you do fundamental research?
Re: demos
Even if that comes under technical research, I feel like it may be useful to classify technical research based on what theory of change it is aimed at. Is the research aimed at “solving the alignment problem” or is it aimed at “convincing AI researchers the alignment problem is real and persuading them to work on it”?
I wrote this post some time back:
https://www.lesswrong.com/posts/oWKgzpB6sscnaxbJX/alignment-research-for-meta-purposes
Some people claim a lot of alignment research isn’t sufficiently laser-focussed on a theory of change. Assuming we do want research laser -focussed on specific theories of change, wouldn’t it be alright to explicitly start classifying research based on what its theory of change is?
Thanks for your reply!
I regret making the original comment, it feels nitpicky, my bad. But given that I started this discussion, I thought I should clarify since I feel we might have gone past each other.
My point was not around how you (Ben Pace) should personally delegate power. My point was around how people in general should delegate power. It is possible that since you already have certain ethical codes or epistemic norms and so on, the best way for you to coordinate with other people is indeed select for people who intrinsically hold very similar values and norms.
But someone in the Chinese Communist Party for instance, who already as a person believes in different ethical codes and epistemic norms, may not find it optimal to use your criteria to pick who to delegate power to. They may be acting optimally if they in fact picked for very different kinds of people.
For a broad variety of tasks I do think people of different codes can do well, as long as they coordinate with the right people (often people similar to them). And for some tasks having your integrity-based code of conduct may be reliably be a disadvantage.
You could argue ofcourse AI alignment just so happens to be the kind of problem where integrity and good epistemics and stuff actually gives you a reliable advantage, over groups coordinating using different norms. This could be true but one needs to separately defend why AI alignment happens to be such a problem rather than claim integrity-based codes of conduct are a dominant coordination solution to all problems.
Thanks this is fair!
Doesn’t your problem apply to all small social media platforms, not just mastodon? Do you have any idea how such platforms handle spam?
Also, could mastodon servers share or trade spam filters? Be it account lists or even just heuristics learned.
in an iterated game you should act as someone who can be given political power without concern about whether you will abuse it,
“whether you will be caught abusing it” not “whether you will abuse it”. For certain kinds of actions it is possible to reliably evade detection and factor in the fact that you can reliably evade detection.
Also, what is defined “abuse” is up to the person granting the power. If an organisation reliably accepts certains forms of corruption, the current leadership may want people who engage in those forms of corruption to be given power and brought into leadership. People seeking power with the organisation will correctly respond to these incentives by being corrupt. It is possible you are seeing people who are responding to the incentives of the landscapes they are trying to seek power in. And it is possible if you create a landscape with different incentives people won’t fall back to their old behaviours but show new ones instead.
Also a minor point but your comment points towards “integrity” being a sort of attractor when it comes to coordination (people with integrity can coordinate well with each other). But many not-so-integrous stratagies allow for coordination with other people running the same strategies. What is important for coordination is people being able to make credible commitments to each other, whether these commitments match any given code of ethics is a separate question.
I had made this point some months before. Glad go see many on LW agree now that it is better fleshed out.
https://www.lesswrong.com/posts/oWKgzpB6sscnaxbJX/alignment-research-for-meta-purposes
This is not surprising at all. If you find this surprising you might be interested in the timelines in which all these concepts were invented.
Morality-as-coordination where morality is usually virtue-ethical or deontological, is an ancient practice. Credible commitments are ancient. Examples include promising revenge on other tribes, or promising to kick people out of your tribe if they violate rules, even if actually executing this comes at cost to yourself.
Formalised study of moral philosophies came about mostly in the 18th century. Probability was invented in the 17th century, and Bentham proposed utilitarianism too in 18th century.
Prisoner’s dilemma was only formalised around 1950, where game theory by default tends to use expected utility maximisers as we understand them today. Deontological agents can trivially win a prisoner’s dilemma.
Acausal decision theory is even newer. It tries to plug this hole in utilitarianism, by sneaking in the morality-as-coordination stuff through a utilitarian framework.
Not sure how Explainable AI is used today, but DARPA’s original XAI program seeked to build AIs in a manner such that their behaviour would then be more explainable. This is different from first building AI using whatever technique makes it more capable, and then trying to understand anything about its behaviour.
https://en.wikipedia.org/wiki/Explainable_artificial_intelligence
Also “explanations” are I’m guessing a fairly strong version of interpretability. Identifying patterns of neurons that correlate with a model seeing french words could be interpretability research. But this may not fully “explain” how the model identifies french words (or if it is even trying to identify french words).
Thanks for the reply!
This is fair, I guess the actual question I was intending to ask was less about clones of you specifically, but more of people with similar values and worldviews around AI risk.
But I do appreciate not having a clear answer yet.
I’ve seen both these links before, thanks for sharing.
Here’s a question I would like to ask.
Do you think these failure modes (esp those you listed in the first ten scenarios) would still be important if the whole world was staffed by clones or near-clones of yourself?
If yes, why? What incentives or structures are preventing even capable leaders from coordinating towards good solutions?
If no, what would it actually take to get more people to see the world through your lens and why are you pessimistic on leaders in the real world being able to adopt this lens? (To attempt an easier version, assume you have an aligned AGI to help you with this task)
I’m not sure this matters, but if it does it makes sense for the entire EA forum and LW to be tagged in such a manner.
I kinda disagree around a lot of the terminology you’ve used. “I am okay with dying” is a value judgment people are allowed to make about themselves, and as a moral antirealist I’m not sold there exists an objectively right way to look at your own death.
And sure, once we have ways to extend lifespans it is likely more people will want longer than they claim to today. I’m not sure why this is epistemically bad. It is consistent to agree that a future event could change your value system if it happened, and yet choose not to change your value system until the event happens.
Humans care about other humans. Humans do not (usually) decide they care more about curing cancer than not killing humans. AGI by default can be indifferent as to our existence, and care only about curing cancer nothing else.
You may like Simulators, especially the section on Solving for physics.
Seems related to the $250k Inverse scaling prize, just wished to flag!
Check out Chapter 13 Trojans
https://course.mlsafety.org/
In theory yes it seems at first glance easy to add a poisoned token (trojan) into the dataset (used to train an LLM) such that it is undetectable to someone just has the poisoned model’s weights, but doesn’t know what prompt triggers the trojan. In practice it seems like there’s a more complicated attack-defence landscape, people aren’t able to hide their attacks as well as the theoretical ideal suggests. It is possible (to the best of my knowledge) though in the future that the landscape falls decisively in favour of attackers, and the only way to avoid trojans is to have trusted people verify every single token in the dataset and ensure no one else added poisoned tokens.
If you’re talking about AI risk that doesn’t involve developers deliberating implanting trojans, but instead the model itself behaving as if it had trojans implanted (without any developer’s intentions to plant one) - then I think you need to do more work to show why models might act like this (with noticeable probability).
The first person to reach AGI has incentives to prevent everyone else from reaching AGI too; they can also build out infrastructure to actually do this. Everyone is also explicitly aware of your scenario and will try to ensure AGI knowledge is more centralised than that
IMO even such people often have people they care about—including their families, nations, parties and so on. It is rare for someone to be born with no capacity to care about anyone whatsoever. If someone does exist of this form, yes they are a threat. An AGI could ofcourse be a bigger threat due to potentially larger intelligence, among other things.