Studying BTech+MTech at IIT Delhi. Check out my comment (and post) history for topics of interest. Please prefer seeing recent comments over older ones as my thoughts have been updating very frequently recently.
Discovered EA in September 2021, was involved in cryptocurrency before that, graduating 2023
profile: samueldashadrach.github.io/EA/
Last updated: June 2022
IMO your post would be more readable if it used the heading formatting available on LW
Thanks this is helpful!
I agree I could’ve phrased my comment better—I meant someone who eventually came to the conclusion that the less virtuous conclusion was better—not someone aiming for that as the target right from the start.
Interesting. I’d be keen to know if there are examples of people using internal double crux to not do the thing that is framed as more virtuous-on-reflection. For instance someone who debates between Netflix and running, and makes all the parts of himself okay with watching Netflix over running. Or someone debating working on AI alignment or making money, and deciding all of his parts are okay with the latter.
Especially in an EA frame, some people seem afraid of tolerating value drift (in themselves) that risks moving them away from whatever their current values and behaviours are. Are we ready to accept that some people will correctly choose less altruistic or ambitious paths as a result of the internal peace-making proposed in OP?
All the best!
FP often leads to long, winding discussions that may end with two researchers agreeing, but the resulting transcript is not great for future readers.
Highly agree. Often valuable content in discussions—either what you wrote or what the other person wrote—just gets lost.
Rereading your old discussions and then distilling the useful stuff into posts is a full time job.
This is a major reason I find very lengthy comments not often worth writing. I wonder if there is a way ro solve this problem. (Maybe more advanced search functionality on your comments?)
Just out of curiosity, how much of the burnout you mention is because of:
Working too hard, focusing too much on narrow plans and sacrificing other areas of your life Versus:
A world where you suddenly see high AI x-risk and s-risk, and nobody working on it, is just a fairly depressing world if you haven’t adequately calibrated to it.
Your post mostly aims at 1, but I wonder how much of it is 2.
Calibration is hard imo. People can (and should celebrate) even 0.1% changes in x-risk, it’s disorienting to suddenly update your whole world model from like 1% x-risk to 40%.
Random suggestion: A way to test whether a person is facing 1 or 2 might be for them to take a short break from work. If it’s 1 they may be more likely to feel better* than if it’s 2.
I wonder if that works, and would be keen on people’s thoughts!
*(Assuming ofcourse, that they accept the reasoning that a short break is good for them, and they don’t stress about the utilons they may feel theyre sacrificing while they’re away.)
Nice to know. 50-80 years might offer hope to people with longer timelines.
Why haven’t you been able to get a source of income?
I see!
How old are you? And where do you live? If you don’t wanna doxx, some rough idea would also work.
(And ofcourse, asking this isn’t meant to condescend or dismiss your thoughts, it’s just that your age seems relevant to plans that let you to change your situation.)
Thanks, will see if I get time!
Yudkowsky mentioned (in a video, sorry, don’t have link) that he doesn’t work on this path because he doesn’t believe we’ll get this tech first. I’m sure atleast some other AI safety researchers also believe the same.
But yeah I don’t have much knowledge on this either so am curious if it could change.
I live out in the middle of nowhere and going places is not generally an easy thing to do for me
This seems like a major bottleneck you should optimise towards solving. It would help if you work on solving this with a friend, or a therapist.
(And in general I’d be keen to know what’s your experience / opinion of therapists.)
Where can I read more on this?
There’s a big gap between “humans will have a communication channel with computers that is faster and higher throughput than keyboards and monitors” and “computers will understand human brain wirings to the point they can extract human values from it, and not be dangerous”.
Yup I don’t find this obvious either, not to 99% certainty for sure.
I get the chain of reasoning from coherent preferences to utility functions to instrumental converge and self-preservation. The case for an AI having coherent preferences seems weaker, esp. in paradigms like DL that at surface-level look non-agenty, or don’t have coherence hard-coded into them.
See also: Rohin Shah on goal-directed behaviour
Agreed!
It could also act as a filter against people who are aware at some level of the risks, but are mentally compartmentalising them too much to be capable of saying that “yes AI could kill everyone on earth”.
Agree with this!
If you’re maximizing expected utility of some sort, you often should focus more on where the long tails are, and less on whether those are in the 1-st order impacts or in n-th order impacts for some n.
Ultimately all impacts will add up (if they’re independent), and often what’s important is which ones have magnitude large enough that all other impacts can be ignored.
If you’re studying state-of-the-art AI you don’t need to know any of these topics.
They’re core to the agent model of AI, with coherent preferences. Once you get coherent preferences you get utility maximization, which gets instrumental convergence, uncorrigibility, self-preservation and so on.
Questions like:
how likely is it we build agent AI? Either explicitly or indirectly
how likely is it our agent AI will have coherent preferences?
Are still open questions, and different researchers will have different opinions.
Is there a longform discussion anywhere on pros and cons of nuking Russia post WW2 and establishing world govt?
It isn’t as obvious to me what the correct answer was, the way it is obvious to ppl in this discussion.
Also this seems like a central clash of opinion that will resurface in the AI race, or any attempts today to reduce nuclear risk.
Thanks for responding.
Opt-in sounds like a lot of cognitive overhead for every single comment
Maybe opt-out is better then, not sure.
Also, giving readers 2 axes to vote on is another form of cognitive overhead. If it is known that the second axis won’t be meaningful for a comment it might make more sense to hide the axis than have every reader independently realise why the second axis is not meaningful for a given comment.
This is a small downside, but may still be larger than the overhead placed on commentors to opt out of the second axis. Not sure.
allows for people to avoid having the truth value of their comments be judged when they make especially key claims in their argument.
I guess this has pros and cons, it isn’t immediately obvious which are more, I’d be keen to read anything about this.
One more thing is that my guess is the agree/disagree voting axis will encourage people to split up their comments more, and state things that are more cleanly true or false.
Possible. Although I feel I’d think more about this when writing comments if the second axis was opt-in and not present everywhere.
I’m not sure I follow.
Just to clarify, do you mean “most AI systems that don’t initially have coherent preferences, will eventually self-modify / evolve / follow some other process and become agents with coherent preferences”?
If yes, I would be keen on some evidence or arguments.
Because in my ontology, things like “without worrying about goodharting” and “most efficient way of handling that is with strong optimization …” comes after you have coherent preferences, not before.