Vote on Interesting Disagreements
Do you have a question you’d like to see argued about? Would you like to indicate your position and discuss it with someone who disagrees?
Add poll options to the thread below to find questions with lots of interest and disagreement.
How to use the poll
- Reacts: Click on the agree/disagree reacts to help people see how much disagreement there is on the topic.
- Karma: Upvote positions that you’d like to read dialogues about.
- New Poll Option: Add new positions for people to take sides on. Please add the agree/disagree reacts to new poll options you make.
The goal is to show people where a lot of interesting disagreement lies. This can be used to find discussion and dialogue topics in the future.
Poll For Topics of Discussion and Disagreement
Use this thread to (a) upvote topics you’re interested in reading about, (b) agree/disagree with positions, and (c) add new positions for people to vote on.
Note: Hit cmd-f or ctrl-f (whatever normally opens search) to automatically expand all of the poll options below.
Prosaic Alignment is currently more important to work on than Agent Foundations work.
LLMs as currently trained run ~0 risk of catastrophic instrumental convergence even if scaled up with 1000x more compute
Academia is sufficiently dysfunctional that if you want to make a great scientific discovery you should basically do it outside of academia.
Pursuing plans that cognitively enhance humans while delaying AGI should be our top strategy for avoiding AGI risk
Current progress in AI governance will translate with greater than 50% probability into more than a 2 year counterfactual delay of dangerous AI systems
Ambitious mechanistic interpretability is quite unlikely[1] to be able to confidently assess[2] whether AIs[3] are deceptively aligned (or otherwise have dangerous propensities) in the next 10 years.
greater than 90% failure
likelihood ratio of 10
I’m refering to which ever AIs are pivotal or cruxy for things to go well prior to human obsolescence.
It is very unlikely AI causes an existential catastrophe (Bostrom or Ord definition) but doesn’t result in human extinction. (That is, non-extinction AI x-risk scenarios are unlikely)
Things will basically be fine regarding job loss and unemployment due to AI in the next several years and those worries are overstated
The current AI x-risk grantmaking ecosystem is bad and could be improved substantially.
People aged 12 to 18 should basically be treated like adults rather than basically treated like children.
It is critically important for US/EU companies to build AGI before Chinese companies.
EAs and rationalists should strongly consider having lots more children than they currently are
Meaningness’s “Geeks Mops and Sociopaths” model is an accurate model of the dynamics that underlie most social movements
Irrefutable evidence of extraterrestrial life would be a good thing.
Someone in the AI safety community (e.g. Yud, Critch, Salamon, you) can currently, within 6 month’s effort, write a 20,000 word document that would pass a threshold for a coordination takeoff on Earth, given that 1 million smart Americans and Europeans would read all of it and intended to try out much of the advice (i.e. the doc succeeds given 1m serious reads, it doesn’t need to cause 1m serious reads). Copy-pasting already-written documents/posts would count.
It was a mistake to increase salaries in the broader EA/Rationality/AI-Alignment ecosystem between 2019 and 2022
Good AGI-notkilleveryoneism-conscious researchers should in general prioritize working at big AGI labs over working independently, for alignment-focused labs, or for academia marginally more than they currently do.
The ratio of good alignment work done at labs vs independently mostly skews toward labs
Good meaning something different from impactful here. Obviously AGI labs will pay more attention to their researchers or researchers from respectable institutions than independent researchers. Your answer should factor out such considerations.
Edit: Also normalize for quantity of researchers.
There is a greater than 20% chance that the Effective Altruism movement has been net negative for the world.
Empirical agent foundations is currently a good idea for a research direction.
A basic deontological and straightforward morality (such as that exmplified by Hermione in HPMOR) is basically right; this is in contrast with counterintuitive moralities that suggest evil-tinted people (like Quirrell in HPMOR) are also valid ways of being moral.
Just like the last 12 months was the time of the chatbots, the next 12 months will be the time of agent-like AI product releases.
The work of agency-adjacent research communities such as artificial life, complexity science and active inference is at least as relevant to AI alignment as LessWrong-style agent foundations research is.
American intelligence agencies consider AI safety to be substantially more worth watching than most social movements
It is possible to make meaningful progress on deceptive alignment using experiments on current models
Having another $1 billion to prevent AGI x-risk would be useful because we could spend it on large compute budgets for safety research teams.
Moloch is winning.
“Polyamory-as-a-default-option” would be a better social standard than “Monogamy-as-a-default-option”.
The rationality community will noticeably spill over into other parts of society in the next ten years. Examples: entertainment, politics, media, art, sports, education etc.
At least one American intelligence agency is concerned about the AI safety movement potentially decelerating the American AI industry, against the administration/natsec community’s wishes
I broadly agree with the claim that “most people don’t do anything and the world is very boring”.
On the current margin most people would be better off involving more text-based communication in their lives than in-person communication.
Agent foundations research should become more academic on the margin (for example by increasing the paper to blogpost ratio, and by putting more effort into relating new work to existing literature).
Current progress in AI governance will translate with greater than 20% probability into more than a 2 year counterfactual delay of dangerous AI systems
It is possible to make meaningful progress on ELK using empirical experiments on current models
Language model agents are likely (>20%) to produce AGI (including the generalization to foundation model-based cognitive architectures)
Current AI safety university groups are overall a good idea and helpful, in expectation, for reducing AI existential risk
Having another $1 billion to prevent AGI x-risk would be useful because we could spend it on large-scale lobbying efforts in DC.
Immersion into phenomena is better for understanding them than trying to think through at the gears-level, on the margin for most people who read LessWrong.
Public mechanistic interpretability research is net positive in expectation.
Among existing alignment research agendas/projects, Superalignment has the highest expected value
Most LWers should rely less on norms of their own (or the LW community’s) design, and instead defer to regular societal norms more.
Rationalists would be better off if they were more spiritual/religious
“Agent” is an incoherent concept.
“Open-source LLM-based agent with hacking abilities starts spreading itself over the Internet because some user asked it to do so or to do something like to conquer the world” is a quite probable point-of-no-return regarding AGI risk.
Investing in early-stage AGI companies helps with reducing x-risk (via mission hedging, having board seats, shareholder activism)
Great art is rarely original and mostly copied.
Rationality should be practiced for Rationality’s sake (rather than for the sake of x-risk).
The younger generation of rationalists are less interesting than the older generation was when that old generation had the same experience as the young generation currently does.
At least one of {Anthropic, OpenAI, Deepmind} is net-positive compared to the counterfactual where just before founding the company, its founders were all discretely paid $10B by a time-travelling PauseAI activist not to found the company and to exit the industry for 30 years, and this worked.
One should basically not invest into having “charisma”.
The most valuable new people joining AI safety will usually take ~1-3 years of effort to begin to be adequately sorted and acknowledged for their worth, unless they are unusually good at self-promotion e.g. gift of gab, networking experience, and stellar resume.
Poll feature on LW: Yay or Nay?
There is a greater than 80% chance that effective altruism has been net-negative for the world.
If rationality took off in China, it would yield higher EV from potentially spreading to the rest of the world than from potentially accelerating China.
Effective altruism can be well modeled by cynically thinking of it as just another social movement, in the sense that those a part of it are mainly jockeying for in-group status, and making costly demonstrations to their in-group & friends that they care about other sentiences more than others in the in-group. Its just that EA has more cerebral standards than others.
Language model agents are very likely (>80%) to produce AGI (including the generalization to foundation model-based cognitive architectures)
Research into getting a mechanistic understanding of the brain for purposes of at least one of: understanding how values/empathy works in people, brain uploading or improving cryonics/plastination is net positive and currently greatly underfunded.
Most persistent disagreements can more usefully be thought of as a difference in priors rather than a difference in evidence or rationality.
A Secular Solstice variation designed to work weekly (akin to Sunday Service or Shabbat) would be positive for rationalists, both for community and for the thought processes of the members.
There is a greater than 50% chance that the Effective Altruism movement has been net negative for the world.
Most LWers are prioritizing their slack too much.
“Intelligence” can be characterized with a similar level of theoretical precision as e.g., heat, motion, and information. (In other words: it’s less like a messy, ad-hoc phenomena and more like a deep, general fact about our world).
You know of a technology that has at least a 10% chance of having a very big novel impact on the world (think the internet or ending malaria) that isn’t included in this list, very similar, or downstream from some element of it: AI, mind uploads, cryonics, human space travel, geo-engineering, gene drives, human intelligence augmentation, anti-aging, cancer cures, regenerative medicine, human genetic engineering, artificial pandemics, nuclear weapons, proper nanotech, very good lie detectors, prediction markets, other mind-altering drugs, cryptocurrency, better batteries, BCIs, nuclear fusion, better nuclear fission, better robots, AR, VR, room-temperature superconductors, quantum computers, polynomial time SAT solvers, cultured meat, solutions to antibiotic resistance, vaccines to some disease, optical computers, artificial wombs, de-extinction and graphene.
Bad options included just in case someone thinks they are good.
Xi Jinping thinks that economic failure in the US or China, e.g. similar to 2008, is one of the most likely things to change the global balance of power.
Having another $1 billion to prevent AGI x-risk would be pretty useful.
If we had access to a brain upload (and maybe a world simulator too) we could in principle extract something like a utility function, and the theory behind it relates more to agents in general than it does to humans in particular.
Any activity or action taken after drinking coffee in the morning will strongly reward/reinforce that action/activity
Humans are the dominant species on earth primarily because our individual intelligence surpassed the necessary threshold to sustain civilization and take control of our environment.
American intelligence agencies are actively planning to defend the American AI industry against foreign threats (e.g. Russia, China).
When people try to discuss philosophy, math, or science, especially pre-paradigmatic fields such as ai safety, they use a lot of metaphorical thinking to extend from familiar concepts to new concepts. It would be very helpful and people would stop talking past each other so much if they practiced being explicitly aware of these mental representations and directly shared them rather than pretending that something more rigorous is happening. This is part of Alfred Korzybski’s original rationality project, something he called ‘consciousness of abstraction.’
If you can write prompt for GPT-2000 such that completion of this prompt results in aligned pivotal act, you can just use knowledge necessary for writing this prompt to Just Build aligned ASI, without necessity to use GPT-2000.
Rationalist rituals like Petrov day or the Secular Solstices should be marginally more emphasized within those collections of people who call themselves rationalists.
Rationality is likely to organically gain popularity in China (e.g. quickly reaching 10,000 people or reaching 100,000 by 2030, e.g. among scientists or engineers, etc).
It is wrong to protest AI labs.
The ratio of good alignment work done at labs vs in academia mostly skews toward labs
Good meaning something different from impactful here. Obviously AGI labs will pay more attention to their researchers or researchers from respectable institutions than academics. Your answer should factor out such considerations.
Edit: Also normalize for quantity of researchers.
The ratio of good alignment work done in academia vs independently mostly skews toward academia
Good meaning something different from impactful here. Possibly AGI labs will pay more attention to academics than independent researchers. Your answer should factor out such considerations.
Edit: Also normalize for quantity of researchers.
At least one mole, informant, or spy has been sent by a US government agency or natsec firm to infiltrate the AI safety community by posing as a new member (even if it’s just to ask questions in causal conversations at events about recent happenings or influential people’s priorities).
The government should build nuclear-driven helicopters, like nuclear subs.
An alignment technique that can fully align GPT-4 is likely (>50%) to also fully align the first existentially dangerous AGI
Most end-to-end “alignment plans” are bad because research will be incremental. For example, Superalignment’s impact will mostly come from adapting to the next ~3 years of AI discoveries and working on relevant subproblems like interp, rather than creating a superhuman alignment researcher.
Conceptual alignment work on concepts like “agency”, “optimization”, “terminal values”, “abstractions”, “boundaries” is mostly intractable at the moment.
Most LWers are not prioritizing their slack enough.
Those who call themselves rationalists or EAs should drink marginally more alcohol at social events.
In human interactions at any scale, it is net good to at least momentarily consider the Elephant and/or the Player, with very few exceptions.
Most of the time, power-seeking behavior in humans is morally good or morally neutral.
Computer science & ML will become lower in relevance/restricted in scope for the purposes of working with silicon-based minds, just as human-neurosurgery specifics are largely but not entirely irrelevant for most civilization-scale questions like economic policy, international relations, foundational research, etc.
Or IOW: Model neuroscience (and to some extent, model psychology) requires more in-depth CS/ML expertise than will the smorgasbord of incoming subfields of model sociology, model macroeconomics, model corporate law, etc.
The virtue of the void is indeed the virtue above all others (in rationality), and fundamentally unformalizable.
There is likely a deep compositional structure to be found for alignment, possibly to the extent that AGI alignment could come from “merely” stacking together “microalignment”, even if in non-trivial ways.
If you can’t write a program that produces aligned (under whatever definition of alignment you use) output being run on unphysically large computer, you can’t deduce from training data or weights of superintelligent neural network if it produces aligned output.
There are arguments for convergent instrumental pressures towards catastrophe, but the required assumptions are too strong for the arguments to clearly go through.
Cultural values are something like preferences over pairs of social environments and things we actually care about. So it makes sense to talk about jointly optimizing them.
It’s good for EA orgs to pay well
The younger generation of EAs are less interesting than the older generation was when that old generation had the same experience as the young generation currently does.
In the context of Offense-defense balance, offense has a strong advantage
Generative AI like LLMs or diffusion will eventually be superseded by human AI researchers coming up with something autonomous.
EA has gotten a little more sympathetic to vibes-based reasoning recently, and will continue to incorporate more of it.
The mind (ie. your mind), and how it is experienced from the inside, is potentially a very rich source of insights for keeping AI minds aligned on the inside.
All else equal, a unit of animal suffering should be accorded the same moral weight as an equivalent unit of human suffering. (i.e. equal consideration for equal interests)
‘Descriptive’ agent foundations research is currently more important to work on than ‘normative’ agent foundations research.
Autism is the extreme male version of a male-female difference in systemic vs empathic thinking.
Developing a solid human intelligence/skill evaluation metric would be a high-EV project for AI safety, e.g. to make it easier to invest in moving valuable AI safety people to the Bay Area/London from other parts of the US/UK.
MadHatter is an original and funny satirist. The universally serious reaction to his jokeposts is a quintessential example of rationalist humorlessness.
People should pay an attractiveness tax to the government.
I really like the idea of this page and gave this post a strong-upvote. Felt like this was worth mentioning, since in recent years I’ve felt increasingly alienated by LessWrong culture. My only major request here is that, if there are future iterations of this page, I’d like poll options to be solicited/submitted before any voting happens (this is so that early submissions don’t get an unfair advantage just by having more eyeballs on them). A second more minor request is to hide the votes while I’m still voting (I’m trying very hard not to be influenced by vote counts and the names of specific people agreeing/disagreeing with things, but it’s difficult).
Is it intended that collapsed polls display the author and expanded ones don’t?
No. This is a one-off fast mockup of a poll just to see what it’s like, and that’s one of the problems that I didn’t think I needed to fix before testing the idea (along with things like agree/disagree not automatically showing up).
Overall I’d say it was like an 75th-80th percentile outcome, I’m pleasantly surprised by how much activity it’s had and how much interesting disagreement it has surfaced. So I’ve updated toward building this as a proper feature.
I feel like LessWrong should just update-all-the-way and ask Manifold Markets for a stylable embed system.
I work at Manifold, I don’t know if this is true but I can easily generate some arguments against:
Manifold’s business model is shaky and Manifold may well not exist in 3 years.
Manifold’s codebase is also shaky and would not survive Manifold-the-company dying right now.
Manifold is quite short on engineering labor.
It seems to me that Manifold and LW have quite different values (Manifold has a typical startup focus on prioritizing growth at all costs) and so I expect many subtle misalignments in a substantial integration.
Personally for these reasons I am more eager to see features developed in the LW codebase than the Manifold codebase.
Can you elaborate on this point? Why wouldn’t the codebase be salvageable?
I don’t have any special knowledge, but my guess is their code is like a spaghetti tower (https://www.lesswrong.com/posts/NQgWL7tvAPgN2LTLn/spaghetti-towers#:~:text=The distinction about spaghetti towers,tower is more like this.) because they’ve prioritized pushing out new features over refactoring and making a solid code base.
I would love to try having dialogues with people about Agent Foundations! I’m on the vaguely-pro side, and want to have a better understanding of people on the vaguely-con side; either people who think it’s not useful, or people who are confused about what it is and why we’re doing it, etc.
I’d love to do this. I think clarifying the types of work that would fall under agent foundations would be very useful. On one hand, it seems as though most of the useful work has probably been done; on the other, some current discussion seems to be missing important points from that work.
I might want to do this. I’m on the vaguely pro side but think we currently don’t have many tractable directions.
Would love if expandable comment sections could be put on poll options. Most won’t benefit, but some might benefit greatly, epsecially if it incentivized low word count.
For example: For EA being >50% of being net negative, I would like to make a short comment like:
I now think that it should go back to the binary yes and no responses, adding bells and whistles will complicate things too much.
At least ternary: an “unsure” option is definitely worth including. (That also seems to be the third most popular option in the questions above.)
I think a fourth option, for “the question is wrong” would also be a good one, but perhaps redundant if there is also a comment section.
Many people disagreed with that. So, apparently many people believe that inescapable dystopias are not-unlikely? (If you’re one of the people who disagreed with the quote, I’m curious to hear your thoughts on this.)
Farmed animals are currently inside a non-extinction X-risk.
Steelmanning a position I don’t quite hold: non-extinction AI x-risk scenarios aren’t limited to inescapable dystopias as we imagine them.
“Kill all humans” is certainly an instrumental subgoal of “take control of the future lightcone” and it certainly gains an extra epsilon of resources compared to any form of not literally killing all humans, but it’s not literally required, and there are all sorts of weird things the AGI could prefer to do with humanity instead depending on what kind of godshatter it winds up with, most of which are so far outside the realm of human reckoning that I’m not sure it’s reasonable to call them dystopian. (Far outside Weirdtopia, for that matter.)
It still seems very likely to me that a non-aligned superhuman AGI would kill humanity in the process of taking control of the future lightcone, but I’m not as sure of that as I’m sure that it would take control.
That makes sense; but:
setting aside the question of what to call such scenarios, with what probability do you think the humans[1] in those scenarios would (strongly) prefer to not exist?
or non-human minds, other than the machines/Minds that are in control
I expect AGI to emerge as part of the frontier model training run (and thus get a godshatter of human values), rather than only emerging after fine-tuning by a troll (and get a godshatter of reversed values), so I think “humans modified to be happy with something much cheaper than our CEV” is a more likely endstate than “humans suffering” (though, again, both much less likely than “humans dead”).
One of the main questions on which I’d like to understand others’ views is something like: Conditional on sentient/conscious humans[1] continuing to exist in an x-risk scenario[2], with what probability do you think they will be in an inescapable dystopia[3]?
(My own current guess is that dystopia is very likely.)
or non-human minds, other than the machines/Minds that are in control
as defined by Bostrom, i.e. “the permanent and drastic destruction of [humanity’s] potential for desirable future development”
Versus e.g. just limited to a small disempowered population, but living in pleasant conditions? Or a large population living in unpleasant conditions, but where everyone at least has the option of suicide?
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?
I think the intelligence community ought to be watching AI risk, whether or not they actually are.
e.g. in the UK, the Socialist Workers Party was heavily infiltrated by undercover agents; widwly suspected at the time, subsequently officially confirmed. Now, you way well disagree with their politics, but it’s pretty clear they didn’t amount to a threat. Infiltration should probably have bailed out after quickly Establishing threat level was low.
AI risk, on the other hand … from the point of view of an intelligence agency, there’s uncertainty about whether there’s a real risk or not. Seems worthwhile getting some undercover agents in place to find out what’s going on.
… though, if in 20 years time it has become clear the AI apocalypse has been averted, and the three letter agencies are still running dozens of agents inside 5he AI companies, we could reasonably say their agents have outlived their usefulness.
I feel like this point is a bit confused.
A person believing this essentially has to have a kind of “Wherever I am is where the party’s at” mindset, in which case he ought to have an instrumental view of academia. Like obviously, if I want to maximize the time I spend reading math books and solving math problems, doing it inside of academia would involve wasting time and is suboptimal. However, if my goal is to do particle accelerator experiments, the easiest way to do this may be to convince people who have one to let me use it, which may mean getting a PhD. Since getting a PhD will still involve spending a bunch of time studying (if slightly suboptimally as compared to doing it outside of academia) then this might be the way to go.
See, we still think that academia is fucked, we just think they have all the particle accelerators. We only have to think academia is not so fucked that it’s a thoroughly unreasonable source of particle accelerator access. We can still publish all our research open access, or even just on our blog (or also on our blog).
I would love to see some sort of integration of the pol.is system, or similar features
Rationalists would be better off if they tore up the old stuff, and started again from beginning.