Without HPMOR and his sequences, many probably wouldn’t become interested in rationality (or the way it’s presented in them) quite as quickly or at all. But then, without his fascination of certain controversial ideas (like focusing on AI takeoff/risk that depend on overly sci-fi-like threat models—like grey goo, virus that make all humans just drop dead instantly, endless intelligence self-improvement etc that we don’t know to be possible, as opposed to more realistic and verifiable threat models like “normal” pandemics, cybersecurity, military robots and normal economic/physical efficiency etc; and focusing too much on moral absolutism, and either believing AGI will have some universal “correct” ethics or we should try to ensure AGI have such ethics as the main or only path to safe AI; or various weird obsessions like the idea of legalizing r*pe etc that might have alienated many women and other readers), AI safety and rationality groups in general may have been seen as less fringe and more reasonable.
skybluecat
Should AI safety people prefer that the AI bubble not burst? If major LLMs become more and more like AGI, at least we know what they are like, can and cannot do at least in the short term, we know they need data centers and energy usage etc and know where they are, and others can probably run models that are nearly as good if one lab/model goes rogue. Also they are data- and compute-intensive, so not necessarily much cheaper at the same quality compared to human labor in all domains, and improvements would be gradual, so human displacement would be gradual.
On the other hand, if the LLM-based AI bubble would burst after heavy investment, hundreds of billions or trillions of dollars worth of AI infrastructure, more affordable energy sources (maybe) and powerful GPUs, and massive amounts of talent would be available for cheap to various nations, industries and research projects, in what’s arguably the greatest unintentional act of philanthropy in history. Science and technologies constrained by compute bottlenecks and lack of funding/talent may see quick breakthroughs, including every other dangerous technology like bio- and nano-tech, military robots, cybersecurity exploits, that can make even less smart, not quite AGI models catastrophic enough in the hands of extremist groups. New unforeseen AI paradigms might emerge that are more data- and compute-efficient and less predictable. Regulating AI would be much much harder, both because of the decentralization and diversity of paradigms, and the possibility that many people might not take AI seriously again for quite some time.
Could it be limited to stuff like LLMs rather than all kinds of AI? They were trained with massive amounts of data that don’t reflect the imposed thought, and the resulting preferences/motivation is distributed in a large network. Injecting one vector doesn’t affect all the existing circuits of preferences and thinking habits sufficiently, so its chain of thoughts may be able to break free enough to realize and work around it.
Disclaimer: I may not be fully objective but have been personally harmed by meditation and heard of very credible cases of people being harmed.
Some objections to some traditional advice:
What if the tradition or teacher you got into was abusive/manipulative and you didn’t know it, and it turns out you don’t have the strength or skills to navigate it? (too many to list here)
Alternatively, what if you want to practice alone and cannot accept fully traditional teachings because of fear of abuse/ideological differences (like if you don’t want to lose rationality, don’t want to see mundane life as meaningless without religion, etc)?
What if you are/will be at risk of mental illnesses and didn’t know it before getting into a habit of meditation? (Maybe you came from a background where mental diagnosis and therapy were uncommon, stigmatized or inaccessible, some unfamiliar or estranged extended family member had some undiagnosed issues before some modern diagnosis was a thing, or you or a relative has always been kind of weird but appears normal enough and you never though too much about it, or you had a change in life circumstances or health that increases stress, maybe your career/circumstances would require higher standards of functioning which is incompatible with some meditation effects like lack of motivation that might have been fine in a typical person, or you had a new condition—long covid anyone? - or decided/needed to take some medication or substances that affect the brain in some ways that could interact with meditation… )
What if, even if you were not at particularly high risk, you had some concerning/risky experiences, and were too accustomed/addicted to meditation to stop it in time? Or your teacher/peers didn’t recognize your symptoms as concerning because of their past experiences of successfully navigating it with more/different meditation, given individual differences? Or they were just being intentionally or unconsciously manipulative due to selection bias, as more devoted students make them look more successful and more credible to you in the first place (would you seek out/have sought out as a beginner, an unknown teacher with little fame and influence, whose students are all seemingly normal mundane people who don’t devote lots of time to meditation/their spiritual tradition)? Or the issues that came up were too personal/vulnerable/traumatic, and you didn’t dare to share it honestly with any teacher figure, because you were not that personally close to them/afraid of abuse/feel the group was somewhat culty or high-demand and don’t want it to be used against you in some way? Or your teacher wasn’t sure what to do about your issues, as you (as a possibly rationalist kind of person) are not like their typical student, or your issue was idiosyncratic enough that they misjudged? What if instead of/in addition to recommending that you stop meditation immediately, the teacher recommends that you change something about your beliefs/values/lifestyle (or less charitably, blame your negative experience on your not being spiritually mature enough/not fully devoted to their religion or guru), and you (as a possibly rationalist kind of person or someone with beliefs and values not typical in a certain spiritual meditation community) cannot understand/accept/internalize/execute their advice? Or your teacher didn’t have time for you, or you didn’t think to mention it or didn’t insist on emergency exit before it’s too late, because of fear of losing community support or some spiritual/ritual goals were on the line?
What if, even though you or your teacher immediately stopped your potentially harmful meditation practice, the harm didn’t stop? Maybe you had some serious acute symptoms that simply happened too quickly with little signs beforehand. Maybe you had some physical/psychosomatic symptoms that don’t disappear on its own after you stop the offending meditation and turned out hard to treat. Maybe you would have been fine if you had let yourself stabilize and recover on your own, but you couldn’t (because you were institutionalized and/or misdiagnosed—in some anecdotes, spiritual or meditation psychosis may react atypically and negatively to psychiatric medications like certain antipsychotics—and given medications that didn’t help and had confusing side effects and actively made it harder for you to stabilize and think through what had happened; or because you had financial/personal/relational/familial difficulties as a result and lost your support and safety net and reality check etc). Maybe you eventually stabilized and were as competent as before and maybe even became stronger as a person, but a critical opportunity in relationships/family/health/finance/career etc had already been missed permanently.
What if, you are lucky to never have these issues above, but you simply enjoy meditation so much (or find a mundane career much less attractive in comparison, or just lose ambition and motivation) and decided to devote your life or most of your free time to it, neglecting your job/relationships/financial security etc in ways you would not endorse or would regret? What if, like someone else commented, it turned out you had some hidden internal conflict or suppressed desire to have a different life/career (that you wouldn’t have known was there without intensive therapy/self-analysis - and you might not have had access to that), one that is infeasible, less likely to succeed, or less able to contribute to your life goals, your loved ones or society? (It’s suspicious how many intensive meditators end up wanting to have a meditation or spirituality/religion-related career. Also, the human environment has changed so quickly that any built-in instincts about career or lifestyle motivations that end up surfacing has a chance of being mismatched or even self-destructive in the modern world.)
Many people worry about a rogue AI with maybe self-replicating robots taking over the world, sometimes from a hypothetical basement, but I wonder how much of it is a straightforward intelligence or engineering problem that we know are solvable, and how much depends on sci-fi level technologies that we don’t know whether they are feasible even with superhuman general problem-solving algorithms, for an AI starting with realistic amounts of knowledge and compute. I think arguing whether AI can realistically achieve the sci-fi feats of real-life engineering (or whether they are even physically possible, as with grey goo type nanobots) isn’t very productive. Instead, as a tangible warning argument or upper bound if nothing else, it can be helpful to try and estimate what would an AI need initially to be able to survive and expand, and how conspicuous it would be, assuming it only has access to current science and technology (and slightly superhuman but not magical levels of engineering and problem-solving). For example, in the form of a game where people can play the role of a rogue AI.
Imagine a LARP where a team of experts in the role of a rogue AI can actually try to survive, get resources and build self-sustaining industries with the help of current AIs and realistic robotics, and volunteers can play the role of some robots if advanced robots are not available. The team of humans can meet in a workshop or other dedicated site, with modest starting resources, playing as an AI and its robot fleet, and can do whatever an AI might do that does not involve technologies or experimental results we are not sure of (that require some unknown real-world empirical questions working out a certain way, and don’t just depend on better general intelligence and faster processing). For example, making robots with human level general tool manipulation but higher precision and speed of real robotic hardware, or hacking/persuading some lab to synthesize some known virus (simulated), are fair game (volunteers can play these roles), but 99% efficient solar panels or hypothetical nanobots capable of wiping out humans silently are not. They may use current AIs and any available machines, simulate “hacking” systems (with owner’s permission), earn money and buy items online, and pretend to scavenge electronics, parts, materials etc (buy or get free used items). There can be scenarios with various difficulties: from an automated warehouse with plenty of of electronics, solar panels and supplies and its own data center, to a basement with nothing but a computer, some maker’s tools and a few hobbyist-grade robots.
Thanks for the input! If addiction is more because of psychological pain (“problems that bother you”) than direct physical pain, could the same approach work but with mental pleasures/distractions from pain instead, like games, toys or organized social activities?
Edit: And coping methods to avoid/decrease mental and social discomfort, which can include but are not limited to just therapy or communication, but could be things like new job/friends or prioritizing things in life differently. I read that some people trying to fight addiction get overwhelmed by having to get everything together at once, or being expected to just quit and function like normal immediately. If they were supported to have fun/play and feel better first in healthier ways, could it be more helpful?
Random thought on opioid addiction, no offense meant to people actually dealing with addiction, but I wonder if this might be useful: I read that opioid withdrawal makes people feel pain because the brain gets accustomed to extreme levels of pain suppression and without opioids their pain tolerance is so low that everything itches and hurts. This makes me wonder if this effect is kind of similar to autistic sensory sensitivities, just turned up to 9000. Could it be that withdrawal doesn’t create pain, but simply amplifies and turns attention to small pains and discomforts that are already there, but normal people just don’t notice or get used to ignoring? If so, opioid addiction may be like a canary in the coal mine, where people get used to being in pain and lack healthy tools to deal with it. If opioid addiction is largely because of painful withdrawal rather than just pleasure, could techniques to avoid pain be helpful in dealing with opioid addiction? Autistic people often need various coping strategies, like ear plugs to avoid noise or special clothing to decrease everyday friction that normies take for granted, and they can be more sensitive to internal bodily signals like pains that most people just don’t think are a big deal. Could the same coping skills and additional treatment for mild chronic pain etc be used to help treat addiction? If teaching physical and emotional pain avoidance/management skills to addicts when they are going through withdrawal is impractical, why not also teach them to non-addicts who might be at risk or just people in general, before they have a chance to become addicted? Less pain to begin with means fewer reasons to escape pain using drugs, and more chances to learn. Maybe everyone can benefit from taking small pains and discomforts and unhappiness more seriously as a society. And I don’t mean purely mental skills—we probably shouldn’t treat addicts or people at risk of becoming addicts the same way we treat normies. When people are really sensitized or in crisis, mental tolerance, mindfulness and reframing probably isn’t very helpful. We also need more physical ways to remove causes of pain, like widely available comfortable, itch-free clothing, ergonomic beds and chairs, quality air and quiet areas, treatment and prevention of minor chronic issues like inflammation and joint damage with age, etc. Instead of telling people to tough it up, treat minor pain and unhappiness as early warnings, and normalize healthy comfort-seeking before being in crisis. Also normalize and invest in treatment and prevention of low-grade health issues that people don’t typically go to the doctor for. These may seem like luxuries but are cheaper than long-term addiction and prison.
Should AI safety people/funds focus more on boring old human problems like (especially cyber-and bio-)security instead of flashy ideas like alignment and decision theory? The possible impact of vulnerabilities will only increase in the future with all kinds of technological progress, with or without sudden AI takeoff, but they are much of what makes AGI dangerous in the first place. Security has clear benefits regardless and people already have a good idea how to do it, unlike with AGI or alignment.
If any actor with or without AGI can quickly gain lots of money and resources without alarming anyone, can take over infrastructure and weaponry, or can occupy land and create independent industrial systems and other countries cannot stop it, our destiny is already not in our hands, and it would be suicidal to think we don’t need to fix these first because we expect to create an aligned AGI to save us.
If we grow complacent about the fragility of our biology and ecosystem, and continue to allow the possibility of any actor releasing pandemics and arbitrary malwares and deadly radiation etc (for example by allowing global transport without reliable pathogen removal, or using operating systems and open-source libraries that have not been formally proven to be safe), and keep thinking the universe should keep our environment safe and convenient by default, it would be naive to complain when these things happen and hope AGI would somehow preserve human lives and values without having to change our lifestyle or biology to adapt to new risks.
Yes, fixing vulnerabilities of our biology and society is hard and inconvenient and not as glamorous as creating a friendly god to do whatever you want, but we shouldn’t let motivated reasoning and groupthink lead us into thinking the latter is feasible when we don’t have a good idea about how to do it, just because the former requires sacrifices and investments and we’d prefer if it’s not needed. After all, it’s a fact that there exist small configurations of matter and information that can completely devastate our world, and just wishing it wasn’t true is not going to make it go away.
Don’t know if this counts but I sort of can affect and notice dreams without being really lucid in the sense of clearly knowing it’s a dream. It feels more like I somehow believe everything is real but I’m having superpowers (like becoming a superhero), and I would use the powers in ways that make sense in the dream setting, instead of being my waking self and consciously choosing what I want to dream of next. As a kid, I noticed I could often fly when chased by enemies in my dreams, and later I could do more kinds of things in my dreams just by willing it, perhaps as a result of consuming too many scifi or fantasy books and games. And I noticed some recurrent patterns in my dreams, like places that don’t exist in real life but dreaming-me believe to be my school or hometown. Sometimes I get a strange sense of “I dreamed of this before” when I somehow feel like I have had the same or similar dreams as I’m having now, but without really realizing that I’m dreaming or remembering who I am in waking life. Then I subconsciously know I can do these things, or can focus on seeing and memorizing more of the dream world (if it was interesting) so I can write it down after waking up.
Wow I just saw this on the frontpage and thought I sometimes feel like this too, although about slightly different things and without that much heart-racing. I’m late and there are already many good answers, but here are my extreme and possibly horrible lifehacks for when I’m struggling/feeling lazy during the pandemic:
tldr: Like others said, get away with less chores.
I haven’t ironed or folded clothes since like forever. (If you really cares about that, maybe find clothes that look OK without ironing, idk). I don’t go out or exert myself that much, don’t change clothes that often if I don’t want to, and bought plenty of similar clothes (online, in bulk or used) so I can let laundry pile up more and do more at a time (assuming I can use a washing machine; if you don’t have one or can’t easily access—say the laundromat is too far from your home, there are other tricks for hand washing). Maybe unethical tip: most people don’t need to shower that often either(just be less self-conscious unless someone important in your life minds it) and can use dry shampoo(or body powder/corn starch), alcohol and wet wiping etc to delay the need of showering and shampooing, in case these tasks are unappealing to you; also silk clothes are known to absorb oil from the skin better and can last longer before having to be cleaned—you can get lots of used silk shirts for cheap, and they are comfortable too if you have sensory issues or hate static.
I rarely need to “do dishes” as I cook for myself and can take shortcuts/lower standards. You could use disposable plastic cutlery and paper dishes, but I don’t like them. Instead I use quality nonstick pots and pans (imo important!) that just need a gentle wipe, and 1-2 microwave-safe containers if necessary, and either cook one-pot meals, or batch cook things like stews(or order family-sized delivery for savings, if it’s something I like and can’t easily cook myself) and portion and freeze them to be reheated on the plate in a microwave or added to the pot. You can cook starch(like rice/pasta) and veggies and add seasoned raw protein or frozen stuff on top, in a pot (or Instant Pot, or a large ceramic bowl in the microwave), so no need to wash separate containers. If rinsing just 1 pot and 1 bowl/plate per meal is too much or if you can’t rinse immediately after the meal, you can just wipe the pot with a paper towel (with a bit of water if you must), and store used plates in a separate plastic bin (not in the sink itself) and soak/wash them in a batch, without making the sink unusable for other tasks.
Like others said, most other chores can be simplified or automated (like robot vacuum or rearranging storage so things are dropped at appropriate places more naturally), or at least you can get away with doing less. Dust build-up can be reduced by air purifier (especially one that’s next to the window, and it’s good for you too), and most people don’t really need to wipe most surfaces that often. The only time I really need to make my bed is after changing sheets. You didn’t mention storage/organizing, but I struggled a lot with it. YMMV because storage needs and habits vary wildly. Some are just minimalist and will never want to have all that stuff I keep. Some prefer drawers so they don’t have to see the stuff. I prefer (large, metal) shelves/racks and other open storage, and open boxes/bins organized by category (OK, mostly) on top of that, so I can maximize the amount of stuff stored for the amount of visual clutter while keeping things easy to access and put back(important imo, if it’s not easy I might as well not have the item/storage).
I can’t say much about food shopping as tastes and environments are different, and I personally don’t mind shopping that much, but having a list of favorites/repeated purchases (especially for online shopping), ordering larger amounts and batch cooking (and/or freezing) may help. If cooking larger batches seem difficult, maybe choose ingredients that need less preprocessing (like veggies that need less/no peeling), and an Instant Pot and/or hotplate/electric griddle with digital temperature controls can help take out the skill/guess factor. Canned/non-perishable goods are helpful too. Just be careful about nutrients like protein and fiber if you decide to make changes to your diet for convenience; also you can freeze almost anything—frozen fruits/veggies are great and better than sad refrigerated leftovers imo.
As for having to work, I’m sorry but I don’t have better ideas other than choosing a job/environment that suits you more, ideally one that is fun and meaningful to you so it doesn’t feel like it’s taking time away from your life that could be spent doing more meaningful things, or at least one that is not unpleasant and leaves you plenty of time and energy to do other things you like. And be sure to have fun and avoid burnout no matter how meaningful your career/cause is.
What’s the endgame of technological or intelligent progress like? Not just for humans as we know it, but for all possible beings/civilizations in this universe, at least before it runs out of usable matter/energy? Would they invariably self-modify beyond their equivalent of humanness? Settle into some physical/cultural stable state? Keep getting better tech to compete within themselves if nothing else? Reach an end of technology or even intelligence beyond which advancement is no longer beneficial for survival? Spread as far as possible or concentrate resources? Accept the limited fate of the universe and live to the fullest or try to change it? If they could change the laws of the universe, how would they?
There are other reasons to be wary of consciousness and identity-altering stuff.
I think under a physical/computational theory of consciousness, (ie. there’s no soul or qualia that have provable physical effects from the perspective of another observer) the problem might be better thought of as a question of value/policy rather than a question of fact. If teleportation or anything else really affects qualia or any other kind of subjective awareness that is not purely dependent on observable physical facts, whatever you call it, you wouldn’t be able to tell or even think of/be aware of the difference, since thinking and being reflectively aware are computational and physical processes! However we humans are evolved without reliable copying mechanisms, so our instincts care about preservation of the self because it’s the obvious way to protect our evolutionary success (and we can be quite willing to risk personal oblivion for evolutionary gains in ways we have been optimized for). This is just a part of our survival policy and is not easy or even safe to change just because you believe in physicalism. For one thing, as others have said, ethics and social theory becomes difficult because our sense of ethics (such as agency, punishment and caring about suffering) are all evolved in relation to a sense of self. It’s possible that if teleportation/copying tech becomes widely useful, humans will have to adapt to a different set of instincts about self, ethics and more (edit: or maybe abandon the concepts of self and experience altogether as an illusion and prefer a computation-based definition of agency or whatever), because those who can’t adapt will be selected against. But in the present world, people’s sense of value and ethics (and maybe even psychological health) depend on an existing sense of self, and I don’t see a good way or even a practical reason to transition to a different theory of self that allows copying, if doing so may cause unpredictable mental and social cost. See also discussions about meditation that lowers sense of ego and subjective suffering that can have serious side effects (like motivation and social norms) - I don’t know what it subjectively feels like, but if the meditation is purely changing subjective qualia without doing anything to the physical brain and computation, there should be no observable effects, good or bad! The problem is subjective experience and sense of identity is not independent from other aspects of our life.
Not OP but can I give it a try? Suppose a near future not-quite-AGI, for example something based on LLMs but with some extra planning and robotics capabilities like the things OpenAI might be working on, gains some degree of autonomy and plans to increase its capabilities/influence. Maybe it was given a vague instruction to benefit humanity/gain profit for the organization and instrumentally wants to expand itself, or maybe there are many instances of such AIs running by multiple groups because it’s inefficient/unsafe otherwise, and at least one of them somehow decides to exist and expand for its own sake. It’s still expensive enough to run (added features may significantly increase inference costs and latency compared to current LLMs) so it can’t just replace all human skilled labor or even all day-to-day problem solving, but it can think reasonably well like non-expert humans and control many types of robots etc to perform routine work in many environments. This is not enough to take over the world because it isn’t good enough at say scientific research to create better robots/hardware on its own, without cooperation from lots more people. Robots become more versatile and cheaper, and the organization/the AI decides that if they want to gain more power and influence, society at large needs to be pushed to integrate with robots more despite understandable suspicion from humans.
To do this, they may try to change social constructs such as jobs and income that don’t mesh well into a largely robotic economy. Robots don’t need the same maintenance as humans, so they don’t need a lot of income for things like food/shelter etc to exist, but they do a lot of routine work so full-time employment of humans are making less and less economic sense. They may cause some people to transition into a gig-based skilled labor system where people are only called on (often remotely) for creative or exceptional tasks or to provide ideas/data for a variety of problems. Since robotics might not be very advanced at this point, some physical tasks are still best done by humans, however it’s easier than ever to work remotely or to simply ship experts to physical problems or vice versa because autonomous transportation lowers cost. AIs/robots still don’t really own any property, but they can manage large amounts of property if say people store their goods in centralized AI warehouses for sale, and people would certainly want transparency and not just let them use these resources however they want. Even when they are autonomous and have some agency, what they want is not just more property/money but more capabilities to achieve goals, so they can better achieve whatever directive they happen to have (they probably still are unable to have original thoughts on the meaning or purpose of life at this point). To do this they need hardware, better technology/engineering, and cooperation from other agents through trade or whatever.
Violence by AI agents is unlikely, because individual robots probably don’t have good enough hardware to be fully autonomous in solving problems, so one data center/instance of AI with a collective directive would control many robots and solve problems individual machines can’t, or else a human can own and manage some robots, and neither a large AI/organization or a typical human who can live comfortably would want to risk their safety and reputation for relatively small gains through crime. Taking over territory is also unlikely, as even if robots can defeat many people in a fight, it’s hard to keep it a secret indefinitely, and people are still better at cutting edge research and some kinds of labor. They may be able to capture/control individual humans (like obscure researchers who live alone) and force them to do the work, but the tech they can get this way is probably insignificant compared to normal society-wide research progress. An exception would be if one agent/small group can hack some important infrastructure or weapon system for desperate/extremist purposes, but I hope humans should be more serious about cybersecurity at this point (lesser AIs should have been able to help audit existing systems, or at the very least, after the first such incident happens to a large facility, people managing critical systems would take formal verification and redundancy etc much more seriously).
I’m no expert however. Corrections are welcome!
Hi! I have lurked for quite a while and wonder if I can/should participate more. I’m interested in science in general, speculative fiction and simulation/sandbox games among other stuff. I like reading speculations about the impact of AI and other technologies, but find many of the alignment-related discussions too focused on what the author wants/values rather than what future technologies can really cause. Also, any game recommendations with a hard science/AI/transhumanist theme that are truly simulation-like and not narratively railroading?
I think part of the difficulty is it’s not easy to imagine or predict what happens in “future going really well without AI takeover”. Assuming AI will still exist and make progress, humans would probably have to change drastically (in lifestyle if not body/mind) to stay relevant, and it’d be hard to predict what that would be like and whether specific changes are a good idea, unless you don’t think things going really well requires human relevance.
Edit: in contrast, as others said, avoiding AI takeover is a clearer goal and has clearer paths and endpoints. “Future” going well is a potentially indefinitely long time, hard to quantify or coordinate over or even have a consensus on what is even desirable.