This account exists only for archival purposes.
[deactivated]
What do you make of the prospect of neurotech, e.g. Neuralink, Kernel, Openwater, Meta/CTRL-Labs, facilitating some kind of merge of biological human intelligence and artificial intelligence? If AI alignment is solved and AGI is safely deployed, then “friendly” or well-aligned AGI could radically accelerate neurotech. This sounds like it might obviate the sort of obsolescence of human intelligence you seem to be worried about, allowing humans alive in a post-AGI world to become transhuman or post-human cyborg entities that can possibly “compete” with AGI in domains like writing, explanation, friendship, etc.
Thanks. Your post makes point #3 from my post, and it makes two additional points I’ll call #5 and #6:
-
Onboard compute for Teslas, which is a constraint on model size, is tightly limited, whereas LLMs that live in the cloud don’t have to worry nearly as much about the physical space they take up, the cost of the hardware, or their power consumption.
-
Self-driving cars don’t get to learn through trial-and-error and become gradually more reliable, whereas LLMs do.
Re: (5), I wonder why the economics of, say, making a ChatGPT Plus subscription profitable wouldn’t constrain inference compute for GPT-4 just as much as for a Tesla.
Re: (6), Tesla customers acting as safety drivers for the “Full Self-Driving Capability” software seems like it contradicts this point.
Curious to hear your thoughts.
-
Do you have children? I don’t but I am under the impression that people who do say that this changes a bit once you have children. (because of soon to be born grandchildren I guess?)
Nope, I don’t have kids. That might change how I feel about things, by a lot.
Anyway, when I said “future generations”, I wasn’t thinking of kids, grandkids, or great-grandkids, but generations far, far into the future, which would — in an optimistic scenario — comprise 99.9%+ of the total population of humans (or transhumans or post-humans) over time.
I wonder how much the typical person or the typical LessWrong enjoyer would viscerally, limbically care about, say, A) all the people alive today and born within the next 1,000 years vs. B) everyone born (or created) in 3024 A.D. onward.
Thanks for your concern. I don’t want my post to be alarming or extremely dark, but I did want to be totally frank about where my head’s at. Maybe someone will relate and feel seen. Or maybe someone will give me some really good advice.
The other stuff, in reverse order:
Frankly, a lot of the stuff that you describe in this post is irrational. It does not make much sense.
I’m genuinely curious what you mean, and why you think so. I’m open to disagreement and pushback; that’s part of why I published this post.
I’m especially curious about:
Some statements do not pass trivial fact-checking.
By all means, please fact-check away!
People, in general, do not fear death of aging. … Might you have thanatofobia? easy to check—there are lots of tests online.
Haha, I thought I was on LessWrong, where radical life extension is a common wish.
I don’t think I have thanatophobia. The first test that shows up on Google is kind of ridiculous. It almost asks, “Do you have thanatophobia?”
Have you asked the people you love if they would prefer dying of aging, to some sort of AI-induced immortality? It is possible that they would go with immortality, but it’s not obvious.
I could ask. My strong hunch is that, if given the choice between dying of aging or reversing their biological aging by, say, 30 years, they would choose the extra 30 years. And if given the choice again 30 years later, and 30 years after that, they would probably choose the extra 30 years again and again.
But you’re right. I don’t know for sure.
Even within your line of thinking why is this bad? It’s quite possible to live until then...
Yes, you’re right. Even six decades is not impossible for me (knock on wood). However, I also think of my older loved ones.
...or do cryonics?
If I knew cryonics had, say, a 99% chance of working, then I’d take great comfort in that. But, as it is, I’m not sure if assigning it a 1% chance of working is too optimistic. I just don’t know.
One hope I have is that newer techniques like helium persufflation — or whatever becomes the next, new and improved thing after that — will be figured out and adopted by Alcor, et al. by the time cryonics becomes my best option.
Nectome is also interesting, but I don’t know enough about biology to say more than, “Huh, seems interesting.”
That begs the question: do AGIs not require an almost-perfect performance?
Helpful comment that gives lots to think about. Thanks!
I really enjoyed reading this post. Thank you for writing it.
This post was enjoyable as heck to read. Thanks for taking the time to write it.
I guess I’m of two minds about the effective altruism of it all.
One one hand: It kinda just seems like a bunch of self-identified effective altruists, who were well-meaning but perhaps naive, got blinded by money and suckered into servitude by a smart and charismatic leader who was successful at scamming a lot of people. Maybe there isn’t a big lesson about EA philosophy or the EA subculture. Maybe this is just like any other cult leader or con artist or corrupt CEO manipulating a lot of smart, sane, good-hearted people.
On the other hand: Maybe there’s something a bit cult-y about EA subculture and something about EA philosophy’s rejection of common sense and folk morality that made people associated with effective altruism extra susceptible to the Sam Bankman-Fried mind virus. Maybe people in the EA movement need more common sense and more folk morality. Maybe EA people also need more intellectual humility and more healthy skepticism of EA, such that they are more willing to balance EA philosophy with common sense and balance utilitarian ethics with folk morality.I’m empathetic to the people who got taken in by SBF and I don’t judge them harshly. I’ve been scammed before. I’ve been overzealous about non-common sense ideas before. A guy who seems really good at making money trading crypto and wants to donate it all to buy anti-malarial bed nets? On the face of it, what’s wrong with that?
Maybe the more interesting question is: why didn’t the exodus of the initial management team at Alameda result in SBF’s reputation getting destroyed in the EA community? Did the people who left not speak up enough to make that happen? Were they silenced by fear of reprisal? Were they too burnt out and defeated to do much after leaving? Were they embarrassed that Sam manipulated them?
Or did others, especially leaders in the EA community, not listen to them? Did they get blinded by dollar signs in their eyes? Did they find it easier to shoo away inconvenient allegations?
Reaction to Elizabeth’s post seemed what I would have expected in 2021, from both LW and EA Forum. … My attempts to get various ideas across don’t feel like they’re getting different reactions from EAs than I would have expected in 2021.
Would you mind explaining both of these things? I’m not very plugged in to this sort of thing.
Situation with common sense morality and honesty seem not to have changed much from where I sit, and note e.g. that Oliver/Ben who interact with them more seem to basically despair on this front.
Also curious to understand more about what this means.
I think there are underlying generators that give a lot of power and respect to people without vetting them or caring about obvious ways in which they are low-integrity, un-principled, and unethical. … I think most people involved will not have the opportunity to have their low ethical standards be displayed so undeniably...
Would you like to say more about this? I’m curious if there are examples you can talk about publicly.
This can be used by intelligence agencies and governments to completely deny access to specific lines of thought. … Clown attacks include having… degrowthers being the main people talking about the possibility that technological advancement/human capabilities will end it’s net-positive trend.
Does this mean you think intelligence agencies and/or governments are deliberately promoting the degrowth movement in order to discredit the idea of AGI x-risk?
If so, why do you think they are doing that?
And how do you think they are doing that? (For example, is the CIA secretly funneling dark money to organizations that promote degrowth?)
Kyle Vogt responded to the New York Times article. He claims 2.5 to 5 miles is the rate that Cruise vehicles request help from remote operators, not the rate that they actually get help. Vogt doesn’t say what the rate they actually get help is.
I’m a bit sus. If that number were so much better than the 2.5-5 miles number cited by the times, why wouldn’t he come out and say it?
What do you mean?
Did you link to the wrong video?
Thanks so much for publishing this. It’s so refreshing to read about alignment plans that people think might work, rather than just reading about why alignment is putatively hard or why naive approaches to the problem would fail.
Demis Hassabis has publicly stated that Google DeepMind’s upcoming Gemini model will be some sort of combination of an RL agent and an LLM, but AFAIK he hasn’t given more details than that. I’m very curious to see what they’ve come up with.
This was a very illuminating newsletter. It is nice to hear a diversity of perspectives on alignment.
Second, we don’t know how difficult AI safety will turn out to be; he gives a probability of ~10% that the problem is as hard as (a caricature of) MIRI suggests, where any design not based on mathematical principles will be unsafe. This is especially true because as we get closer to AGI we’ll have many more powerful AI techniques that we can leverage for safety.
How accurate is it to say that MIRI believes a mathematical solution to the alignment problem is the only solution? Does MIRI think that without a formal proof of an AGI’s safety, it will cause human extinction?
I think folks are being less charitable than they could be to LeCun. LeCun’s views and arguments about AI risk are strongly counterintuitive to many people in this community who are steeped in alignment theory. His arguments are also more cursory and less fleshed-out than I would ideally like. But he’s a Turing Award winner, for God’s sake. He’s a co-inventor of deep learning.
LeCun has a rough sketch of a roadmap to AGI, which includes a rough sketch of a plan for alignment and safety. Ivan Vendrov writes:
Broadly, it seems that in a world where LeCun’s architecture becomes dominant, useful AI safety work looks more analogous to the kind of work that goes on now to make self-driving cars safe. It’s not difficult to understand the individual components of a self-driving car or to debug them in isolation, but emergent interactions between the components and a diverse range of environments require massive and ongoing investments in testing and redundancy.
For this reason, LeCun thinks of AI safety as an engineering problem analogous to aviation safety or automotive safety. Conversely, disagreeing with LeCun on AI safety would seem to imply a different view of the technical path to developing AGI.
Very lucidly written. Thanks.
Broadly, it seems that in a world where LeCun’s architecture becomes dominant, useful AI safety work looks more analogous to the kind of work that goes on now to make self-driving cars safe. It’s not difficult to understand the individual components of a self-driving car or to debug them in isolation, but emergent interactions between the components and a diverse range of environments require massive and ongoing investments in testing and redundancy.
I think this is the crux of the matter. This is why LeCun tweeted:
One cannot just “solve the AI alignment problem.” Let alone do it in 4 years. One doesn’t just “solve” the safety problem for turbojets, cars, rockets, or human societies, either. Engineering-for-reliability is always a process of continuous & iterative refinement.
LeCun, like Sam Altman, believes in an empirical, iterative approach to AI safety. This is in sharp contrast to the highly theoretical, figure-it-all-out-far-in-advance approach of MIRI.
I don’t get why some folks are so dismissive of the empirical, iterative approach. Is it because they believe in a fast takeoff?
Now, it doesn’t immediately follow that the AI will actually want to start buying chair-straps and heroin, for a similar reason as why I personally am not trying to get heroin right now.
This seems important to me. What is the intrinsic cost in a human brain like mine or yours? Why don’t humans have an alignment problem (e.g. if you radically enhanced human intelligence, you wouldn’t produce a paperclip maximiser)?
Maybe the view of alignment pessimists is that the paradigmatic human brain’s intrinsic cost is intractably complex. I don’t know. I would like more clarity on this point.
I’m very glad that you’re raising this topic for discussion. I’ve recently been thinking about the same thing myself. If I could delay the advent of AGI/TAI by, say, 100 years, would I? There are at least two relevant considerations here: 1) the commonly held (but not, to me, obviously true) assumption that delaying AGI/TAI increases the probability of it being safely deployed and 2) the costs of delaying AGI/TAI.
(2) motivates thinking really hard about whether (1) is true. General concern about AI safety also motivates thinking hard about whether (1) is true, since if delaying AGI/TAI does not increase the probability of a safe deployment, then we should think about what would increase it.