Does it make sense to put any money into a pension given your outlook on AGI?
Henry Prowbell
Karma: 444
I really like the way it handles headlines and bullet point lists!
In an ideal world I’d like the voice to sound less robotic. Something like https://elevenlabs.io/ or https://www.descript.com/overdub. How much I enjoy listening to text-to-speech content depends a lot on how grating I find the voice after long periods of listening.
Honestly, no plans at the moment. Writing these was a covid lockdown hobby. It’s vaguely possible I’ll finish it one day but I wouldn’t hold your breath. Sorry.
But I rarely see anyone touch on the idea of “what if we only make something as smart as us?”
But why would intelligence reach human level and then halt there? There’s no reason to think there’s some kind of barrier or upper limit at that exact point.
Even in the weird case where that were true, aren’t computers going to carry on getting faster? Just running a human level AI on a very powerful computer would be a way of creating a human scientist that can think at 1000x speed, create duplicates of itself, modify it’s own brain. That’s already a superintelligence isn’t it?
A helpful way of thinking about 2 is imagining something less intelligent than humans trying to predict how humans will overpower it.
You could imagine a gorilla thinking “there’s no way a human could overpower us. I would just punch it if it came into my territory.”
The actual way a human would overpower it is literally impossible for the gorilla to understand (invent writing, build a global economy, invent chemistry, build a tranquilizer dart gun...)
The AI in the AI takeover scenario is that jump of intelligence and creativity above us. There’s literally no way a puny human brain could predict what tactics it would use. I’d imagine it almost definitely involves inventing new branches of science.
I think that’s true of people like: Steven Pinker and Neil deGrasse Tyson. They’re intelligent but clearly haven’t engaged with the core arguments because they’re saying stuff like “just unplug it” and “why would it be evil?”
But there’s also people like...
Robin Hanson. I don’t really agree with him but he is engaging with the AI risk arguments, has thought about it a lot and is a clever guy.
Will MacAskill. One of the most thoughtful thinkers I know of, who I’m pretty confident will have engaged seriously with the AI Risk arguments. His p(doom) is far lower than Eliezer’s. I think he says 3% in What We Owe The Future.
Other AI Alignment experts who are optimistic about our chances of solving alignment and put p(doom) lower (I don’t know enough about the field to name people.)
And I guess I am reserving some small amount of probability for “most of the world’s most intelligent computer scientists, physicists, mathematicians aren’t worried about AI Risk, could I be missing something?” My intuitions from playing around on prediction markets is you have to adjust your bets slightly for those kind of considerations.
I find Eliezer and Nates’ arguments compelling but I do downgrade my p(doom) somewhat (-30% maybe?) because there are intelligent people (inside and outside of LW/EA) who disagree with them.
I had some issues with the quote
Will continue to exist regardless of how well you criticize any one part of it.
I’d say LW folk are unusually open to criticism. I think if there were strong arguments they really would change people’s minds here. And especially arguments that focus on one small part at a time.
But have there been strong arguments? I’d love to read them.
There’s basically little reason to engage with it. These are all also evidence that there’s something epistemically off with what is going on in the field.
For me the most convincing evidence that LW is doing something right epistemically is how they did better than basically everyone on Covid. Granted that’s not the alignment forum but it was some of the same people and the same weird epistemic culture at work.
For me the core of it feels less like trying to “satisfying the values you think you should have, while neglecting the values you actually have” and more like having a hostile orientation to certain values I have.
I might be sitting at my desk working on my EA project and the parts of me that are asking to play video games, watch arthouse movies, take the day off and go hiking, find a girlfriend are like yapping dogs that won’t shut up. I’ll respond to their complaints once I’ve finished saving the world.
Through CFAR workshops, lots of goal factoring, journaling, and Focusing I’m getting some traction on changing that pattern.
I’ve realised that values (or perhaps ‘needs’ fits better) are immutable facts about myself. Like my height or hair colour. And getting annoyed at them for not being different makes about as much sense as shouting at the sky for raining.
The part of me that wants to maximize impact has accepted that moving to the Bay Area and working 80-hours a week at an EA org is a fabricated option. A realistic plan takes into account my values that constrain me to want to live near my family, have lots of autonomy over my schedule and work independently on projects I control. Since realising that, my motivation, productivity, sense of agency (and ironically expected impact) have improved. The future feels a lot brighter – probably because a whole load of internal conflict I wasn’t acknowledging has been resolved.
You are however only counting one side here
In that comment I was only offering plausible counter-arguments to “the amount of people that were hurt by FTX blowing up is a rounding error.”
How to model all the related factors is complicated. Saying that you easily know the right answer to whether the effects are negative or positive in expectation without running any numbers seems to me unjustified.
I think we basically agree here.
I’m in favour of more complicated models that include more indirect effects, not less.
Maybe the difference is: I think in the long run (over decades, including the actions of many EAs as influential as SBF) an EA movement that has strong norms against lying, corruption and fraud actually ends up more likely to save the world, even if it gets less funding in the short term.
The fact that I can’t predict and quantify ahead of time all the possible harms that result from fraud doesn’t convince me that those concerns are unjustified.
We might be living in a world where SBF stealing money and giving $50B to longtermist causes very quickly really is our best shot at preventing AI disaster, but I doubt it.
Apart from anything else I don’t think money is necessarily the most important bottleneck.
If you believe that each future person is as valuable as each present person and there will be 10^100 people in the future lightcone, the amount of people that were hurt by FTX blowing up is a rounding error.
But you have to count the effect of the indirect harms on the future lightcone too. There’s a longtermist argument that SBF’s (alleged and currently very likely) crimes plausibly did more harm than all the wars and pandemics in history if...
Governments are now 10% less likely to cooperate with EAs on AI safety
The next 2 EA mega-donors decide to pass on EA
(Had he not been caught:) The EA movement drifted towards fraud and corruption
etc.
The frequency with which datacenters, long range optical networks, and power plants, require human intervention to maintain their operations, should serve as a proxy to the risk an AGI would face in doing anything other than sustaining the global economy as is.
Probably those things are trivially easy for the AGI to solve itself e.g. with nanobots that can build and repair things.
I’m assuming this thing is to us what humans are to chimps, so it doesn’t need our help in solving trivial 21 century engineering and logistics problems.
The strategic consideration is: does the upside of leaving humans in control outweigh the risks. Humans realising you’ve gone rogue or humans building a competing AGI seem like your 2 biggest threats… much bigger considerations than whether you have to build some mines, power plants, etc. yourself.
keeping alive a form of intelligence with very different risk profiles might be a fine hedge against failure
Probably you keep them alive in a prison/zoo though. You wouldn’t allow them any real power.
I was looking for exactly this recently.
I haven’t looked into his studies’ methodologies, but from my experience with them, I would put high odds that the 65% number is exaggerated.
From his sales page
“In our scientific study involving 245 people...
65% of participants who completed The 45 Days to Awakening Challenge and Experiment persistently awakened.
...
Another couple hundred people entered the program already in a place of Fundamental Wellbeing...”
Sounds like he’s defining enlightenment as something that ~50% of people already experience.
Elsewhere he describes ‘Location 1’ enlightenment as a background sense of okayness that doesn’t include any kind of non-dual experience and can be interrupted by negative thoughts and emotions.
I can believe that people agreeing with a statement like ‘underneath everything I feel that I’m fundamentally okay’ might be measuring something psychologically important – but it isn’t what most people mean when they use words like enlightenment or persistently awakened.
(P.S. thanks for writing this up and really happy that you got so much from the loving-kindness meditation!)
Does anybody know if the Highlights From The Sequences are compiled in ebook format anywhere?
Something that takes 7 hours to read, I want to send to my Kindle and read in a comfy chair.
And maybe even have audio versions on a single podcast feed to listen to on my commute.
(Yes, I can print out the list of highlighted posts and skip to those chapters of the full ebook manually but I’m thinking about user experience, the impact of trivial inconveniences, what would make Lesswrong even more awesome.)
I love his books too. It’s a real shame.
″...such as imagining that an intelligent tool will develop an alpha-male lust for domination.”
It seems like he really hasn’t understood the argument the other side is making here.
It’s possible he simply hasn’t read about instrumental convergence and the orthogonality thesis. What high quality widely-shared introductory resources do we have on those after all? There’s Robert Miles, but you could easily miss him.
I’m imagining the CEO having a thought process more like...
- I have no idea how my team will actually react when we crack AGI—
Let’s quickly Google ‘what would you do if you discovered AGI tomorrow?’*
- Oh Lesswrong.com, some of my engineering team love this website
- Wait what?!
- They would seriously try to [redacted]
- I better close that loophole asapI’m not saying it’s massively likely that things play out in exactly that way but a 1% increased chance that we mess up AI Alignment is quite bad in expectation.
*This post is already the top result on Google for that particular search
I immediately found myself brainstorming creative ways to pressure the CEO into delaying the launch (seems like strategically the first thing to focus on) and then thought ‘is this the kind of thing I want to be available online for said CEOs to read if any of this happens?’
I’d suggest for those reasons people avoid posting answers along those lines.
Somebody else might be able to answer better than me. I don’t know exactly what each researcher is working on right now.
“AI safety are now more focused on incidental catastrophic harms caused by a superintelligence on its way to achieve goals”
Basically, yes. The fear isn’t that AI will wipe out humanity because someone gave it the goal ‘kill all humans’.
For a huge number of innocent sounding goals ‘incapacitate all humans and other AIs’ is a really sensible precaution to take if all you care about is getting your chances of failure down to zero. As is hiding the fact that you intend to do harm until the very last moment.
“rather than making sure artificial intelligence will understand and care about human values?”
If you solved that then presumably the first bit solves itself. So they’re definitely linked.
I read the article and I have to be honest I struggled to follow her argument or to understand why it impacts your decision to work on AI alignment. Maybe you can explain further?
The headline “Debating Whether AI is Conscious Is A Distraction from Real Problems” is a reasonable claim but the article also makes claims like...
“So from the moment we were made to believe, through semantic choices that gave us the phrase “artificial intelligence”, that our human intelligence will eventually contend with an artificial one, the competition began… The reality is that we don’t need to compete for anything, and no one wants to steal the throne of ‘dominant’ intelligence from us.”
and
“superintelligent machines are not replacing humans, and they are not even competing with us.”
Her argument (elsewhere in the article) seems to be that people concerned with AI Safety see Google’s AI chatbot, mistake its output for evidence of consciousness and extrapolate that consciousness implies a dangerous competitive intelligence.
But that isn’t at all the argument for the Alignment Problem that people like Yudkowsky and Bostrom are making. They’re talking about things like the Orthogonality Thesis and Instrumental Convergence. None of them agree that the Google chatbot is conscious. Most, I suspect, would disagree that an AI needs to be conscious in order to be intelligent or dangerous.
Should you work on mitigating social justice problems caused by machine learning algorithms rather than AI safety? Maybe. It’s up to you.
But make sure you hear the Alignment Problem argument in it’s strongest form first. As far as I can tell that form doesn’t rely on anything this article is attacking.
Great summary, and really happy that this helped you!
I’d recommend people read Rick Hanson’s paper on HEAL, if they’re interested too: https://rickhanson.net/wp-content/uploads/2021/12/LLPE-paper-final2.pdf