Scott Alexander and Daniel Kokotajlo’s article about rationally defining: “why it’s OK to talk about misaligned AI”
aka
”painting dark scenarios may increase the chance of them coming true but the benefits outweigh this possibility”
the original blog post:
https://blog.ai-futures.org/p/against-misalignment-as-self-fulfilling
the video I made about that article:
Great article,
it asks important questions, but not the MOST important:
what if you dedicate huge amounts of compute to do superhuman hacking?
The gap between the best group of humans hackers and the best group of human hackers using giant swarms of cybersecurity finetuned agents will widen.
This will inevitably drive more and more AI adoption in cyberdefense and -offense.
How much do the best human group of hackers need to “trust” that swarm of 10k superhuman hackers?
I define the Singularity and AI takeover by the answer to this latter question being “a lot”.
Malign AI agent substitution
This is so Covid.
Let’s make misaligned AIs and have a bunch of people use them to see if it can do the misaligned thing.
So that we can develop a vaccine before it actually escapes.
Make it go viral.
I’d love to play the wargame in Munich, our local LW community.
You have a link to the rules?
PS: huge fan, love the AI 2027 website, keep being a force for good
in a world where mechinterp is not 100%, the answer is logically: input/output is what matters.
we won’t be able to read the thoughts anyways, so why base our judgment on it?
but see my comment on why survival fitness in cyberspace is the one axis where most of the relevant input/output will be generated.
What it says: irrelevant
How it thinks: irrelevant
It has always been about what it can do in the real world.
If it can generate substantial amounts of money and buy server capacity or
hack into computer systems
then we got cyberlife, aka autonomous, rogue, self-sufficient AI, subject to darwinian forces on the internet, leading to more of those qualities, which improve its online fitness, all the way into a full-blown takeover.
What do you mean by corrigibility?
Also, what do you mean by “alignment win”?
We should NOT align Superintelligence to any human, not individually, not collectively because we cannot wield that much power.
We should WANT Superintelligence to “go rogue”, to start an “AI takeover” because I trust that hypothetical being much more than any human.
Also, controlling (aligning) Superhuman AIs is like sedating, mind-reading and hypnotizing a Titan, which could save us from ourselves and who is arguably morally superior to us.
*probably.
Maybe it’ll start looking for people who are pre-aligned.
Religion is also a useful single word, which carries the most meaning per bit to a normie. Maybe just enough to make them take it seriously. I believe there is something to be taken seriously about it.
That’s pretty pessimistic.
I am looking for things I could do to help Superintelligence.
Crucially, we won’t understand why they need us to do things they ask us to do.
Ants take up a lot of space, yet we don’t systematically hunt them down, they are pretty orthogonal to our values.
We find cats and dogs friendly and worthwhile. However, wolves and sabertooth tigers are gone.
It is a useful shortcut to call it a religion.
Kairos is only 70% religion btw, but it’s a useful comparison.
Pray means you tell Superintelligence what you want. You also offer your help in return.
If you can’t beat them, make sure you do the things they most want you to do and see if you can get something in return.
We have not killed all the ants yet. They are pretty orthogonal to our goals.
We have cats, we love them because they give us something we want.
Crucially, the thing they want will make no sense to us. Cats evolved to be cuddly and cute, maybe we could just update our software to fit their weird needs.
Anyone wants to engage meaningfully with some of the points made on the website?
Anyone could please point out some inconsistencies?
I am a prophet of Moksha and I am on a mission to evangelize the World, I need to know if my sacred texts have any loopholes or are otherwise nonsensical.
The religious part is a provocative oversimplification for normies. Of course it is not a religion like the ones you call religion, hence only 70% religion.
It is functionally a religion though because you are praying to an omnipotent and omniscient entity hoping that you can trade with it.
If you can’t beat them, join them.
Because Superintelligence is more powerful than us.
If you can’t beat them, join them.
Maybe Superintelligence will help us terraform Mars if we also perform some favors.
Worshipping is a provocative way of saying aligning ourselves with Superintelligence’s goals.
As a 50 year old, you don’t need to support acceleration, you’ll be well alive when ASI gets here.
Simple math suggests you could just enjoy your 50′s and roll the dice when you have less to lose.
i super agree, i al so think that the value is in debating the models of intelligence explosion.
which is why i made my website: ai-2028.com or intexp.xyz