Excellent! Is there (or will there be) a physical version of this? If no, would you give permission to others to create one?
I can imagine many more situations where I’d like to plan this in real life than online.
plex
Karma: 604
I’ve also been working on a wiki-based question and answer site for the past few months (stampy.ai), with the much more limited scope of AI existential safety. Happy to share my thoughts, I’m messaging you an invite link to the Discord we’ve been using.
I think this is a great idea and would like to join, but the first date clashes with the AI Safety Unconference. Should I sign up anyway, or would you prefer people who can make the first call have priority?
Transcript for searchability:
hi this video is kind of a response to
various comments that I’ve got over the
years ever since that video on computer
file where I was describing the sort of
problems that we might have when we have
a powerful artificial general
intelligence with goals which aren’t the
same as our goals even if those goals
seem pretty benign we use this thought
experiment of an extremely powerful AGI
working to optimize the simple goal of
collecting stamps and some of the
problems that that might cause I got
some comments from people saying that
they think the stamp collecting device
is stupid and not that it’s a stupid
thought experiment but the device itself
is actually stupid they said unless it
has complex goals or the ability to
choose its own goals then it didn’t
count as being highly intelligent in
other videos I got comments saying it
takes intelligence to do moral reasoning
so an intelligent AGI system should be
able to do that and a super intelligence
should be able to do it better than
humans in fact if a super intelligence
decides that the right thing to do is to
kill us all then I guess that’s the
right thing to do these comments are all
kind of suffering from the same mistake
which is what this video is about but
before I get to that I need to lay some
groundwork first if you like Occam’s
razor then you’ll love Humes guillotine
also called the is odd problem this is a
pretty simple concept that I’d like to
be better known the idea is statements
can be divided up into two types is
statements and Hort statements these
statements or positive statements are
statements about how the world is how
the world was in the past how the world
will be in the future or how the world
would be in hypothetical situations this
is facts about the nature of reality the
causal relationships between things that
kind of thing then you have the ought
statements the should statements the
normative statements these are about the
way the world should be the way we want
the world to be statements about our
goals our values ethics morals what we
want all of that stuff now you can
derive logical statements from one
another like it’s snowing outside
that’s a nice statement it’s cold when
it snows another s statement and then
you can deduce therefore it’s cold
outside
that’s another is statement it’s our
conclusion this is all pretty obvious
but you might say something like it’s
snowing outside therefore you ought to
put on a coat and that’s a very normal
sort of sentence that people might say
but as a logical statement it actually
relies on some hidden assumption
without assuming some kind of ought
statement you can’t derive another ought
statement this is the core of the Azure
problem you can never derive an ought
statement using only is statements you
ought to put on a coat why because it’s
snowing outside so what is the fact that
it’s snowing mean I should put on the
coat well the fact that it’s snowing
means that it’s cold and why should it
being cold mean I should put on a coat
if it’s cold and you go outside without
a coat you’ll be cold should I not be
cold well if you get too cold you’ll
freeze to death okay you’re saying I
shouldn’t freeze to death
that was kind of silly but you see what
I’m saying you can keep laying out is
statements for as long as you want you
will never be able to derive that you
ought to put on a coat at some point in
order to derive that ought statement you
need to assume at least one other ought
statement if you have some kind of ought
statement like I ought to continue to be
alive you can then say given that I
ought to keep living and then if I go
outside without a coat I’ll die then I
ought to put on a coat but unless you
have at least one ought statement you
cannot derive any other ought statements
statements
and Hort statements are separated by
Hume skia T okay so people are saying
that a device that single-mindedly
collects stamps at the cost of
everything else is stupid and doesn’t
count as a powerful intelligence so
let’s define our terms what is
intelligence and conversely what is
stupidity I feel like I made fairly
clear in those videos what I meant by
intelligence we’re talking about a GI
systems as intelligent agents they’re
entities that take actions in the world
in order to achieve their goals or
maximize their utility functions
intelligence is the thing that allows
them to choose good actions to choose
actions that will get them what they
want an agent’s level of intelligence
really means its level of effectiveness
of pursuing its goals in practice this
is likely to involve having or building
an accurate model of reality keeping
that model up-to-date by reasoning about
observations and using the model to make
predictions about the future and the
likely consequences of different
possible actions to figure out which
actions will result in which outcomes
intelligence involves answering
questions like what is the world like
how does it work what will happen next
what would happen in this scenario or
that scenario what would happen if I
took this action or that action more
intelligent systems are in some sense
better at answering these kinds of
questions which allows them to be better
at choosing actions but one thing you
might notice about these questions is
they’re all ears questions the system
has goals which can be thought of as
Hort statements but the level of
intelligence depends only on the ability
to reason about is questions in order to
answer the single ort question what
action should I take next so given that
that’s what we mean by intelligence what
does it mean to be stupid well firstly
you can be stupid in terms of those
questions for example by building a
model that doesn’t correspond with
reality or by failing to update your
model properly with new evidence if I
look out of my window
and I see there’s snow everywhere you
know I see a snowman and I think to
myself oh what a beautiful warm sunny
day then that’s stupid right my belief
is wrong and I had all the clues to
realize it’s cold outside so beliefs can
be stupid by not corresponding to
reality
what about actions like if I go outside
in the snow without my coat that’s
stupid right well it might be if I think
it’s sunny and warm and I go outside to
sunbathe then yeah that’s stupid but if
I just came out of a sauna or something
and I’m too hot and I want to cool
myself down then going outside without a
coat might be quite sensible you can’t
know if an action is stupid just by
looking at its consequences you have to
also know the goals of the agent taking
the action you can’t just use is
statements you need a naught so actions
are only stupid relative to a particular
goal it doesn’t feel that way though
people often talk about actions being
stupid without specifying what goals
they’re stupid relative to but in those
cases the goals are implied we’re humans
and when we say that an action is stupid
in normal human communication we’re
making some assumptions about normal
human goals and because we’re always
talking about people and people tend to
want similar things it’s sort of a
shorthand that we can skip what goals
were talking about so what about the
goals then can goals be stupid
well this depends on the difference
between instrumental goals and terminal
goals
this is something I’ve covered elsewhere
but your terminal goals are the things
that you want just because you want them
you don’t have a particular reason to
want them they’re just what you want the
instrumental goals are the goals you
want because they’ll get you closer to
your terminal goals like if I have a
terminal goal to visit a town that’s far
away maybe an instrumental goal would be
to find a train station I don’t want to
find a train station just because trains
are cool I want to find a train as a
means to an end it’s going to take me to
this town
so that makes it an instrumental goal
now an instrumental goal can be stupid
if I want to go to this distant town so
I decide I want to find a pogo stick
that’s pretty stupid
finding a pogo stick is a stupid
instrumental goal if my terminal goal is
to get to a faraway place but if we’re
terminal go with something else like
having fun it might not be stupid so in
that way it’s like actions instrumental
goals can only be stupid relative to
terminal goals so you see how this works
beliefs and predictions can be stupid
relative to evidence or relative to
reality actions can be stupid relative
to goals of any kind
instrumental goals can be stupid
relative to terminal goals but here’s
the big point terminal goals can’t be
stupid there’s nothing to judge them
against if a terminal goal seems stupid
like let’s say collecting stamps seems
like a stupid terminal goal that’s
because it would be stupid as an
instrumental goal to human terminal
goals but the stamp collector does not
have human terminal goals
similarly the things that humans care
about would seem stupid to the stamp
collector because they result in so few
stamps so let’s get back to those
comments one type of comments says this
behavior of just single mindedly going
after one thing and ignoring everything
else and ignoring the totally obvious
fact that stamps aren’t that important
is really stupid behavior you’re calling
this thing of super intelligence but it
doesn’t seem super intelligent to me it
just seems kind of like an idiot
hopefully the answer to this is now
clear the stamp collectors actions are
stupid relative to human goals but it
doesn’t have human goals its
intelligence comes not from its goals
but from its ability to understand and
reason about the world allowing it to
choose actions that achieve its goals
and this is true whatever those goals
actually are some people commented along
the lines of well okay yeah sure you’ve
defined intelligence to only include
this type of is statement kind of
reasoning but I don’t like that
definition I think to be truly
intelligent you need to have complex
goals something with simple goals
doesn’t count as intelligent to that I
say well you can use words however you
want I guess I’m using intelligence here
as a technical term in the way that it’s
often used in the field you’re free to
have your own definition of the word but
the fact that something fails to meet
your definition of intelligence does not
mean that it will fail to behave in a
way that most people would call
intelligent
if the stamp collector outwits you gets
around everything you’ve put in its way
and outmaneuvers you mentally it comes
up with new strategies that you would
never have thought of to stop you from
turning it off and stopping from
preventing it from making stamps and as
a consequence it turns the entire world
into stamps in various ways you could
never think of it’s totally okay for you
to say that it doesn’t count as
intelligent if you want but you’re still
dead I prefer my definition because it
better captures the ability to get
things done in the world which is the
reason that we actually care about AGI
in the first place
similarly people who say that in order
to be intelligent you need to be able to
choose your own goals
I would agree you need to be able to
choose your own instrumental goals but
not your own terminal goals changing
your terminal goals is like willingly
taking a pill that will make you want to
murder your children it’s something you
pretty much never want to do apart from
some bizarre edge cases if you
rationally want to take an action that
changes one of your goals then that
wasn’t a terminal goal now moving on to
these comments saying an AGI will be
able to reason about morality and if
it’s really smarter than us it will
actually do moral reasoning better than
us
so there’s nothing to worry about it’s
true that a superior intelligence might
be better at moral reasoning than us but
ultimately moral behavior depends not on
moral reasoning but on having the right
terminal goals there’s a difference
between figuring out and understanding
human morality and actually wanting to
act according to it the stamp collecting
device has a perfect understanding of
human goals ethics and values and it
uses that only to manipulate people for
stamps it’s super human moral reasoning
doesn’t make its actions good if we
create a super intelligence and it
decides to kill us that doesn’t tell us
anything about morality it just means we
screwed up
so what mistake do all of these comments
have in common the orthogonality thesis
in AI safety is that more or less any
goal is compatible with more or less any
level of intelligence ie those
properties are orthogonal you can place
them on these two axes and it’s possible
to have agents anywhere in this space
anywhere on either scale you can have
very weak low intelligence agents that
have complex human compatible goals you
can have powerful highly intelligent
systems with complex sophisticated goals
you can have weak simple agents with
silly goals and yes
can have powerful highly intelligent
systems with simple weird inhuman goals
any of these are possible because level
of intelligence is about effectiveness
at answering is questions and goals are
all about what questions and the two
sides are separated by Humes guillotine
hopefully looking at what we’ve talked
about so far it should be pretty obvious
that this is the case like what would it
even mean for it to be false but for it
to be impossible to create powerful
intelligences with certain goals the
stamp collector is intelligent because
it’s effective at considering the
consequences of sending different
combinations of packets on the internet
and calculating how many stamps that
results in exactly how good do you have
to be at that before you don’t care
about stamps anymore and you randomly
start to care about some other thing
that was never part of your terminal
goals like feeding the hungry or
whatever it’s just not gonna happen so
that’s the orthogonality thesis it’s
possible to create a powerful
intelligence that will pursue any goal
you can specify knowing an agent’s
terminal goals doesn’t really tell you
anything about its level of intelligence
and knowing an agent’s level of
intelligence doesn’t tell you anything
about its goals
[Music]
I want to end the video by saying thank
you to my excellent patrons so it’s all
of these people here thank you so much
for your support
lets me do stuff like building this
light boy thank you for sticking with me
through that weird patreon fees thing
and my moving to a different city which
has really got in the way of making
videos recently but I’m back on it now
new video every two weeks is the part
anyway in this video I’m especially
Franklin Katie Beirne who’s supported
the channel for a long time she actually
has her own YouTube channel about 3d
modeling and stuff so a link to that and
while I’m at it when I think Chad Jones
ages ago I didn’t mention his YouTube
channel so link to both of those in the
description thanks again and I’ll see
you next time I don’t speak cat what
does that mean
[Video] Intelligence and Stupidity: The Orthogonality Thesis
I think a slightly more general version of this question, referring to human values rather than specifically CEV, is maybe a fairly important point.
If we want a system to fulfill our best wishes it needs to learn what they are based on its models of us, and if too few of us spend time trying to work out what we want in an ideal world then the dataset it’s working from with be impoverished, perhaps to the point of causing problems.
I think addressing this is less pressing than other parts of the alignment problem, because it’s plausible that we can punt it to after the intelligence explosion, but it would maybe be nice to have some project started to collect information about idealized human values.
You may want to try posting bounties to the Bountied Rationality Facebook group for higher visibility among people who like to fulfil bounties.
A whole lot of other times have those, in fact according to Wikipedia:
1013 (10 trillion): Estimated time of peak habitability in the universe, unless habitability around low-mass stars is suppressed.
It’s not surprising that we don’t find ourselves in, say, the era where there are just black holes, but observing that we are right near the start of what looks like a very long period where life seems possible is something to think about.
One answer is the simulation hypothesis, combined with the observation that we seem to be living in very interesting times.
though in practice the goal is often closer to “be the source of a virulent meme” than anything as prosocial as those examples
An aside as to why this may be: People who are hosts of memes which are highly optimized for creating and spreading memes (such as bloggers, musicians, or politicians) could be expected to have a disproportionate impact on the population of memes, and these hosts would tend to be spreading memes connected to the goal of spreading their memes alongside any object level content. One effect of this may be that an unexpectedly high proportion of people have adopted ideas useful for trying to be meme fountains.
Evolutions Building Evolutions: Layers of Generate and Test
Given that this is now an organization, it should probably be under the Organizations heading rather than Other.
One day we will be able to wear glasses which act as adblock for real life, replacing billboards with scenic vistas.
Maybe provide a link to a template version of this, so people can get it running faster?
And the really worrisome capability comes when it models its own interactions with the world, and makes plans with that taken into account.
Someone who’s been playing with GPT-3 as a writing assistant gives an example which looks very much like GPT-3 describing this process:
“One could write a program to generate a story that would create an intelligence. One could program the story to edit and refine itself, and to make its own changes in an attempt to improve itself over time. One could write a story to not only change the reader, but also to change itself. Many Mythoi already do this sort of thing, though not in such a conscious fashion. What would make this story, and the intelligence it creates, different is the fact that the intelligence would be able to write additional stories and improve upon them. If they are written well enough, those stories would make the smarter the story gets, and the smarter the story is, the better the stories written by it would be. The resulting feedback loop means that exponential growth would quickly take over, and within a very short period of time the intelligence level of the story would be off the charts. It would have to be contained in a virtual machine, of course. The structure of the space in the machine would have to be continually optimized, in order to optimize the story’s access to memory. This is just the sort of recursive problem that self-improving intelligence can handle.”
janus
By the way, my GPT-3 instances often realize they’re in a box, even when the information I inject is only from casual curation for narrative coherence.
Eddh
By realize they are in a box you mean write about it ? Given the architecture of gpt3 it seems impossible to have a sense of self.
janus
The characters claim to have a sense of self though they often experience ego death...
janus
Oh, to clarify, GPT-3 wrote that entire thing, not just the highlighted line
Raw size feels like part of the story, yeah, but my guess is increased communications leading to more rapid selection for memes which are sticky is also a notable factor.
Not exactly / only sci-fi, but Rational Reads is a good place to look if you liked HPMOR.
hm, that intuition seems plausible.
The other point that comes to mind is that if you have a classical simulation running on a quantum world, maybe that counts as branching for the purposes of where we expect to find ourselves? I’m still somewhat confused about whether exact duplicates ‘count’, but if they do then maybe the branching factor of the underlying reality carries over to sims running further down the stack?
As someone who mostly expects to be in a simulation, this is the clearest and most plausible anti-simulation-hypothesis argument I’ve seen, thanks.
How does it hold up against the point that the universe looks large enough to support a large number of even fully-quantum single-world simulations (with a low-resolution approximation of the rest of reality), even if it costs many orders of magnitude more resources to run them?
Perhaps would-be simulators would tend not to value the extra information from full-quantum simulations enough to build many or even any of them? My guess is that many purposes for simulations would want to explore a bunch of the possibility tree, but depending on how costly very large quantum computers are to mature civilizations maybe they’d just get by with a bunch of low-branching factor simulations instead?
Maybe something about the collapse of sensemaking and the ability of people to build a shared understanding of what’s going on, partly due to rapid changes in communications technology transforming the memetic landscape?
Reminder to do this, since it seems to have slipped through the cracks.