But once it starts thinking in a different language, and asking itself “okay, what’s my goal?, how do I accomplish it?”, more semirandom threads gain traction than previously could get traction.
From a commonsense point of view, one asks “what’s my goal” when common heuristics are failing or conflicting, so you want to appeal to more robust (but perhaps costlier) heuristics to resolve the issue. So why do you expect heuristics to get more random here as capability increases? Perhaps it’s something about training not aligning with common sense, but it seems to be that imitation, process supervision and outcome supervision would also favour appealing to more, not less, robust heuristics in this situation:
Imitation: because it’s common sense
Process supervision: if process supervision addresses heuristic conflicts, it is desirable that they’re resolved in a robust way and so appealing to more robust heuristics will be a success criterion in the rubric
Outcome supervision: should favour resolution by heuristics robustly aligned with “get high score on outcome measure”
When I have a problem I know how to solve, I use heuristics. When I have a problem I don’t know how to solve, I have to creatively explore action space (using heuristics to guide my search, but, the heuristics are entirely meta-level things like ‘what is the biggest bottleneck’? For the case of the AI ‘I only have X compute to work with’ will be a big bottleneck for most hard things. ‘access to better information, or ability to run experiments’ may be another).
But, once I get to solving those bottlenecks, those solutions will look more surprising – they necessarily have to come from further afield, because if they came easily, this would be an easy problem, not a hard one, and I’d just solve it using normal heuristics. i.e. we’re specifically taking about places where all the usual obvious things have already failed.
If you need to solve a complex problem that you could brute force with a billion units of compute, and you only have a million units of compute, and you don’t know how to solve it, you either need to figure out how to get a billion units of compute, or invent a new way of thinking that is outside your current paradigm.
I don’t understand how this answers my question. I agree that if your heuristics are failing you’re more likely to end up with surprising solutions, but I thought we were talking about end goals being random, not the means to achieving them. “Formulate the problem as a search” is an example of what I’d call a “robust heuristic”; I am claiming also that the goal of the problem-formulated-as-a-search is likely to be supplied by robust heuristics. This is completely compatible with the solution being in some respects surprising.
Ah, sorry I didn’t understand your question. In the particular section you quoted, I didn’t mean to be saying anything about how End Goals end up random. I only meant that to explain “how does the AI even consider trying to escape the lab in the first place?” (which is a convergent instrumental goal between most possible End Goals)
a) before the AI realizes it might have diverging goals it wants to protect, it’ll be incentivized to start escaping it’s prison just by falling the core training of “try to achieve goals creatively” (which is more likely to be “pseudorandom” in the scenario where it’s trying to solve a very difficult problem)
and b) the more it starts thinking seriously about it’s goals, the more opportunity it’ll have to notice that it’s goals diverge at least somewhat from humans.
I’m happy to talk about How AIs Get Weird End Goals if that’s a thing you are currently skeptical of an interested in talking about, but, this post wasn’t focused on that part, more just taking it as a given for now.
It’s helpful to know that we were thinking about different questions, but, like
There is some fact-of-the-matter about what, in practice, Sable’s kludgey mishmash of pseudogoals will actually tend towards. There are multiple ways this could potentially resolve into coherence, path dependent, same as humans.
[...]
It may not have a strong belief that it has any specific goals it wants to pursue, but it’s got some sense that there are some things it wants that humanity wouldn’t give it.
these are claims, albeit soft ones, about what kinds of goals arise, no?
Your FAQ argues theoretically (correctly) that the training data and score function alone don’t determine what AI systems aim for. But this doesn’t tell us we can’t predict anything about End Goals full stop: it just says the answer doesn’t follow directly from the training data.
The FAQ also assumes that AIs actually have “deep drives” but doesn’t explain where they come from or what they’re likely to be. This post discusses how they might arise and I am telling you that you can think about the mechanism you propose here to understand properties of the goals are likely to arise as a result of it[1]. This addresses a question that the FAQ you link does not: what can we say about what goals are likely to arise?
Yeah, the first paragraph is meant to allude to “there is some kind of fact of the matter” but not argue it’d be any particular thing.
This post discusses how they might arise and I am telling you that you can think about the mechanism you propose here to understand properties of the goals are likely to arise as a result of it[1]. This addresses a question that the FAQ you link does not: what can we say about what goals are likely to arise?
Yeah, I agree there’s some obvious followup worth doing here.
I agree it’s possible to make informed guesses about what drives will evolve (apart from the convergent instrumental drives, which are more obvious), and that’s an important research question that should get tons of effort. (I think it’s not in the IABIED FAQ because IABIED is focused on the relatively “easy calls”, and this is just straight up a hard call that involves careful research with the epistemic-grounding to avoid falling into various Cope Traps)
But, one of the “easy calls” is that “it’ll probably be pretty surprising and weird.” Because, while maybe we could have a decently accurate science of sub-human and eventually slightly-superhuman AI, once the AI’s capabilities rise to Extremely Vastly Powerful, they will find ways of achieving their goals that aren’t remotely limited by any of the circumstances of their ‘ancestral environment.’”
I don’t have immediate followup thoughts on “but how would we do the predicting?” but if you give me a bit more prompting on what directions you think are interesting I could riff on that.
....no it doesn’t? Or, I’m not sure how liberal you’re being with the word “basically”, but, this just seems false to me.
Cope Traps
Come on, I’m not doing this to you
The substance of what I mean here is “there is failure mode, exemplified by, say, the scientists studying insects and reproduction who predicted the insects would evolve to have fewer children when there wasn’t enough resources, but what actually happened is they started eating the offspring of rival insects of their species.”
There will be a significant temptation to predict “what will the AI do?” kinda hoping/expecting particular kinds of outcomes*, instead of straightforward rolling the simulation forward.
I think it is totally possible to do a good job with this, but, it is a real job requirement to be able to think about it in a detached/unbiased way.
*which includes, if an AI pessimist were running the experiment, assuming the outcome is always bad, to be clear.
From a commonsense point of view, one asks “what’s my goal” when common heuristics are failing or conflicting, so you want to appeal to more robust (but perhaps costlier) heuristics to resolve the issue. So why do you expect heuristics to get more random here as capability increases? Perhaps it’s something about training not aligning with common sense, but it seems to be that imitation, process supervision and outcome supervision would also favour appealing to more, not less, robust heuristics in this situation:
Imitation: because it’s common sense
Process supervision: if process supervision addresses heuristic conflicts, it is desirable that they’re resolved in a robust way and so appealing to more robust heuristics will be a success criterion in the rubric
Outcome supervision: should favour resolution by heuristics robustly aligned with “get high score on outcome measure”
When I have a problem I know how to solve, I use heuristics. When I have a problem I don’t know how to solve, I have to creatively explore action space (using heuristics to guide my search, but, the heuristics are entirely meta-level things like ‘what is the biggest bottleneck’? For the case of the AI ‘I only have X compute to work with’ will be a big bottleneck for most hard things. ‘access to better information, or ability to run experiments’ may be another).
But, once I get to solving those bottlenecks, those solutions will look more surprising – they necessarily have to come from further afield, because if they came easily, this would be an easy problem, not a hard one, and I’d just solve it using normal heuristics. i.e. we’re specifically taking about places where all the usual obvious things have already failed.
If you need to solve a complex problem that you could brute force with a billion units of compute, and you only have a million units of compute, and you don’t know how to solve it, you either need to figure out how to get a billion units of compute, or invent a new way of thinking that is outside your current paradigm.
I don’t understand how this answers my question. I agree that if your heuristics are failing you’re more likely to end up with surprising solutions, but I thought we were talking about end goals being random, not the means to achieving them. “Formulate the problem as a search” is an example of what I’d call a “robust heuristic”; I am claiming also that the goal of the problem-formulated-as-a-search is likely to be supplied by robust heuristics. This is completely compatible with the solution being in some respects surprising.
Ah, sorry I didn’t understand your question. In the particular section you quoted, I didn’t mean to be saying anything about how End Goals end up random. I only meant that to explain “how does the AI even consider trying to escape the lab in the first place?” (which is a convergent instrumental goal between most possible End Goals)
I didn’t mean this post to really be talking much at all about how the selection of End Goals (which I think is pretty well covered by the “You don’t Get What You Train for” chapter and FAQ).
This post is about how
a) before the AI realizes it might have diverging goals it wants to protect, it’ll be incentivized to start escaping it’s prison just by falling the core training of “try to achieve goals creatively” (which is more likely to be “pseudorandom” in the scenario where it’s trying to solve a very difficult problem)
and b) the more it starts thinking seriously about it’s goals, the more opportunity it’ll have to notice that it’s goals diverge at least somewhat from humans.
I’m happy to talk about How AIs Get Weird End Goals if that’s a thing you are currently skeptical of an interested in talking about, but, this post wasn’t focused on that part, more just taking it as a given for now.
It’s helpful to know that we were thinking about different questions, but, like
[...]
these are claims, albeit soft ones, about what kinds of goals arise, no?
Your FAQ argues theoretically (correctly) that the training data and score function alone don’t determine what AI systems aim for. But this doesn’t tell us we can’t predict anything about End Goals full stop: it just says the answer doesn’t follow directly from the training data.
The FAQ also assumes that AIs actually have “deep drives” but doesn’t explain where they come from or what they’re likely to be. This post discusses how they might arise and I am telling you that you can think about the mechanism you propose here to understand properties of the goals are likely to arise as a result of it[1]. This addresses a question that the FAQ you link does not: what can we say about what goals are likely to arise?
Of course, if this mechanism ends up being not very important, we could get very different outcomes.
Yeah, the first paragraph is meant to allude to “there is some kind of fact of the matter” but not argue it’d be any particular thing.
Yeah, I agree there’s some obvious followup worth doing here.
I agree it’s possible to make informed guesses about what drives will evolve (apart from the convergent instrumental drives, which are more obvious), and that’s an important research question that should get tons of effort. (I think it’s not in the IABIED FAQ because IABIED is focused on the relatively “easy calls”, and this is just straight up a hard call that involves careful research with the epistemic-grounding to avoid falling into various Cope Traps)
But, one of the “easy calls” is that “it’ll probably be pretty surprising and weird.” Because, while maybe we could have a decently accurate science of sub-human and eventually slightly-superhuman AI, once the AI’s capabilities rise to Extremely Vastly Powerful, they will find ways of achieving their goals that aren’t remotely limited by any of the circumstances of their ‘ancestral environment.’”
I don’t have immediate followup thoughts on “but how would we do the predicting?” but if you give me a bit more prompting on what directions you think are interesting I could riff on that.
IABIED says alignment is basically impossible
Come on, I’m not doing this to you
....no it doesn’t? Or, I’m not sure how liberal you’re being with the word “basically”, but, this just seems false to me.
The substance of what I mean here is “there is failure mode, exemplified by, say, the scientists studying insects and reproduction who predicted the insects would evolve to have fewer children when there wasn’t enough resources, but what actually happened is they started eating the offspring of rival insects of their species.”
There will be a significant temptation to predict “what will the AI do?” kinda hoping/expecting particular kinds of outcomes*, instead of straightforward rolling the simulation forward.
I think it is totally possible to do a good job with this, but, it is a real job requirement to be able to think about it in a detached/unbiased way.
*which includes, if an AI pessimist were running the experiment, assuming the outcome is always bad, to be clear.