There’s a fair bit on decision theory and on bayesean thinking, both of which are instrumental rationality. There’s not much on heuristics or how to deal with limited capacity. Perhaps intentionally—it’s hard to be rigorous on those topics.
Also, I think there’s an (unstated, and that should be fixed and the topic debated) belief that instrumental rationality without epistemic rationality is either useless or harmful. Certainly thta’s the FAI argument, and there’s no reason to believe it wouldn’t apply to humans. As such, a focus on epistemic rationality first is the correct approach.
That is, don’t try to improve your ability to meet goals unless you’re very confident in those goals.
Also, I think there’s an (unstated, and that should be fixed and the topic debated) belief that instrumental rationality without epistemic rationality is either useless or harmful.
I think some people agree with that, but I consider it backwards.
I’ll take winning over accurately predicting. Winning is the desired end; accurate prediction is a means, and not the only one.
Why not? If you haven’t yet decided what your goals are, being able to meet many goals is useful.
The AGI argument is that its goals might not be aligned with ours, are you saying that we should make sure that our future self’s goals be aligned with our current goals?
For example, if I know I am prone to hyperbolic discounting, I should take power from my future self so it will act according to my wishes rather than its own?
Being able to meet many goals is useful. Actually meeting wrong goals is not.
Your hyperbolic discounting example is instructive, as without a model of your goals, you cannot know whether your current or future self is correct. Most people come to the opposite conclusion—a hyperbolic discount massively overweights the short-term in a way that causes regret.
a hyperbolic discount massively overweights the short-term in a way that causes regret.
I meant that—when planning for the future, I want my future self to care about each absolute point in time as much as my current self does, or barring that, to only be able to act as if it did, hence the removal of power.
The correct goal is my current goal, obviously. After all, it’s my goal. My future self may disagree, prefering its own current goal. Correct is a two-place word.
If I let my current goal be decided by my future self, but I don’t know yet what it will decide, then I should accomodate as many of its possible choices as possible.
There’s a fair bit on decision theory and on bayesean thinking, both of which are instrumental rationality. There’s not much on heuristics or how to deal with limited capacity. Perhaps intentionally—it’s hard to be rigorous on those topics.
Also, I think there’s an (unstated, and that should be fixed and the topic debated) belief that instrumental rationality without epistemic rationality is either useless or harmful. Certainly thta’s the FAI argument, and there’s no reason to believe it wouldn’t apply to humans. As such, a focus on epistemic rationality first is the correct approach.
That is, don’t try to improve your ability to meet goals unless you’re very confident in those goals.
I think some people agree with that, but I consider it backwards.
I’ll take winning over accurately predicting. Winning is the desired end; accurate prediction is a means, and not the only one.
Umm, that’s what I’m trying to say. If you don’t know what “winning” is, you don’t know whether your accurate predictions help you win or not.
Were you? I’m not seeing what you’re saying align with what I said.
On a perhaps related issue, you don’t need to know what winning is, to win.
Competence without comprehension, a la Dennett.
Sure, but that’s luck, not rationality.
Why not? If you haven’t yet decided what your goals are, being able to meet many goals is useful.
The AGI argument is that its goals might not be aligned with ours, are you saying that we should make sure that our future self’s goals be aligned with our current goals?
For example, if I know I am prone to hyperbolic discounting, I should take power from my future self so it will act according to my wishes rather than its own?
Being able to meet many goals is useful. Actually meeting wrong goals is not.
Your hyperbolic discounting example is instructive, as without a model of your goals, you cannot know whether your current or future self is correct. Most people come to the opposite conclusion—a hyperbolic discount massively overweights the short-term in a way that causes regret.
I meant that—when planning for the future, I want my future self to care about each absolute point in time as much as my current self does, or barring that, to only be able to act as if it did, hence the removal of power.
The correct goal is my current goal, obviously. After all, it’s my goal. My future self may disagree, prefering its own current goal. Correct is a two-place word.
If I let my current goal be decided by my future self, but I don’t know yet what it will decide, then I should accomodate as many of its possible choices as possible.