Interesting. I’ve never really had a great grasp on decision theory and also don’t have a great grasp on why exactly decision theory is analogous to low-level physics or Bayes’ theorem, but at the same time, I sorta see it. And to that extent, I feel like it has moved my understanding of decision theory up a rung, which I appreciate.
Here’s where I’m at with my understanding of decision theory. As a starting point, you can say that it makes sense to maximize expected utility, with utility basically referring to “what you care about”. But then there’s weird situations like Newcomb’s problem where “maximize expected utility” arguably doesn’t lead to the outcomes you want, and so, arguably, you’d want to use a different decision theory in those situations. It’s always seemed to me like those situations are contrived though and don’t actually occur in real life. Which doesn’t mean decision theory is useless (from what I understand it’s super important for AI alignment) or not intellectually interesting, just that it seems (very) rare that it’d be useful.
Interesting. I’ve never really had a great grasp on decision theory and also don’t have a great grasp on why exactly decision theory is analogous to low-level physics or Bayes’ theorem, but at the same time, I sorta see it. And to that extent, I feel like it has moved my understanding of decision theory up a rung, which I appreciate.
Here’s where I’m at with my understanding of decision theory. As a starting point, you can say that it makes sense to maximize expected utility, with utility basically referring to “what you care about”. But then there’s weird situations like Newcomb’s problem where “maximize expected utility” arguably doesn’t lead to the outcomes you want, and so, arguably, you’d want to use a different decision theory in those situations. It’s always seemed to me like those situations are contrived though and don’t actually occur in real life. Which doesn’t mean decision theory is useless (from what I understand it’s super important for AI alignment) or not intellectually interesting, just that it seems (very) rare that it’d be useful.