Seems like that only makes sense because you specified that “increasing production capacity” and “upgrading machines” are the things that I’m not allowed to do, and those are things I have a conceptual grasp on. And even then—am I allowed to repair machines that break? What about buying a new factory? What if I force workers to work longer hours? What if I create effective propaganda that causes other people to give you paperclips? What if I figure out that by using a different source of steel I can reduce the defect rate? I am legitimately conceptually uncertain whether these things count as “increasing production capacity / upgrading machines”.
As another example, what does it mean to optimize for “curing cancer” without becoming more able to optimize for “curing cancer”?
Sorry, forgot to reply. I think these are good questions, and I continue to have intuitions that there’s something here, but I want to talk about these points more fully in a later post. Or, think about it more and then explain why I agree with you.
Seems like that only makes sense because you specified that “increasing production capacity” and “upgrading machines” are the things that I’m not allowed to do, and those are things I have a conceptual grasp on. And even then—am I allowed to repair machines that break? What about buying a new factory? What if I force workers to work longer hours? What if I create effective propaganda that causes other people to give you paperclips? What if I figure out that by using a different source of steel I can reduce the defect rate? I am legitimately conceptually uncertain whether these things count as “increasing production capacity / upgrading machines”.
As another example, what does it mean to optimize for “curing cancer” without becoming more able to optimize for “curing cancer”?
Sorry, forgot to reply. I think these are good questions, and I continue to have intuitions that there’s something here, but I want to talk about these points more fully in a later post. Or, think about it more and then explain why I agree with you.