The relevant tradeoff to consider is the cost of prediction and the cost of influence. As long as the cost of predicting an “impressive output” is much lower than the cost of influencing the world such that an easy-to-generate output is considered impressive, then it’s possible to generate the impressive output without risking misalignment by bounding optimization power at lower than the power required to influence the world.
So you can expect an impressive AI that predicts the weather but isn’t allowed to, e.g., participate in prediction markets on the weather nor charter flights to seed clouds to cause rain, without needing to worry about alignment. But don’t expect alignment-irrelevance from a bot aimed at writing persuasive philosophical essays, nor an AI aimed at predicting the behavior of the stock market conditional on the trades it tells you to make, nor an AI aimed at predicting the best time to show you an ad for the AI’s highest-paying company.
The relevant tradeoff to consider is the cost of prediction and the cost of influence. As long as the cost of predicting an “impressive output” is much lower than the cost of influencing the world such that an easy-to-generate output is considered impressive, then it’s possible to generate the impressive output without risking misalignment by bounding optimization power at lower than the power required to influence the world.
So you can expect an impressive AI that predicts the weather but isn’t allowed to, e.g., participate in prediction markets on the weather nor charter flights to seed clouds to cause rain, without needing to worry about alignment. But don’t expect alignment-irrelevance from a bot aimed at writing persuasive philosophical essays, nor an AI aimed at predicting the behavior of the stock market conditional on the trades it tells you to make, nor an AI aimed at predicting the best time to show you an ad for the AI’s highest-paying company.