the rate of increase in the AGI’s intelligence when it is used to solve tasks
ChatGPT doesn’t even remember conversations, so taken literally, the value of this quantity is “zero”.
However, by being a user, you potentially or actually provide OpenAI with feedback that will help them improve their product, so in that sense, there can be a nonzero relationship.
If you’re going to write equations like these, you may as well as model the AI’s “level of alignment” as well as its intelligence. The AI is an extinction risk, only if it is unaligned when it reaches superintelligence. So you should model the effect of user choices on AI alignment as well.
I agree on both counts. You’re right that I should model the alignment of the system as well as its intelligence. I guess the alignment could be thought of as minimizing the distance of high dimensional vectors representing the players’ and the AI’s values. So each user (and the AI, too) could have a value vector associated with it, and the cost functions of the user could then incorporate how much they care about their own alignment (to the rest of the users), and the cost function of the AI needs to be tuned so that it is enough aligned when it reaches a critical threshold of intelligence. That way, you could express how important it is that the AI is aligned, as a function of its intelligence.
ChatGPT doesn’t even remember conversations, so taken literally, the value of this quantity is “zero”.
However, by being a user, you potentially or actually provide OpenAI with feedback that will help them improve their product, so in that sense, there can be a nonzero relationship.
If you’re going to write equations like these, you may as well as model the AI’s “level of alignment” as well as its intelligence. The AI is an extinction risk, only if it is unaligned when it reaches superintelligence. So you should model the effect of user choices on AI alignment as well.
I agree on both counts. You’re right that I should model the alignment of the system as well as its intelligence. I guess the alignment could be thought of as minimizing the distance of high dimensional vectors representing the players’ and the AI’s values. So each user (and the AI, too) could have a value vector associated with it, and the cost functions of the user could then incorporate how much they care about their own alignment (to the rest of the users), and the cost function of the AI needs to be tuned so that it is enough aligned when it reaches a critical threshold of intelligence. That way, you could express how important it is that the AI is aligned, as a function of its intelligence.