Reading this thread, I wonder if the apparent disagreement doesn’t come from the use of the world “honestly”. The way I understand Paul’s statement of the problem is that “answer questions honestly” could be replaced by “answer questions appropriately to the best of your knowledge”. And his point is that “answer what a human would have answered” is not a good proxy for that (yet still an incentivized one due to how we train neural nets)
From my reading of it, this post’s proposal does provide some plausible ways to incentivize the model to actual search for appropriate answers instead of the ones human would have given, and I don’t think it assumes the existence of true categories and/or essences.
Reading this thread, I wonder if the apparent disagreement doesn’t come from the use of the world “honestly”. The way I understand Paul’s statement of the problem is that “answer questions honestly” could be replaced by “answer questions appropriately to the best of your knowledge”. And his point is that “answer what a human would have answered” is not a good proxy for that (yet still an incentivized one due to how we train neural nets)
From my reading of it, this post’s proposal does provide some plausible ways to incentivize the model to actual search for appropriate answers instead of the ones human would have given, and I don’t think it assumes the existence of true categories and/or essences.