To clarify a bit, here is one of many examples that shows token prediction resulting in self-contradictory replies, apparently because the meaning of its response is not something ChatGPT can model well:
Q: What country is south of Rwanda?
A: Rwanda is a landlocked country located in central Africa. It is bordered by Uganda to the north, Tanzania to the east, Burundi to the south, and the Democratic Republic of the Congo to the west. There are no countries south of Rwanda.
To clarify a bit, here is one of many examples that shows token prediction resulting in self-contradictory replies, apparently because the meaning of its response is not something ChatGPT can model well:
Q: What country is south of Rwanda?
A: Rwanda is a landlocked country located in central Africa. It is bordered by Uganda to the north, Tanzania to the east, Burundi to the south, and the Democratic Republic of the Congo to the west. There are no countries south of Rwanda.
((The question is from the paper https://arxiv.org/pdf/2212.03551.pdf, mentioned by Sarah Constantin in https://twitter.com/s_r_constantin/status/1603809422278418433 .)