Participating in a game like this sounds like a lot of fun. I want to do this too. But I am not willing to move to Chicago and start studying history for it. :D
Do you know of things that are similar where I can join?
Actually, I wonder if there is a way of combining this with the idea of experimental history you described above. You wrote that LLMs and historians have the problem that they already know what happens in 1492. Well, I don’t. Maybe one could use LLMs only for creating the starting point, the character material, for the evaluation of the results and maybe for playing NPCs, and then let random strangers on the internet play the characters. Of course one would need to deal with trolls and cheaters (e.g. falsely claiming to not know the scenario), but this could be done with people getting something like a trustworthiness score after each game and then one would only use those games for studying, that are played by people who already played a number of games before and have a high trustworthiness.
I could imagine that I am not the only person who would be excited to participate in a study like this. This probably scales worse than the pure LLM version, but maybe better than the pure human version of Ada Palmer.
Source for the “meme from LinkedIn”: https://x.com/waitbutwhy/status/1367871165319049221