Thanks for the feedback. I agree on both points; I will probably change the wording to what you suggested.
tetronian2
Karma: 306
This is a followup to a post I made in the open thread last week; apologies if this comes off as spammy. I will be running a program equilibrium iterated prisoner’s dilemma tournament (inspired by the one last year). There are a few key differences from last year’s tournament: First, the tournament is in Haskell rather than Scheme. Second, the penalty for bots that do not finish their computation within the pre-set time limit has been reduced. Third, bots have the ability to run/simulate each other but cannot directly view each other’s source code.
Here are the rules and a brief tutorial (which are significantly more fleshed out than they were last week). I intend to open up the tournament and announce it via a discussion post in a few days, but until then, I would love to hear your feedback and suggested changes to the rules/implementation, no matter how major or minor. When the tournament opens, LW users with 50+ karma who do not know Haskell can PM me with an algorithm/psuedocode, and I will translate it into a working bot for them.
Thank you! These changes both make sense; I will adjust the tournament structure as you described and enter your bot when the tournament is open.
Can you explain the rationale behind the elimination setup a little more? An elimination tournament seems less fair than pure round-robin. Moreover, I ran some tests with various combinations of the example bots from the tutorial, and what generally happens is the bots with strategies along the lines of “I’ll cooperate if you do” (i.e. tit-for-tat, justiceBot, mirrorBot) rise to the top and then just cooperate with each other, resulting in a multi-way tie. If the actual pool of submissions contain enough bots with that kind of strategy, a large tie seems inevitable. This result doesn’t sound very exciting for a competition, but is there is some sense in which it is theoretically “better” than round-robin?
Edit: This outcome doesn’t always happen, but it happens most of the time.
Thank you, that is an excellent explanation and you have changed my mind; I will implement an elimination tournament as you described.
Announcing the 2014 program equilibrium iterated PD tournament
Excellent questions. Here are the rules pertaining to the tournament structure (from the readme):
One submission per person.
The tournament will be round-robin elimination: Each bot will play one match against all other bots, where a match consists of 100 rounds of the prisoner’s dilemma. At the end of the round-robin round, the lower-scoring half of the tournament pool will be eliminated. This process will be repeated until only one bot remains, or there is a tie. The whole tournament will be run 1000 times, and the bot that places first most frequently will be declated the overall winner.
In other words, bots do not play against themselves, but they will play against all other bots at least once per tournament iteration. Let me know if this is unclear. (This setup was suggested to me by James Miller.)
I agree. I chose Haskell because I was intrigued by how easy it was to restrict what bots are able to do, using only Haskell’s type system. I would actually love it if a large number of non-Haskellers participated—I enjoy coding Haskell, and I don’t mind putting aside some time to help if the result is a larger, more interesting tournament.
Exactly right.
I figured that this game is sufficiently different from the classical PD that the same Nash equilibrium/style of thinking isn’t going to apply, and I was interested to see what participants can do with the fixed-size rounds. So it’s just an arbitrary experimental parameter that I’m curious about.
Unlike last year’s tournament, you can’t, and this is part of the challenge. Simulations are a precious resource, and you have to make an educated guess about your opponent based on the limited simulations you have.
Good points. Simulations do give much less information than visible source code—this was a deliberate design choice on my part, but you are probably right that this kind of setup is not really “program equilibrium.” I admit, I don’t have a very good understanding of where the limits of the PD game really are, and I hadn’t considered the problem of the all-players tie (fortunately I don’t think it is going to happen in practice). Mutual cooperation and cliques are still viable strategies here, but they may not be the Nash equilibrium anymore. Regardless, I agree that this tournament is horribly misnamed.
Thank you for submitting! Yes, I am accepting one bot per person.
Whoops, fixed. And thanks, I’m curious to see what happens with this ruleset.
Thank you very much for catching this and for explaining it in detail here and on the github page; I’m going to play around with your proposed solution today and brainstorm any other ways of dealing with this.
While this is true, there can be a distinction between a character with flaws and a character who is extremely irritating to read about. And this is one of those judgement calls where The Audience is Always Right; it seems very reasonable to stop reading a story if the protagonist noticeably irritates you.
In general, commentary to the effect of “you should like this thing” is not very useful, especially if you are trying to figure out why someone reacted negatively.
(These discussions in which one group has an overwhelmingly strong “squick” or “ew” reaction and another group does not are fascinating to me, not least of all because they seem to pop up quite frequently here, e.g. about Eliezer’s OKCupid profile and NYC cuddle piles. Both sides spew huge amounts of ink explaining their emotional reactions, and yet there never seems to be any actual sharing of understanding. In the interests of trying harder...I was also very aggravated by the first few chapters of HPMOR, and would be happy to discuss it calmly here.)
Yes, I did find that section grating. I’m describing my emotions post-hoc here (which is not generally reliable), but what I found irritating about the first few chapters was the fact that Harry acts in an extremely arrogant way, nearly to the point of coercing the adult characters, and the story-universe appears to back him up at every turn. This is the “Atlas Shrugged” effect described downthread by CellBioGuy. Harry is probably right in most of those arguments, but he is only so effortlessly correct and competent because the story-universe is designed so that he can teach these rationality lessons to other characters. It feels like the world is unfair, and unfair in the favor of an unlikeable and arrogant character. There is a real-world corollary of this, of course—very arrogant people who always get what they want—and I suspect my emotional reactions to these real and fictional situations are very similar.
(I have since caught up with the rest of the fic, and enjoyed most of it.)
The correct deadline is Sept. 15th—I extended it by 15 days because of an error in the tournament code that went unfixed for two weeks, which delayed some people from writing their bots. I’ve updated the post to reflect this, thanks for catching it. Anyone who already submitted a bot is welcome to re-submit if they want to make changes to their bot before the 15th.
Has anyone else seen the television show Brain Games? It is essentially intro-to-cognitive-biases aimed at the level of the average TV watcher; I was pleasantly surprised by how well it explains some basic biases with simple examples (though I have only seen an assortment of episodes from the 3rd and 4th season). However, most of the material given is not very actionable and is designed more for entertainment rather than self-improvement. Nevertheless, those interested in raising the sanity waterline and/or sparking interest in LW subjects among more average folk than we are might want to take a look at it.
Sometime in the near future, I will be running an iterated prisoner’s dilemma tournament in which bots can access their opponents’ source code, similar to the IPD tournament that was held last year. This tournament will be open to the Internet at large (i.e. not just LW) and will probably include some Hacker News folks and some folks from my real-life social network, who are primarily programmers and people in the finance world. Once everything is officially announced, there will be a large window (a month?) in which users can submit entries before the tournament is run. Also, to help out non-programmer participants, I will be translating some participants’ pseudocode/descriptions of algorithms into code. More details on this later.
The (work in progress) code that will be used to run the tournament is here.
Right now everything is still in the preliminary stages, so I would appreciate:
comments about what made last year’s tournament good/bad/etc.
suggestions for the rules and payoff matrix/feature requests to make the tournament more interesting
code review
general comments (i.e., yell at me)