There’s an interesting sense in which we all failed this one. Most other players used AI to help them accomplish tasks they’d personally picked out; I eschewed AI altogether and constructed my model with brute force and elbow grease; after reaching a perfect solution, I finally went back and used AI correctly, by describing the problem on a high level (manually/meatbrainedly distilled from my initial observations) and asking the machine demiurge what approach would make most sense[1]. From this I learned about the fascinating concept of Symbolic Regression and some associated python libraries, which I eagerly anticipate using to (attempt to) steamroll similarly-shaped problems.
(There’s a more mundane sense in which I specifically failed this one, since even after building a perfect input-output relation and recognizing the two best archetypes as rebatemaxxing and corpsemaxxing, I still somehow fell at the last hurdle and failed to get a (locally-)optimal corpsemaxxing solution; if the system had followed the original plan, I’d be down a silver coin and up a silver medal. Fortunately for my character’s fortunes and fortune, Fortune chose to smile.)
Reflections on the challenge:
A straightforward scenario, but timed and executed flawlessly. In particular, I found the figuring-things-out gradient (admittedly decoupled from the actually-getting-a-good-answer gradient) blessedly smooth, starting with picking up on the zero-randomness premise[2] and ending with the fun twist that the optimal solution doesn’t involve anything being taxed at the lowest rate[3].
I personally got a lot out of this one: for an evening’s exacting but enjoyable efforts, I learned about an entire new form of model-building, about the utility and limits of modern AI, and about Banker’s Rounding. I vote four-out-of-five for both Quality and Complexity . . . though I recognize that such puzzle-y low-variance games are liable to have higher variance in how they’re received, and I might be towards the upper end of a bell curve here.
For a lark, I also tried turning on all ChatGPT’s free capabilities and telling it to solve the problem from scratch. It thought for ~30 seconds and then spat out a perfect solution; I spent ~30 further seconds with paperclips dancing before my eyes; I then discovered it hadn’t even managed to download the dataset, and was instead applying the not-unreasonable heuristic “if abstractapplic and simon agree on an answer it’s probably true”.
There’s something fun about how “magic”, “games”, “bureaucracy”, and “magical game bureaucracy” are equally good justifications for a “wait, what paradigm am I even in here?” layer of difficulty.
Looked into it more and you’re right: conventional symbolic regression libraries don’t seem to have the “calculate a quantity then use that as a new variable going forward” behavior I’d have needed to get Total Value and then decide&apply a tax rate based on that. I . . . probably should have coded up a proof-of-concept before impugning everyone including myself.
Interesting link on symbolic regression. I actually tried to get an AI to write me something similar a while back[1] (not knowing that the concept was out there and foolishly not asking, though in retrospect it obviously would be).
From your response to kave:
calculate a quantity then use that as a new variable going forward
In terms of the tree structure used in symbolic regression (including my own attempt), I would characterize this as wanting to preserve a subtree and letting the rest of the tree vary.
Possible issues:
If the coding modifies the trees leaf-first, trees with different roots but common subtrees aren’t treated as close to each other. This is an issue that my own version would likely have had even if actually implemented[2]. However, I think PySR might at least partially address this issue (It uses genetic programming and the pictures in the associated paper seem to indicate that it is generating trees which at least sometimes preserve subtrees.) (Though the genetic programming approach is likely to make it hard to find the very simplest solutions in practice imo.[3])
Even if you are treating trees with common subtrees as close to each other, if your evaluation of trees is only comparing final calculated values on the entire dataset, then it’s hard to make the call “I know this subtree is important even if I don’t know the rest of the tree” because the results are not likely to be all that close unless you already have a reasonable guess for the rest of the tree. One partial (heh) answer might be to award part marks to solutions that work well for some of the data even if wildly off for other parts. Careful thinking might be required to do this in a way that doesn’t backfire horribly, though. Hmm—or maybe you CAN do that in the existing paradigm by including if/then nodes in the tree? Say, a node that has three child nodes/subtrees, and chooses between two of them based on the value of the third? And then (in some genetic-programming-like approach perhaps) explore what happens if you copy those subtrees elsewhere, or existing subtrees into new if-then nodes?) (I can imagine the horrific unreadable mess already though...)
edited to add: it might be more appropriate to say that I had been planning on asking an AI to code something, but the initial prototype was sufficiently lame and gave me enough insight into the difficulties ahead I didn’t continue. Claude chat link if anyone’s interested.
edited to further add: hmm, what you are wanting (“new variable”) is probably not just preserving a subtree, but for the mutation system to be able to copy that subtree to other parts of the tree (and the complexity calculator to not give to much penalty to that, I guess). Interestingly, it seems that PySR’s backend at least (SymbolicRegression.jl) does have the capability to do this already, using a “form_random_connection!” mutation function that apparently allows the same subtree to appear as child of multiple parents, making a DAG instead of a tree. In general, I’ve been pretty impressed looking at SymbolicRegression.jl. Maybe other symbolic regression software is as feature-rich, but haven’t checked.
This is informed by (a tiny amount of) practical experience. After SarahNibs’ comment suggested genetic programming would have worked on the “Arena of Data”, I attempted genetic programming on it and on my initial attempt got … a horrific unreadable mess. Maybe it wasn’t “halfway decently regularized” but I updated my intuition to say: complicated ways to do things so greatly outnumber the simple ways that anything too reliant on randomness is not likely to find the simple way.
Reflections on my performance:
There’s an interesting sense in which we all failed this one. Most other players used AI to help them accomplish tasks they’d personally picked out; I eschewed AI altogether and constructed my model with brute force and elbow grease; after reaching a perfect solution, I finally went back and used AIcorrectly, by describing the problem on a high level (manually/meatbrainedly distilled from my initial observations) and asking the machine demiurge what approach would make most sense[1]. From this I learned about the fascinating concept ofSymbolic Regressionand some associated python libraries, which I eagerly anticipate using to (attempt to) steamroll similarly-shaped problems.(There’s a more mundane sense in which I specifically failed this one, since even after building a perfect input-output relation and recognizing the two best archetypes as rebatemaxxing and corpsemaxxing, I still somehow fell at the last hurdle and failed to get a (locally-)optimal corpsemaxxing solution; if the system had followed the original plan, I’d be down a silver coin and up a silver medal. Fortunately for my character’s fortunes and fortune, Fortune chose to smile.)
Reflections on the challenge:
A straightforward scenario, but timed and executed flawlessly. In particular, I found the figuring-things-out gradient (admittedly decoupled from the actually-getting-a-good-answer gradient) blessedly smooth, starting with picking up on the zero-randomness premise[2] and ending with the fun twist that the optimal solution doesn’t involve anything being taxed at the lowest rate[3].
I personally got a lot out of this one: for an evening’s exacting but enjoyable efforts, I learned about an entire new form of model-building, about the utility and limits of modern AI, and about Banker’s Rounding. I vote four-out-of-five for both Quality and Complexity . . . though I recognize that such puzzle-y low-variance games are liable to have higher variance in how they’re received, and I might be towards the upper end of a bell curve here.
For a lark, I also tried turning on all ChatGPT’s free capabilities and telling it to solve the problem from scratch. It thought for ~30 seconds and then spat out a perfect solution; I spent ~30 further seconds with paperclips dancing before my eyes; I then discovered it hadn’t even managed to download the dataset, and was instead applying the not-unreasonable heuristic “if abstractapplic and simon agree on an answer it’s probably true”.
There’s something fun about how “magic”, “games”, “bureaucracy”, and “magical game bureaucracy” are equally good justifications for a “wait, what paradigm am I even in here?” layer of difficulty.
I know that part wasn’t intentional, but I think rebatemaxxing>corpsemaxxing is nontrivially more compelling than the other way round.
I had terrible luck with symbolic regression, for what its worth.
Looked into it more and you’re right: conventional symbolic regression libraries don’t seem to have the “calculate a quantity then use that as a new variable going forward” behavior I’d have needed to get Total Value and then decide&apply a tax rate based on that. I . . . probably should have coded up a proof-of-concept before impugning everyone including myself.
Interesting link on symbolic regression. I actually tried to get an AI to write me something similar a while back[1] (not knowing that the concept was out there and foolishly not asking, though in retrospect it obviously would be).
From your response to kave:
In terms of the tree structure used in symbolic regression (including my own attempt), I would characterize this as wanting to preserve a subtree and letting the rest of the tree vary.
Possible issues:
If the coding modifies the trees leaf-first, trees with different roots but common subtrees aren’t treated as close to each other. This is an issue that my own version would likely have had even if actually implemented[2]. However, I think PySR might at least partially address this issue (It uses genetic programming and the pictures in the associated paper seem to indicate that it is generating trees which at least sometimes preserve subtrees.) (Though the genetic programming approach is likely to make it hard to find the very simplest solutions in practice imo.[3])
Even if you are treating trees with common subtrees as close to each other, if your evaluation of trees is only comparing final calculated values on the entire dataset, then it’s hard to make the call “I know this subtree is important even if I don’t know the rest of the tree” because the results are not likely to be all that close unless you already have a reasonable guess for the rest of the tree. One partial (heh) answer might be to award part marks to solutions that work well for some of the data even if wildly off for other parts. Careful thinking might be required to do this in a way that doesn’t backfire horribly, though. Hmm—or maybe you CAN do that in the existing paradigm by including if/then nodes in the tree? Say, a node that has three child nodes/subtrees, and chooses between two of them based on the value of the third? And then (in some genetic-programming-like approach perhaps) explore what happens if you copy those subtrees elsewhere, or existing subtrees into new if-then nodes?) (I can imagine the horrific unreadable mess already though...)
edited to add: it might be more appropriate to say that I had been planning on asking an AI to code something, but the initial prototype was sufficiently lame and gave me enough insight into the difficulties ahead I didn’t continue. Claude chat link if anyone’s interested.
edited to further add: hmm, what you are wanting (“new variable”) is probably not just preserving a subtree, but for the mutation system to be able to copy that subtree to other parts of the tree (and the complexity calculator to not give to much penalty to that, I guess). Interestingly, it seems that PySR’s backend at least (SymbolicRegression.jl) does have the capability to do this already, using a “form_random_connection!” mutation function that apparently allows the same subtree to appear as child of multiple parents, making a DAG instead of a tree. In general, I’ve been pretty impressed looking at SymbolicRegression.jl. Maybe other symbolic regression software is as feature-rich, but haven’t checked.
Apparently November 2024. Feels longer ago somehow.
I hadn’t actually gone beyond breadth-first search though.
This is informed by (a tiny amount of) practical experience. After SarahNibs’ comment suggested genetic programming would have worked on the “Arena of Data”, I attempted genetic programming on it and on my initial attempt got … a horrific unreadable mess. Maybe it wasn’t “halfway decently regularized” but I updated my intuition to say: complicated ways to do things so greatly outnumber the simple ways that anything too reliant on randomness is not likely to find the simple way.