It looks like I was basically right. I could have done slightly better by looking more closely at interactions between features, ore types especially; still, I (comfortably) survived and (barely) proved my point to the King, so I’m happy with the outcome I got.
(I’m also very pleased by the fact that I picked up on the ore-based-vs-wood-based distinction; or, rather, that the ML library I’ve been building automatically picked up on it. Looks like my homebaked interpretability tools work outside their usual contexts!)
Reflections on the challenge:
Another excellent entry, and a hard act to follow. The jokes landed, the premise was fun but coherent, and the scenario was challenging yet tractable.
Having multiple quantitative success metrics was a fascinating choice. To be honest, I think there was some missed potential here; if the best strategy wasn’t simultaneously survival-optimal and money-optimal, there could have been some interesting tension from players deciding their blood-to-treasure exchange rates. I’ll have to try and work something like that into a future game.
I’m curious as to what exactly you found there. Ore-based vs wood-based production wasn’t really an intended distinction—rather, ore and wood were intended to be used together. I added woodcrafting as an afterthought late in development (when Crafters were performing very poorly due to only having two craftable ores), and it still isn’t a major source of income. Even in your SHAMEFULLY ELFISH fort, your Crafters spend most of their time on Silver, and only make things out of wood when poor mining yield/alcohol-fueled crafting frenzies make them run out of silver. The intended distinctions were:
Wood-based vs Coal-based fuel—you need one or the other, with Wood becoming more important the less coal you have.
Precious vs nonprecious metals—you need Crafters to work gold and silver, but Smiths to work iron/bronze/etc.
Having multiple quantitative success metrics was a fascinating choice. To be honest, I think there was some missed potential here; if the best strategy wasn’t simultaneously survival-optimal and money-optimal, there could have been some interesting tension from players deciding their blood-to-treasure exchange rates. I’ll have to try and work something like that into a future game.
There were two goals on my end from this, one of which succeeded and one of which did not:
Providing multiple levels of success.
The aim was to make ‘stay alive’ a relatively easy goal that most players could accomplish with a little work.
‘Maximize value’ was meant to be a harder goal that required more effort.
I’m reasonably happy with how this went—most players hit 100% survival, and everyone found some effects (e.g. Farmers).
Providing a relatively clean environment where effects stood out, as a window into deeper mechanics that players could use to help understand the world-model. This is a bit fuzzy, and didn’t actually end up happening, so I’ll flail at a couple vague examples of what I mean and hope I can convey it:
Several people noticed that there were two ways a fort could die (‘starving’ and ‘digging too deep’.) Some people noticed that Digging Too Deep happened a lot with 7 miners, fairly often wih 6, and rarely with 5 (I think simon was the most explicit here).
A thing that could have been noticed but wasn’t was that all of the Digging Too Deep forts had either 7+ miners or 5-6 miners plus at least one Brewer. No fort with e.g. 6 Miners and no Brewer ever Dug Too Deep.
If players had noticed that, I think it could have been a very powerful window into game mechanics: it could have pointed very strongly both to ‘brewers help other dwarves work harder’ and to ‘hitting explicit thresholds of # Miners is important’. The 100% nature of ‘survive/die’, and the complete lack of any way a fort could die other than the two above, made the data for this clearer than anywhere else in the dataset.
Similarly, Warriors were important for exterior access. This had a clear and visible effect on survival via Farmers: again, simon gave the most detail here, successfully realizing that 3 Farmers/Brewers were needed to feed your fort, but 2 could work so long as at least 1 was a Farmer and you had 2 or more Warriors.
This also affected Woodcutters—Woodcutters are a very important role, but looked weak in the dataset because of their need for exterior access. However, the effect of Warriors on Woodcutters only affected Fort Value, and this effect was mixed up with the ability to buy wood, the need for ore & crafting professions as well for wood to have an effect on value, different values of different ores, adamantine screwing with values, etc.
I was hoping that the more visible Warrior-Farmer effect could help provide insight into ‘warriors help you do stuff outside’ - doing similar survival analysis in other biomes could have revealed that 1 Warrior was sufficient in Plains, 2 in Forest or Tundra, and lots in Jungle, and poking into the interaction of ‘enough warriors’ with other professions could have explained Woodcutters.
Briefly: I told my learner “assume there are two sources of income for Light Forest forts; assume they are log-linked functions of the data provided with no interactions between features; characterize these income sources.”
The output graphs, properly interpreted, said back:
The larger source of income benefits greatly from Miners, benefits from the presence of every ore (especially Haematite), likes coal, and benefits from having one Smith.
The smaller source of income benefits from Woodcutters, benefits from having two (but not more) Warriors, hard-requires at least one Woodcutter and Warrior in order to be viable, actively dislikes Coal, doesn’t care about ores (except Copper for some reason), and strongly benefits from Crafters.
(In reviewing my graphs in retrospect I also see a small bump in performance for both sources associated with having exactly one Brewer; I missed that the first time because it looked like noise and I’d assumed Brewers only mattered to the survival half of the challenge.)
This wasn’t 100% right, and missed some important detail, but given the bad assumptions I built it on—an additive model with a lot of interactions sprinkled on top would have been a better match—I’m pleasantly surprised by how closely it matches (a valid interpretation of) ground truth.
The larger source of income benefits greatly from Miners, benefits from the presence of every ore (especially Haematite), likes coal, and benefits from having one Smith.
The smaller source of income benefits from Woodcutters, benefits from having two (but not more) Warriors, hard-requires at least one Woodcutter and Warrior in order to be viable, actively dislikes Coal, doesn’t care about ores (except Copper for some reason), and strongly benefits from Crafters.
Ah, I see! That is a meaningful interpretation of reality, but rather than ‘ore-based vs wood-based’ I’d phrase it as a distinction between:
Staying inside and mining. Benefits from all ores, and miners. Makes only a few finished goods (smelting only with coal) but still benefits from higher coal level and one or two dwarves to smelt.
Also getting outside and getting fuel. Needs warriors to get you outside, benefits a lot from woodcutters as well, smelts whatever ores are available and crafts wood if it’s left over.
Reflections on my attempt:
It looks like I was basically right. I could have done slightly better by looking more closely at interactions between features, ore types especially; still, I (comfortably) survived and (barely) proved my point to the King, so I’m happy with the outcome I got.
(I’m also very pleased by the fact that I picked up on the ore-based-vs-wood-based distinction; or, rather, that the ML library I’ve been building automatically picked up on it. Looks like my homebaked interpretability tools work outside their usual contexts!)
Reflections on the challenge:
Another excellent entry, and a hard act to follow. The jokes landed, the premise was fun but coherent, and the scenario was challenging yet tractable.
Having multiple quantitative success metrics was a fascinating choice. To be honest, I think there was some missed potential here; if the best strategy wasn’t simultaneously survival-optimal and money-optimal, there could have been some interesting tension from players deciding their blood-to-treasure exchange rates. I’ll have to try and work something like that into a future game.
I’m curious as to what exactly you found there. Ore-based vs wood-based production wasn’t really an intended distinction—rather, ore and wood were intended to be used together. I added woodcrafting as an afterthought late in development (when Crafters were performing very poorly due to only having two craftable ores), and it still isn’t a major source of income. Even in your SHAMEFULLY ELFISH fort, your Crafters spend most of their time on Silver, and only make things out of wood when poor mining yield/alcohol-fueled crafting frenzies make them run out of silver. The intended distinctions were:
Wood-based vs Coal-based fuel—you need one or the other, with Wood becoming more important the less coal you have.
Precious vs nonprecious metals—you need Crafters to work gold and silver, but Smiths to work iron/bronze/etc.
There were two goals on my end from this, one of which succeeded and one of which did not:
Providing multiple levels of success.
The aim was to make ‘stay alive’ a relatively easy goal that most players could accomplish with a little work.
‘Maximize value’ was meant to be a harder goal that required more effort.
I’m reasonably happy with how this went—most players hit 100% survival, and everyone found some effects (e.g. Farmers).
Providing a relatively clean environment where effects stood out, as a window into deeper mechanics that players could use to help understand the world-model. This is a bit fuzzy, and didn’t actually end up happening, so I’ll flail at a couple vague examples of what I mean and hope I can convey it:
Several people noticed that there were two ways a fort could die (‘starving’ and ‘digging too deep’.) Some people noticed that Digging Too Deep happened a lot with 7 miners, fairly often wih 6, and rarely with 5 (I think simon was the most explicit here).
A thing that could have been noticed but wasn’t was that all of the Digging Too Deep forts had either 7+ miners or 5-6 miners plus at least one Brewer. No fort with e.g. 6 Miners and no Brewer ever Dug Too Deep.
If players had noticed that, I think it could have been a very powerful window into game mechanics: it could have pointed very strongly both to ‘brewers help other dwarves work harder’ and to ‘hitting explicit thresholds of # Miners is important’. The 100% nature of ‘survive/die’, and the complete lack of any way a fort could die other than the two above, made the data for this clearer than anywhere else in the dataset.
Similarly, Warriors were important for exterior access. This had a clear and visible effect on survival via Farmers: again, simon gave the most detail here, successfully realizing that 3 Farmers/Brewers were needed to feed your fort, but 2 could work so long as at least 1 was a Farmer and you had 2 or more Warriors.
This also affected Woodcutters—Woodcutters are a very important role, but looked weak in the dataset because of their need for exterior access. However, the effect of Warriors on Woodcutters only affected Fort Value, and this effect was mixed up with the ability to buy wood, the need for ore & crafting professions as well for wood to have an effect on value, different values of different ores, adamantine screwing with values, etc.
I was hoping that the more visible Warrior-Farmer effect could help provide insight into ‘warriors help you do stuff outside’ - doing similar survival analysis in other biomes could have revealed that 1 Warrior was sufficient in Plains, 2 in Forest or Tundra, and lots in Jungle, and poking into the interaction of ‘enough warriors’ with other professions could have explained Woodcutters.
Briefly: I told my learner “assume there are two sources of income for Light Forest forts; assume they are log-linked functions of the data provided with no interactions between features; characterize these income sources.”
The output graphs, properly interpreted, said back:
The larger source of income benefits greatly from Miners, benefits from the presence of every ore (especially Haematite), likes coal, and benefits from having one Smith.
The smaller source of income benefits from Woodcutters, benefits from having two (but not more) Warriors, hard-requires at least one Woodcutter and Warrior in order to be viable, actively dislikes Coal, doesn’t care about ores (except Copper for some reason), and strongly benefits from Crafters.
(In reviewing my graphs in retrospect I also see a small bump in performance for both sources associated with having exactly one Brewer; I missed that the first time because it looked like noise and I’d assumed Brewers only mattered to the survival half of the challenge.)
This wasn’t 100% right, and missed some important detail, but given the bad assumptions I built it on—an additive model with a lot of interactions sprinkled on top would have been a better match—I’m pleasantly surprised by how closely it matches (a valid interpretation of) ground truth.
Ah, I see! That is a meaningful interpretation of reality, but rather than ‘ore-based vs wood-based’ I’d phrase it as a distinction between:
Staying inside and mining. Benefits from all ores, and miners. Makes only a few finished goods (smelting only with coal) but still benefits from higher coal level and one or two dwarves to smelt.
Also getting outside and getting fuel. Needs warriors to get you outside, benefits a lot from woodcutters as well, smelts whatever ores are available and crafts wood if it’s left over.