D&D.Sci April 2021 Evaluation and Ruleset
This is a followup to the D&D.Sci post I made last week; if you haven’t already read it, you should do so now before spoiling yourself.
Here is the web interactive I built to let you evaluate your solution; below is an explanation of the rules used to generate the dataset. You’ll probably want to test your answer before reading any further.
Ruleset
(Note: to make writing this easier, I’m using standard D&D dice notation, in which “3+4d8” means “roll four eight-sided dice, sum the results, then add three”.)
Enemies
Sharks
Sharks are 1⁄6 of encounters.
They attack in groups of 2+1d4, each of which does 1d10 points of damage.
Demon Whales
Demon Whales are 1⁄14 of encounters. (If that fraction seems high, you’re failing to account for all the sunk ships that couldn’t report encountering them.)
An attack from a Demon Whale does 17d12 points of damage.
Crabmonsters
Crabmonsters are 1⁄14 of encounters.
A Crabmonster repeatedly rolls 1d80 as it tears through the ship, adding a point of damage with each roll, until it rolls a 1 (that is, encounters someone or something that stops it).
Pirates
Though the Navy’s records don’t bother to distinguish, Pirates come in two categories: Brigands (local criminals who had the poor fortune to cross paths with Naval supply ships while flying the black flag, and/or to mistake them for civilian cargo ships) and Privateers (agents of an enemy government, harassing your Navy’s fleet using hit-and-run tactics). Brigands are 1⁄6 of random encounters during your voyages, Privateers 1⁄21.
A fight with Brigands does 4d8 points of damage; a fight with Privateers does 6d12.
Merpeople
Surface-dwellers are unaware of the intricacies of underwater society, and record both Atlantean Merfolk (1/14 of encounters) and Alexandrian Merfolk (2/21 of encounters) as “Merpeople”. Fortunately, the two city-states are close enough politically that befriending one will cause them both to allow you free passage.
Atlanteans do 20+3d20 damage; Alexandrians do 1d8*1d8*1d8+1d20 damage.
Kraken
Kraken are 2⁄21 of encounters.
They do 12d8 points of damage.
Nessie
Nessie is 1⁄21 of encounters.
She does 40+10d8 points of damage.
Harpies
Harpies are 1⁄14 of encounters.
They do 1d4+1d8+1d12 points of damage.
Water Elementals
Water Elementals are 2⁄21 of encounters.
The Navy has countering the powerful but predictable attacks of Water Elementals down to an art; there are well-known methods for ensuring they only almost destroy a given ship. They do 73+1d12 points of damage.
Direction
Direction is irrelevant from perspectives both practical (you have no control over how many trips you take each way) and epistemic (direction happens to have no effect on outcomes).
Time effects
Time has almost no effect. The one exception is that Privateers used to be much more common (and other encounters therefore slightly less common) before 4/1401; this is when your nation’s main rival changed tactics and stopped hiring mercenaries to attack supply ships.
Sinking Risk by Enemy
In the absence of interventions, ~50% of shipwrecks are caused by Demon Whales, ~18% by Crabmonsters, ~31% by Merpeople, ~1% by Nessie, and 0% by other threats.
Strategy
If attempting to optimize odds of survival, your best choices are to buy all oars, arm carpenters, tribute the Merpeople, and buy one extra cannon; congratulations to simon, GuySrinivasan and Measure for reaching this conclusion.
However, since Pirates never sink ships and Nessie is pretty bad at it, you may wish to take the money you’d spend on the cannon and either hold onto it (to impress the Navy’s accountants) or spend it on foam swords (to impress the Navy’s dockworkers).
Reflections
All else equal, there’s a little extra uncertainty when predicting quantities instead of categories: “is that sudden peak at 14% noise, or a clue to the generating function?”, etc. However, the main reason this challenge was so much more speculative than its predecessors is that the most important information – details of attacks that did 100%+ damage – was censored by the mechanics of the world. In the absence of hard evidence, small errors in inference compound, priors pick up the slack, and considerations like “what genres apply here?” or “is the scenario designer enough of a troll to have Demon Whale damage arbitrarily cap out at 99%?” take on a significance they wouldn’t otherwise.
This is both good and bad. Good because the personal touch adds intrigue to what would otherwise just be data-wrangling; bad because every unit of effort spent psychoanalyzing the GM is a unit of effort not spent on getting better at data-wrangling or on psychoanalyzing reality’s GM (i.e. studying Math and Science). I enthusiastically solicit feedback on this point, as well as on every other point.
Scheduling
The next D&D.Sci challenge should be ready sometime earlyish next month, but nebulous and open-ended work commitments mean I can’t promise anything.
I thought this was less speculative than its predecessors, not more. All of the distributions were reasonable. While there could have been unknown unknowns like another type of water elemental or pirates that either succeed or don’t do much damage, there in fact weren’t. There were known unknowns that were pretty much called out explicitly. This was all signaled by the title which I read as “explicitly not a black swan scenario”. The somewhat hidden second society of merpeople was strongly clued by having a very obvious two-humped pirate distribution. Altogether this felt like a good meld of data science and fair puzzle.
The one “GM modeling” that I think would probably be important for doing well on your subsequent puzzles is “usually the underlying model is decently simple; also usually it involves dice rolls like 10+2d6 or 1d8*1d8″.
AAR:
I modeled crabmonsters as triangular rather than exponential. Two ways I could have gotten this right: (1) GM-modeling; I was pretty sure all of these were dice rolls but forgot that for crabmonsters, or (2) uncertainty hedging; if there was an obvious next encounter after merpeople to extend further than modeled, it was crabmonsters, not e.g. pirates or water elementals.
Demon whales were spot-on. I said 76.4% fatality rate, actual was 78%.
My notes on merpeople were correct, imprecise, and as follows:
Nessie I gave slightly more mean and slightly less variance than I should have, guessing 60+4d12 (86, 6.90^2) when the truth was 40+10d8 (85, 7.25^2). Still, I said 2.39% chance of fatality and actual fatality was “~2%”.
I did much better on this one than the last. :) Relatedly, the primary difficulty at my “how do I get money” work is that when we tell an e-commerce merchant “this particular order? we’re not going to insure this order against fraud, it’s too risky”, they don’t ship it, and we never find out much info in that part of the distribution.
I underestimated the crabmonsters and overestimated the (entirely out-of-sample) possibilities of super-water-elementals and super-pirates. Worst mistake was eyeballing the crabmonsters rather than making a serious attempt to model the distribution, which would have made it clear that they actually do quite a lot of sinking.
All the distributions look pretty much as I expected except the merpeople; while I suspected multiple peaks and/or a long tail I didn’t expect that the lower-damage visible(ish) peak had the long tail.
One thing you didn’t discuss though, is how you generated the number of trips and trip directions. One thing I noticed, but didn’t remark on, is that the variation in total number of trips each month is smaller than the variation in northbound trips and southbound trips separately. That seems unexpected given the scenario, but my main theory is that you simply had some formula for the total trips per month and then randomly assigned each trip a direction?
Your theory is correct. I had some sort of clever justification for why that made sense in-universe but I forget what it was; the Doylist reason was that I wanted the red herring in this puzzle to be as simple as possible.
The original generation code is up here if you want to take a look.
I posted my answer a bit late but this was a ton of fun!
Besides the huge miss that ships were fixed on every stop, I had also missed that merpeople were actually two different distributions. Without looking at my resulting model too much I would guess that this is the reason that my model didn’t account for around 700 of the crashed ships in the data.
With that in mind, I think my simplified model where I didn’t use the number of sunken ships (except through a log-normal prior on total number of encounters per encounter type) when estimating MLE for distribution parameters did fairly well.