Arjun Panickssery(ModusTrollens)

• Arjun Panickssery 25 Jul 2024 21:38 UTC

  3 points

  1

  on: Ar­jun Pan­ickssery’s Shortform

  What’s the actual probability of casting a decisive vote in a presidential election (by state)?

  I remember the Gelman/​Silver/​Edlin “What is the probability your vote will make a difference?” (2012) methodology:

  1. Let E be the number of electoral votes in your state. We estimate the probability that these are necessary for an electoral college win by computing the proportion of the 10,000 simulations for which the electoral vote margin based on all the other states is less than E, plus ¹⁄₂ the proportion of simulations for which the margin based on all other states equals E. (This last part assumes implicitly that we have no idea who would win in the event of an electoral vote tie.) [Footnote: We ignored the splitting of Nebraska’s and Maine’s electoral votes, which retrospectively turned out to be a mistake in 2008, when Obama won an electoral vote from one of Nebraska’s districts.]

  2. We estimate the probability that your vote is decisive, if your state’s electoral votes are necessary, by working with the subset of the 10,000 simulations for which the electoral vote margin based on all the other states is less than or equal to E. We compute the mean M and standard deviation S of the vote margin among that subset of simulations and then compute the probability of an exact tie as the density at 0 of the Student-t distribution with 4 degrees of freedom (df), mean M, and scale S.

  The product of two probabilities above gives the probability of a decisive vote in the state.

  This gives the following results for the 2008 presidential election, where they estimate that you had less than one chance in a hundred billion of deciding the election in DC, but better than a one in ten million chance in New Mexico. (For reference, 131 million people voted in the election.)

  Is this basically correct?

  (I guess you also have to adjust for your confidence that you are voting for the better candidate. Maybe if you think you’re outside the top ~20% in “voting skill”—ability to pick the best candidate—you should abstain. See also.)

An­a­lyz­ing Deep­Mind’s Prob­a­bil­is­tic Meth­ods for Eval­u­at­ing Agent Capabilities

Un­der­rated Proverbs

• Arjun Panickssery 11 Jun 2024 15:27 UTC

  30 points

  8

  on: Ar­jun Pan­ickssery’s Shortform

  FiveThirtyEight released their prediction today that Biden currently has a 53% of winning the election | Tweet

  The other day I asked:

  Should we anticipate easy profit on Polymarket election markets this year? Its markets seem to think that

  • Biden will die or otherwise withdraw from the race with 23% likelihood

  • Biden will fail to be the Democratic nominee for whatever reason at 13% likelihood

  • either Biden or Trump will fail to win nomination at their respective conventions with 14% likelihood

  • Biden will win the election with only 34% likelihood

  Even if gas fees take a few percentage points off we should expect to make money trading on some of this stuff, right (the money is only locked up for 5 months)? And maybe there are cheap ways to transfer into and out of Polymarket?

  Probably worthwhile to think about this further, including ways to make leveraged bets.

• Arjun Panickssery 7 Jun 2024 15:57 UTC

  20 points

  10

  on: Ar­jun Pan­ickssery’s Shortform

  Should we anticipate easy profit on Polymarket election markets this year? Its markets seem to think that

  • Biden will die or otherwise withdraw from the race with 23% likelihood

  • Biden will fail to be the Democratic nominee for whatever reason at 13% likelihood

  • either Biden or Trump will fail to win nomination at their respective conventions with 14% likelihood

  • Biden will win the election with only 34% likelihood

  Even if gas fees take a few percentage points off we should expect to make money trading on some of this stuff, right (the money is only locked up for 5 months)? And maybe there are cheap ways to transfer into and out of Polymarket?

• Arjun Panickssery 4 Jun 2024 12:13 UTC

  14 points

  9

  on: Just ad­mit that you’ve zoned out

  I like “Could you repeat that in the same words?” so that people don’t try to rephrase their point for no reason.

  In addition to daydreaming, sometimes you’re just thinking about the first of a series of points that your interlocutor made one after the other (a lot of rationalists talk too fast).

• Arjun Panickssery 3 Jun 2024 19:44 UTC

  2 points

  0

  in reply to: sunwillrise’s comment on: Ar­jun Pan­ickssery’s Shortform

  By “subscriber growth” in OP I meant both paid and free subscribers.

  My thinking was that people subscribe after seeing posts they like, so if they get to see the body of a good post they’re more likely to subscribe than if they only see the title and the paywall. But I guess if this effect mostly affects would-be free subscribers then the effect mostly matters insofar as free subscribers lead to (other) paid subscriptions.

  (I say mostly since I think high view/​subscriber counts are nice to have even without pay.)

• Arjun Panickssery 3 Jun 2024 19:31 UTC

  10 points

  4

  on: Ar­jun Pan­ickssery’s Shortform

  Paid-only Substack posts get you money from people who are willing to pay for the posts, but reduce both (a) views on the paid posts themselves and (b) related subscriber growth (which could in theory drive longer-term profit).

  So if two strategies are

  1. entice users with free posts but keep the best posts behind a paywall

  2. make the best posts free but put the worst posts behind the paywall

  then regarding (b) above. the second strategy has less risk of prematurely stunting subscriber growth, since the best posts are still free. Regarding (a), it’s much less bad to lose view counts on your worst posts.

“Suc­cess­ful lan­guage model evals” by Ja­son Wei

• Arjun Panickssery 24 May 2024 14:07 UTC

  12 points

  8

  on: Ar­jun Pan­ickssery’s Shortform

  [Book Review] The 8 Mansion Murders by Takemaru Abiko

  As a kid I read a lot of the Sherlock Holmes and Hercule Poirot canon. Recently I learned that there’s a Japanese genre of honkaku (“orthodox”) mystery novels whose gimmick is a fastidious devotion to the “fair play” principles of Golden Age detective fiction, where the author is expected to provide everything that the attentive reader needs to come up with the solution himself. It looks like a lot of these honkaku mysteries include diagrams of relevant locations, genre-savvy characters, and a puzzle-like aesthetic. A bunch have been translated by Locked Room International.

  The title of The 8 Mansion Murders doesn’t refer to the number of murders, but to murders committed in the “8 Mansion,” a mansion designed in the shape of an 8 by the eccentric industrialist who lives there with his family (diagrams show the reader the layout). The book is pleasant and quick—it didn’t feel like much over 50,000 words. Some elements feel very Japanese, like the detective’s comic-relief sidekick who suffers increasingly serious physical-comedy injuries. The conclusion definitely fits the fair-play genre in that it makes sense, could be inferred from the clues, is generally ridiculous, and doesn’t offer much in the way of motive.

  If you like mystery novels, I would recommend reading one of these honkaku mysteries for the novelty. Maybe not this one, since there are more famous ones (this one was on libgen).

• Arjun Panickssery 23 May 2024 15:13 UTC

  54 points

  17

  on: Ar­jun Pan­ickssery’s Shortform

  Ask LLMs for feedback on “the” rather than “my” essay/​response/​code, to get more critical feedback.

  Seems true anecdotally, and prompting GPT-4 to give a score between 1 and 5 for ~100 poems/​stories/​descriptions resulted in an average score of 4.26 when prompted with “Score my …” versus an average score of 4.0 when prompted with “Score the …” (code).

• Arjun Panickssery 20 May 2024 16:02 UTC

  6 points

  1

  in reply to: AnnaSalamon’s comment on: Stephen Fowler’s Shortform

  https://​​x.com/​​panickssery/​​status/​​1792586407623393435

• Arjun Panickssery 29 Apr 2024 16:30 UTC

  7 points

  −3

  in reply to: metachirality’s comment on: Ar­jun Pan­ickssery’s Shortform

  If I understand the term “double crux” correctly, to say that something is a double crux is just to say that it is “crucial to our disagreement.”

• Arjun Panickssery 29 Apr 2024 14:08 UTC

  3 points

  −6

  on: Ar­jun Pan­ickssery’s Shortform

  Quick Take: People should not say the word “cruxy” when already there exists the word “crucial.” | Twitter

  Crucial sometimes just means “important” but has a primary meaning of “decisive” or “pivotal” (it also derives from the word “crux”). This is what’s meant by a “crucial battle” or “crucial role” or “crucial game (in a tournament)” and so on.
  
  So if Alice and Bob agree that Alice will work hard on her upcoming exam, but only Bob thinks that she will fail her exam—because he thinks that she will study the wrong topics (h/​t @Saul Munn)—then they might have this conversation:

  Bob: You’ll fail
  Alice: I won’t, because I’ll study hard.
  Bob: That’s not crucial to our disagreement.

“Why I Write” by Ge­orge Or­well (1946)

• Arjun Panickssery 24 Apr 2024 2:07 UTC

  5 points

  0

  in reply to: jacobjacob’s comment on: Ex­press in­ter­est in an “FHI of the West”

  The older nickname was “Cornell of the West.” Stanford was modeled after Cornell.

• Arjun Panickssery 18 Apr 2024 19:36 UTC

  14 points

  0

  on: [Fic­tion] A Confession

  This story is inspired by The Trouble With Being Born, a collection of aphorisms by the Romanian philosopher Emil Cioran (discussed more here), including the following aphorisms:

  A stranger comes and tells me he has killed someone. He is not wanted by the police because no one suspects him. I am the only one who knows he is the killer. What am I to do? I lack the courage as well as the treachery (for he has entrusted me with a secret—and what a secret!) to turn him in. I feel I am his accomplice, and resign myself to being arrested and punished as such. At the same time, I tell myself this would be too ridiculous. Perhaps I shall go and denounce him all the same. And so on, until I wake up.
  
  The interminable is the specialty of the indecisive. They cannot mark life out for their own, and still less their dreams, in which they perpetuate their hesitations, pusillanimities, scruples. They are ideally qualified for nightmare.

  Here on the coast of Normandy, at this hour of the morning, I needed no one. The very gulls’ presence bothered me: I drove them off with stones. And hearing their supernatural shrieks, I realized that that was just what I wanted, that only the Sinister could soothe me, and that it was for such a confrontation that I had got up before dawn.

[Fic­tion] A Confession

LLM Eval­u­a­tors Rec­og­nize and Fa­vor Their Own Generations

• Arjun Panickssery 31 Mar 2024 22:16 UTC

  2 points

  0

  on: Your LLM Judge may be biased

  See “Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions” (Pezeshkpour and Hruschka, 2023):

  Large Language Models (LLMs) have demonstrated remarkable capabilities in various NLP tasks. However, previous works have shown these models are sensitive towards prompt wording, and few-shot demonstrations and their order, posing challenges to fair assessment of these models. As these models become more powerful, it becomes imperative to understand and address these limitations. In this paper, we focus on LLMs robustness on the task of multiple-choice questions—commonly adopted task to study reasoning and fact-retrieving capability of LLMs. Investigating the sensitivity of LLMs towards the order of options in multiple-choice questions, we demonstrate a considerable performance gap of approximately 13% to 75% in LLMs on different benchmarks, when answer options are reordered, even when using demonstrations in a few-shot setting. Through a detailed analysis, we conjecture that this sensitivity arises when LLMs are uncertain about the prediction between the top-2/​3 choices, and specific options placements may favor certain prediction between those top choices depending on the question caused by positional bias. We also identify patterns in top-2 choices that amplify or mitigate the model’s bias toward option placement. We found that for amplifying bias, the optimal strategy involves positioning the top two choices as the first and last options. Conversely, to mitigate bias, we recommend placing these choices among the adjacent options. To validate our conjecture, we conduct various experiments and adopt two approaches to calibrate LLMs’ predictions, leading to up to 8 percentage points improvement across different models and benchmarks.

  
  Also “Benchmarking Cognitive Biases in Large Language Models as Evaluators” (Koo et al., 2023):

  Order Bias is an evaluation bias we observe when a model tends to favor the model based on the order of the responses rather than their content quality. Order bias has been extensively studied (Jung et al., 2019; Wang et al., 2023a; Zheng et al., 2023), and it is well-known that state-of-the-art models are still often influenced by the ordering of the responses in their evaluations. To verify the existence of order bias, we prompt both orderings of each pair and count the evaluation as a “first order” or “last order” bias if the evaluator chooses the first ordered (or last ordered) output in both arrangements respectively.