Optimization Process 9 Feb 2021 9:59 UTC
16 points
on: Promoting Prediction Markets With Meaningless Internet-Point Badges
I would love to live in this world.

This seems like a really hard problem: if a market like this “wins,” so that having a lot of points makes you high-status, people will try to game it, and if gaming it is easy, this will kill respect for the market.

Specific gaming strategies I can think of:
- Sybil attacks: I create one “real” account and 100 sock puppets; my sock puppets make dumb bets against my real account; my real account gains points, and I discard my sock puppets. Defenses I’ve heard of against Sybil attacks: make it costly to participate (e.g. proof-of-work); make the cost of losing at least as great as the benefit of winning (e.g. make “points” equal money); or do Distributed Trust Stuff (e.g. Rangzen, TrustDavis).
- Calibration-fluffing: if the market grades me on calibration, then I can make dumb predictions but still look perfectly calibrated by counterbalancing those with more dumb predictions (e.g. predict “We’ll have AGI by Tuesday, 90%”, then balance that out with nine “The sun will rise tomorrow, 90%” predictions). To protect against this… seems like you’d need some sort of way to distinguish “predictions that matter” from “calibration fluff.”
- Buying status: pay people to make dumb bets against you. The Metaculus equivalent of buying Likes or Amazon reviews. On priors, if Amazon can’t squash this problem, it probably can’t be squashed.

How can I reconcile these COVID test false-negative numbers?

Optimization Process27 Oct 2020 23:56 UTC

15 points

5 comments2 min readLW link

Optimization Process 15 Feb 2022 3:10 UTC
14 points
on: Voting Results for the 2020 Review
I donated for some nonzero X:
- $X to johnswentworth for “Alignment By Default”, which gave a surprisingly convincing argument for something I’d dismissed as so unlikely as to be not worth thinking about.
- $2X to Daniel Kokotajlo for “Against GDP as a metric for timelines and takeoff speeds”, for turning me, uh, Against GDP as a metric for timelines and takeoff speeds.
- $2X to johnswentworth for “When Money Is Abundant, Knowledge Is The Real Wealth”, which I think of often.
- $10X to Microcovid.org, which has provided me many times that much value.

Optimization Process 6 Sep 2021 15:44 UTC
14 points
in reply to: Rob Bensinger’s comment on: Rough notes on the Sam Altman Q&A: GPT and AGI
Everything in the OP matches my memory / my notes, within the level of noise I would expect from my memory / my notes.

Two children’s stories

Optimization Process12 Nov 2023 20:29 UTC

11 points

1 comment7 min readLW link

[Question] Does there exist a detailed Bayesian COVID tracker?

Optimization Process12 Nov 2020 5:06 UTC

8 points

5 comments1 min readLW link

Board Game Theory

Optimization Process3 Apr 2023 6:23 UTC

8 points

0 comments3 min readLW link

Optimization Process 1 Jan 2023 19:59 UTC
8 points
−1
on: Models Don’t “Get Reward”
I found this lens very interesting!
Upon reflection, though, I begin to be skeptical that “selection” is any different from “reward.”
Consider the description of model-training:
To motivate this, let’s view the above process not from the vantage point of the overall training loop but from the perspective of the model itself. For the purposes of demonstration, let’s assume the model is a conscious and coherent entity. From it’s perspective, the above process looks like:
- Waking up with no memories in an environment.
- Taking a bunch of actions.
- Suddenly falling unconscious.
- Waking up with no memories in an environment.
- Taking a bunch of actions.
- and so on.....
The model never “sees” the reward. Each time it wakes up in an environment, its cognition has been altered slightly such that it is more likely to take certain actions than it was before.
What distinguishes this from how my brain works? The above is pretty much exactly what happens to my brain every millisecond:
- It wakes up in an environment, with no memories^[1]; just a raw causal process mapping inputs to outputs.
- It receives some inputs, and produces some outputs.
- It’s replaced with a new version—almost identical to the old version, but with some synapse weights and activation states tweaked via simple, local operations.
- It wakes up in an environment...
- and so on...
Why say that I “see” reward, but the model doesn’t?
1. ^
  Is it cheating to say this? I don’t think so. Both I and GPT-3 saw the sentence “Paris is the capital of France” in the past; both of us had our synapse weights tweaked as a result; and now both of us can tell you the capital of France. If we’re saying that the model doesn’t “have memories,” then, I propose, neither do I.