benjamincosman

Karma: 211

benjamincosman 28 Jun 2022 17:38 UTC
2 points
0 ∶ 0
in reply to: burmesetheater’s comment on: Air Conditioner Repair
I do not of course know your intentions, but this comment really rubbed me the wrong way:
- Most importantly, there’s the everybody-knows dynamic (which, unrelatedly, Zvi has written about). Something that you happen to know is usually not as common knowledge as you think, and even if this case actually is mostly common knowledge, you could probably have found a way to write it that sounds nicer (i.e. less of a you’re-an-ignorant-outsider vibe) and/or better supported (got any stats/links showing how common this kind of scamming really is?)
- Less importantly, the ‘modern USA’ phrasing feels to me like it’s taking a dig at something, like (a less extreme version of) whichever of the following feels most unfair to you: “of course this kind of scamming is common—welcome to capitalism”, or “of course this kind of scamming is common—welcome to Biden’s USA”.

benjamincosman 25 Jun 2022 0:35 UTC
4 points
on: Do alignment concerns extend to powerful non-AI agents?
If you, the reader, or, say, Paul Christiano or Eliezer gets uploaded and obtains self-improvement, self-modification and processing speed/power capabilities, will your goals converge to damaging humanity as well? If not, what makes it different? How can we transfer this secret sauce to an AI agent?

The Orthogonality Thesis states that values and capabilities can vary independently. The key question then is whether my/Paul’s/Eliezer’s values are actually as aligned with humanity as they appear to be, or if instead we are already unaligned and would perform a Treacherous Turn once we had the power to get away with it. There are certainly people who are already obviously bad choices, and people who would perform the Treacherous Turn (possibly most people^[1]), but I believe there are people who are sufficiently aligned, so let’s assume going forward we’ve picked one of those. At this point “If not, what makes it different?” answers itself: by assumption we’ve picked a person for whom the Value Loading Problem is already solved. But we have no idea how to “transfer this secret sauce to an AI agent”—the secret sauce is hidden somewhere along this person’s particular upbringing and more importantly their multi-billion year evolutionary history.
1. The adage “power tends to corrupt, and absolute power corrupts absolutely” basically says that treacherous turns are commonplace for humans—we claim to be aligned and might even believe it ourselves while we are weak, but then when we get power we abuse it. This adage existing does not of course mean it’s universally true. ↩︎

benjamincosman 24 Jun 2022 15:10 UTC
8 points
on: Intelligence in Commitment Races

Eliezer would never precommit to probably turn down a rock with an un-Shapley offer painted on its front (because non-agents bearing fixed offers created ex nihilo cannot be deterred or made less likely through any precommitment)

But we do not live in a universe where such rocks commonly appear ex nihilo. So when you come across even a non-agent making an unfair offer, you have to consider the likelihood that it was created by an adversarial agent (as is indeed the case in your Bot example).

benjamincosman 23 Jun 2022 7:29 UTC
12 points
in reply to: Yitz’s comment on: What’s the contingency plan if we get AGI tomorrow?
My guess is that there is virtually zero value in working on 24-hour-style emergency measures, because:
- The probability we end up with a known 24-hour-ish window is vanishingly small. For example I think all of the following are far more likely:
  - no-window defeat (things proceed as at present, and then with no additional warning to anyone relevant, the leading group turns on unaligned AGI and we all die)
  - no-window victory (as above, except the leading group completely solves alignment and there is much rejoicing)
  - various high-variance but significantly-longer-than-24hr fire alarms (e.g. stories in the same genre as yours, except instead of learning that we have 24 hours, the new research results / intercepted intelligence / etc makes the new best estimate 1-4 years / 1-10 months / etc)
- The probability that anything we do actually affects the outcome is much higher in the longer term version than in the 24-hour version, which means that even the scenarios were equally likely, we’d probably get more EV out of working on the “tractable” (by comparison) version.
- Work on the “tractable” version is more likely to generalize than work on the emergency version, e.g. general alignment researchers might incidentally discover some strategy which has a chance of working on a short time horizon, but 24-hour researchers are less likely to incidentally discover longer-term solutions, because the 24-hour premise makes long-term categories of things (like lobbying for regulations) not worth thinking about.

benjamincosman 30 May 2022 23:05 UTC
2 points
in reply to: lc’s comment on: lc’s Shortform
I think we should reward admitting-of-ignorance, or at the very least not punish it.

benjamincosman 30 May 2022 23:02 UTC
4 points
in reply to: Pattern’s comment on: lc’s Shortform
Bayes’ Theorem, presumably.

benjamincosman 18 May 2022 16:07 UTC
6 points
on: Why don’t 4 lottery wins prove information from a higher power?
As a partial answer, note that you will get extremely different results for P(H|E) depending on your choice of hypothesis H and evidence E. In particular, while naively E here is “Ms X won the lottery 4 times”, I think you’d still be posting the same question if instead you’d heard that “Mr Y won the lottery 4 times”, or more importantly, any other extremely unlikely positive-valence event (“Ms Z was struck by a bullet in the tiny area that happened to be protected by an object in her pocket”, etc). Which means that your E should perhaps be “some lucky unlikely thing happened to someone at some point in history”, which no longer seems very low probability after all.

Also the jump from “there’s something weird here” to “the Christian god did it” is too big—even if we do decide that some extraordinary explanation is required, our hypothesis space should at the very least include other gods :) and perhaps more realistically, things like bugs in the lottery program, collusion with corrupt lottery officials, etc.

benjamincosman 11 May 2022 17:22 UTC
2 points
in reply to: Yoav Ravid’s comment on: ProjectLawful.com: Eliezer’s latest story, past 1M words
The author’s (user)name is always the final line of the inset box. If there are three lines in that box (e.g. “Carissa Sevar // to-let-you-in // lintamande”), then lintamande is the author, Carissa Sevar is the character, and “to-let-you-in” can be ignored (it’s some sort of thematic tag for the character).

Authors other than the main two don’t appear for an extremely long time so I’d worry about that after getting that far :)

benjamincosman 30 Apr 2022 15:54 UTC
8 points
in reply to: Said Achmiz’s comment on: Salvage Epistemology
Who is “we”? This is more or less what I’ve observed from 100% of my (admittedly small sample size of) rationalist acquaintances who have taken psychedelics.

benjamincosman 30 Apr 2022 13:03 UTC
7 points
in reply to: Said Achmiz’s comment on: Salvage Epistemology

it’s especially egregious when you apply this “salvage epistemology” approach to, say, taking drugs

I’m not so certain of that? Of the two extreme strategies “Just Say No” and “do whatever you want man, it feels goooooood”, Just Say No is the clear winner. But when I’ve interacted with reasonable-seeming people who’ve also done some drugs, it looks like “here’s the specific drug we chose and here’s our safety protocol and here’s everything that’s known to science about effects and as you can see the dangers are non-zero but low and we think the benefits are also non-zero and are outweighing those dangers”. And (anecdotally of course) they and all their friends who act similarly appear to be leading very functional lives; no one they know has gotten into any trouble worse than the occasional bad trip or exposure to societal disapproval (neither of which was ultimately a big deal, and both of which were clearly listed in their dangers column).

Now it is quite possible they’re still ultimately definitively wrong—maybe there are serious long-term effects that no one has figured out yet; maybe it turns out that the “everyone they know turns out ok” heuristic is masking the fact that they’re all getting really lucky and/or the availability bias since the ones who don’t turn out ok disappear from view; etc. And you can certainly optimize for safety by responding to all this with “Just Say No”. But humans quite reasonably don’t optimize purely for safety, and it is not at all clear to me that what these ones have chosen is crazy.

benjamincosman 30 Apr 2022 0:32 UTC
1 point
in reply to: TLW’s comment on: Dath Ilan vs. Sid Meier’s Alpha Centauri: Pareto Improvements
Simon’s services are only offered to those who submit a treaty over a full range of possible outcomes. Alice could try to bully Bob into accepting a bullshit treaty (“if I win you give me X; if I lose you still give me X”), but just like today Bob has the backup plan of refusing arbitration and actually waging war. (Refusing arbitration is allowed; it’s only going to arbitration and then not abiding by the result that is verboten.) And Alice could avoid committing to concessions-on-loss by herself refusing arbitration and waging war, but she wouldn’t actually be in a better position by doing so, since the real-world war also involves her paying a price in the (to her mind) unlikely event that Bob wins and can extract one. Basically the whole point is that, as long as the simulated war and the agreed-upon consequences of war (minus actual deaths and other stuff) match a potential real-world war closely enough, then accepting the simulation should be a strict improvement for both sides regardless of their power differential and regardless of the end result, so both sides should (bindingly) accept.

benjamincosman 29 Apr 2022 15:27 UTC
3 points
in reply to: TLW’s comment on: Dath Ilan vs. Sid Meier’s Alpha Centauri: Pareto Improvements
Good point, so continuing with the superhuman levels of coordination and simulation: instead of Alice and Bob saying “we’re thinking of having a war” and Simon saying “if you did, Bob would win with probability p”; Alice and Bob say “we’ve committed to simulating this war and have pre-signed treaties based on various outcomes”, and then Simon says “Bob wins with probability p by deploying secret weapon X, so Alice you have to pay up according to that if-you-lose treaty”. So Alice does learn about the weapon but also has to pay the price for losing, exactly like she would in an actual war (except without the associated real-world damage).

benjamincosman 29 Apr 2022 13:15 UTC
2 points
in reply to: TLW’s comment on: Dath Ilan vs. Sid Meier’s Alpha Centauri: Pareto Improvements
As an example, part of your military strength might be your ability to crash enemy systems with zero-day software exploits (or any other kind of secret weapon they don’t yet have counters for). At least naively, you can’t demonstrate you have such a weapon without rendering it useless. Though this does suggest a (unrealistically) high-coordination solution to at least this version of the problem: have both sides declare all their capabilities to a trusted third party who then figures out the likely costs and chances of winning for each side.

benjamincosman 19 Apr 2022 3:27 UTC
13 points
in reply to: mukashi’s comment on: A Quick Guide to Confronting Doom
Note that an observer must always see that all previous prophets of existential doom were wrong. (Yay anthropics. I’m not sure if or how much this should change our reasoning though.)

benjamincosman 17 Apr 2022 5:50 UTC
2 points
on: Fiction: My alternate earth story.
Neat story :) a few typos to fix:
- “coal minor” → coal miner
- “went rouge” → went rogue
- “their I was” → there I was

benjamincosman 14 Apr 2022 22:48 UTC
3 points
in reply to: Lblack’s comment on: Lies Told To Children
Your feelings about lies depend on the context—for example I assume you’d be willing to play the game “Two Truths and a Lie”, and you would not feel harmed by the certainty that someone is lying to you? In fact people enjoy the game, since seeing through the lie is a fun puzzle. Now outside of games like that, the majority of the time someone lies to you on Earth, it’s to profit at your expense—they want to take your stuff, your vote, etc. So with the exception of games, you’ve quite reasonably developed strong negative emotions about being lied to, and those emotions may transfer even to the rarer cases where the lie isn’t directly hurting you. But dath ilan is extremely high trust and high coordination; from childhood you will experience that the vast majority of the times someone lies to you, it’s clearly-in-retrospect for your own benefit, and the vast majority of the remaining times, it’s solidly for the benefit of Civilization and they’re willing to eventually tell you the truth and pay for your inconvenience. So while you still try to see through lies whenever you can, it’s much more like the Earth game setting: most lies are just harmless puzzles. So you don’t grow up with the same internalized feelings that anyone lying to you is hurting you? (Which also reduces the price they’d have to pay you, since they’re not trying to compensate you for an Earth-level of negative emotions.)

benjamincosman 13 Apr 2022 17:06 UTC
11 points
in reply to: Thomas Kwa’s comment on: How dath ilan coordinates around solving alignment
I originally downvoted because I couldn’t figure out the point. The title definitely contributed to that, though not for the reason you suggest: rather, the title made it sound like this fact about a fictional world was the point, in the same way that you would expect to come away from a post titled “Rowling: Wizarding Britain has a GDP of 1 million Galleons” pretty convinced about a fictional GDP but with possibly no new insights on the real world. I think the new title makes it a bit clearer what one might expect to get out of this: it’s more like “here’s one (admittedly fictional) story of how alignment was handled; could we do something similar in our world?”. I’m curious if that’s the intended point? If so then all the parts analyzing how well it worked for them (e.g. the stuff about cryonics and 97%) still don’t seem that relevant: “here’s an idea from fiction you might consider in reality” and “here’s an idea from fiction which worked out for them you might consider in reality” provide me almost identical information, since the valuable core worth potentially adopting is the idea, but all discussion of whether it worked in fiction tells me extremely little about whether it will work in reality (especially if, as in this case, very little detail is given on why it worked in fiction).

benjamincosman 4 Apr 2022 2:14 UTC
4 points
on: The Jordan Peterson vs Sam Harris Debate
For reference, I think the referenced “old Jewish story” is The Oven of Akhnai. Though it doesn’t particularly matter for the present discussion, I will note that the disagreement was between one rabbi and a bunch of other rabbis (rather than between members of a congregation); since the story was also written by the rabbis, this makes it analogous to our Marbury v. Madison.

(though it would also not be very surprising if the story has multiple versions/settings)

benjamincosman 27 Mar 2022 21:46 UTC
1 point
in reply to: benjamincosman’s comment on: How to evaluate (50%) predictions
I think a lot of the pushback against the post that I’m seeing in the older comments is generated by the fact that this “confidence > baseline” rule is presented in its final form without first passing through a stage where it looks more symmetrical. By analogy, imagine that in the normal calibration setting, someone just told you that you are required to phrase all your predictions such that the probabilities are >= 50%. “But why,” you’d think; “doesn’t the symmetry of the situation almost guarantee that this has to be wrong—in what sane world can we predict 80% but we can’t predict 20%?” So instead, the way to present the classical version is that you can predict any value between 0 and 100, and then precisely because of the symmetry noticed above, for the purpose of scoring we lump together the 20s and 80s. And that one possible implementation of this is to do the lumping at prediction-time instead of scoring-time by only letting you specify probabilities >= 50%. Similarly in the system from this post, the fundamental thing is that you have to provide your probability and also a direction that it differs from baseline. And then come scoring time, we will lump together “80%, baseline is higher” with “20%, baseline is lower”. Which means one possible implementation is to do the lumping at prediction-time by only allowing you to make “baseline is lower” predictions. (And another implementation, for anyone who finds this lens useful since it’s closer to the classical setting, would be to only allow you to make >=50% predictions but you also freely specify the direction of the baseline.)

benjamincosman 27 Mar 2022 21:28 UTC
1 point
on: How to evaluate (50%) predictions
I think the core idea of this post is valid and useful, but that the specific recommendation that predictors phrase their predictions using the “confidence > baseline” rule is a misguided implementation detail. To see why, first consider this variant recommendation we could make:

Variant A: There are no restrictions on the wordings of predictions, but along with their numeric prediction, a predictor should in some manner provide the direction by which the prediction differs from baseline.

Notice that any computation that can be performed in your system can also be performed using Variant A, since I could always at scoring time convert all “I predict [claim] with confidence p; baseline is higher” predictions into “I predict [negation of claim] with confidence 1-p; baseline is lower” to conform to your rule. Now even Variant A is I believe a minor improvement on the “confidence > baseline” rule, because it gives predictors more freedom to word things in human-convenient ways (e.g. “The stock price will be between 512 and 514“ is far nicer to read than either ”...not be between 512 and 514”, or the very clunky ”...less than 512 or more than 514″). But more importantly, Variant A is a useful stepping stone towards two much more significant improvements:

Variant B: …a predictor should in some manner provide the direction by which the prediction differs from baseline, or better yet, an actual numeric value for the baseline.

First, notice as before that any computation that can be performed in Variant A can also be performed using Variant B, since we can easily derive the direction from the value. Now of course I recognize that providing a value may be harder or much harder than reporting direction alone, so I am certainly not suggesting we mandate that the baseline’s value always be reported. But it’s clearly useful to have, even for your original scheme: though you don’t say so explicitly, it seems to be the key component in your “estimate their boldness” step at the end, since boldness is precisely the magnitude of the difference between the prediction and the baseline. So it’s definitely worth at least a little extra effort to get this value, and occasionally the amount of effort required will actually be very small or none anyway: one of the easiest ways to satisfy your original mandate of providing the direction will sometimes be to go find the numeric value (e.g. in a prediction market).

Variant C: …a predictor should in some manner provide the direction by which the prediction differs from baseline, or better yet, some third party should provide this direction.

Now again I recognize that I can’t make anybody, including unspecified “third parties”, do more work than they were going to. But to whatever extent we have the option of someone else (such as whoever is doing the scoring?) providing the baseline estimations, it definitely seems preferable: in the same way that in the classical calibration setting, a predictor has an incentive to accidentally (or “accidentally”) phrase predictions a certain way to guarantee perfect calibration at the 50% level, here they have an incentive to mis-estimate the baselines for the same reason. There might also be other advantages to outsourcing the baseline estimations, including
- this scheme would become capable of scoring predictors who do not provide baselines themselves (e.g. because they simply don’t know they’re supposed to, or because they don’t want to put in that effort)
- we might get scale and specialization advantages from baseline-provision being a separate service (e.g. a prediction market could offer this as an API, since that’s kind of exactly what they do anyway)
So overall, I’d advocate using the tweaks introduced in all the variants above: A) we should not prefer that predictors speak using extra negations when there are equally good ways of expressing the same information, B) we should prefer that baseline values are recorded rather than just their directions, and C) we should prefer that third parties decide on the baselines.