Jisk, formerly Jacob. (And when Jacobs are locally scarce, still Jacob.)
LW has gone downhill a lot from its early days and I disapprove of most of the moderation choices but I’m still, sometimes, here.
It should be possible to easily find me from the username I use here, though not vice versa, for interview reasons.
This might be a good idea if it was feasible to measure in an objective way, but that is absolutely not the case. And I’m not really sure it would be a good idea even then.
You’re looking at it backwards. A large fraction of ratfic readers also read royalroad. But a very small portion of royalroad reads (or wants to read) ratfic. Not enough to affect site trends or gather momentum for an audience. (Though there’s some good near-ratfic in the classic RR-bait categories, like Sky Pride in cultivation.)
Personally I wrote a novel on RR and posted chapters of it to /r/rational last year and… never got any comments. Not even critique. I think I gave up after a month. Was it too generic RR-bait? Was it just not appealing for some reason? I don’t know, and I don’t think I ever will.
That soured me on linking anything there.
The genie faces a single, static decision over probability distributions. There is no temporal sequence, no compounding, no intermediate node at which the genie might re-evaluate. The genie simply picks the best plan, and the best plan is the one whose probability distribution over outcomes ranks highest.
I don’t see how this is a contrast. Is this not what an ergodic strategy is doing, also? A resolute choice? All updateless decision theories are stepping outside time; there’s a reason the first step in that direction was called ‘timeless’ decision theory. Benja’s pretty explicit that these plans include probabilities; it is selecting choices that make sense at particular intermediate steps. If the correct strategy to make bets to achieve long-run growth, that optimizes the bankroll at the time labeled by the genie as ‘outcomes’ is to Kelly-bet, and that’s what you care about, the genie will choose to Kelly-bet.
However, it will not do that, because long-run growth is not what we care about. Optimizing long-run growth is correct, AIUI, only if losing your last dollar is infinitely bad and there is no bankroll size at which you cease to desire it to grow at the same strength you presently desire it. Certainly I prefer a quadrillion dollars to a trillion, but if I have a trillion (in 2025!USD-equivalent), I will cease to be particularly interested in what choice optimizes the long-run growth of that bankroll, and will instead be much more conservative.
I am far from certain. But I believe this thought experiment deviates from what you prescribe only in circumstances where you become the one relying on unnecessary and inappropriate assumptions. It looks to me like the argument there continues to hold for exactly the processes you suggest are superior.
Jensen’s inequality
Take any set of points, a, b, c, …
Take any convex function (=function that is ‘rising’ at every point, such that any line between two points on the function’s graph is above the function, =function which is concave up, = function whose second derivative (assuming it has one) is never negative) which can be applied to those points.
Then the mean of the functions mean(f(a), f(b), f(c), …) is less than the function of the mean f(mean(a, b, c, …)).
via the ergodic mapping, which finds the transformation that renders the wealth process ergodic, so that time and ensemble averages coincide
‘Ergodic’ kept looking like noise to me, and this seemed to be intended as an explanation but, to me, clarified nothing. (Feldspars problem, presumably.) I’ll attempt to rephrase, reflecting my present understanding after a few minutes of research:
Most processes that vary do so differently over time than at any present moment. The ensemble average is looking at a bunch of similar processes and how they’re varying right now, like forming the S&P 500 from 500 individual stocks, essentially buying the ensemble average of their growth (now and in the future). The time average of the growth, is the long run performance of one stock; the (past and/or future) time average of any individual stock is not the same as the ensemble. This makes prediction hard.
Something is ergodic if it does not have this problem. That could mean a steady state; a dollar bill’s value, denominated in USD, is mostly ergodic, though imperfectly because we might someday go full Weimar, abolish greenbacks, or make it an old print that’s a collector’s item. But you can construct things to be ergodic on purpose, as the examples describe; Kelly-betting is a strategy whose ‘many-worlds’ possible present instances have the same bankroll growth ensemble average that, in the long run, the time average predicts.
I am unclear if ‘the ergodic mapping’ is a specific coherent concept or a more flexible process that you apply to a phenomenon in imprecise ways to arrive at a result you can then prove is ergodic.
Perfect glomarization is glomarizing for everything. Glomarize over whether your son is learning to play guitar at all, and for piano, flute, bagpipes, timpani, soccer, baseball, handegg, caber tossing, buzkashi, karate, and arnis, all at once, if he might find it embarrassing to be known to be learning any of them until he’s publicly admitted to it.
So no, examples in everyday conversation are not really possible.
I think this is poorly stated, in that the first two or three times I read it I think I got the opposite claim that was intended, and so did the one commenter before me. To rephrase verbosely to be sure:
If you are honest and privacy-preserving, your glomarization must be perfect.
If you can’t glomarize and want to preserve privacy, you must be dishonest.
If you can’t glomarize and are honest, you cannot protect privacy
Though you can go partway − 80%-perfect honesty and 80%-perfect glomarization is probably good for 80%-preserved privacy.
So in the extreme you’d say that perfect privacy and perfect honesty mean no glomarization at all? That seems like it also removes privacy, to a sufficiently determined nosy ‘attacker.’
I don’t think it’s been trained on and all present frontier models one-shot it.
While implementing both threats simultaneously would presumably involve sending contradictory orders, I see nothing contradictory about making the two threats.
Thank you for writing this part and saving me the trouble.
As I allude to in the footnote, I find this easier in significant part because it is easier to put someone in the category “does not deserve the truth” (sometimes merely temporarily) than to concoct a mixture of half-truths I can believe enough to say them.
An example I sometimes relate, is an essay I was supposed to write for my high school vice-principal. He had instituted a weekly charitable service requirement, and we had to write five hundred words, weekly, about our experiences. I mostly did Habitat, because at least I could be sure I did something useful for someone, which was not common among my classmates, and the writing was harder than the labor, because… it wasn’t making me a better person. I tried to find some lesson that I could plausibly pretend mattered, and that was enormously difficult.
And then for the last one, I wrote about a thousand words on my reflection of the whole program, and whether it was achieving its goals, and precisely why I thought it had completely failed, both for me and all my classmates. Politely, but honestly.
He flipped his shit. Called me into his office, called my parents, I forget the rest.
So that night I wrote another essay, telling him exactly what he wanted to hear. My dad asked why I didn’t write that the first time. I replied that there wasn’t a single word in it I believed. And that was because, in that day, my vice principal had left the category of ‘people to be reasoned with’ and entered the category ‘idiot to be routed around.’ I was not dealing with this man like an equal[1], but an inferior.
This remains a much easier mental motion than bullshitting. This says good things about my honesty and, I expect, bad things about the rest of my moral character.
Let alone superior. I’m not sure I’ve ever related to anyone like a superior in my life, and it sure wasn’t in high school if I did. That got me into considerably more trouble than this bit did.
I don’t think I find most people to think like me enough for that to work. Even other rationalists.
“What a frightening mind you have,” the Elder murmured. “It’s not that you are brilliant, it’s that you are paying attention all the time, and you remember what you are told. You believe it too, provisionally, and discard it ruthlessly if what you are told doesn’t match your observations.”
Tian blinked at that one. “Respectfully, Elder, isn’t that what everyone does?”
“To a degree.”
-- Sky Pride, Warby Picus
That’s what cooperating in the Trembling-Hand Iterated Prisoner’s Dilemma feels like. That the other side is betraying and betraying and betraying. You are not immune to corrupted hardware.
The bar for keeping an existing job and sabotaging should be extremely high, higher than leaving. That’s being litigated plenty across this comment thread, so I’ll just broadly agree and move on.
I will instead argue that it should also be much higher than taking a job intending, from the outset, to be bad at it. I considered, at one point, whether I considered it ethically acceptable to join OpenAI, which I have confidently believed was evil and destructive since the Battle of the Board, as Zvi called it, and intend to do a bad job. I decided it was, ethically, a narrow margin and difficult call, and it would suck, and I wasn’t willing to make sacrifices for something that was merely likely positive in effect but ambivalent and dubious, rather than enormously good. But I did consider it, and had I had more confidence, or other stories for how it might do good to have a principled person inside OpenAI, I might have gone for it. Yes, I’d be making an agreement in which I did not intend to live up to the spirit, only the letter, but it’s something I could conceivably be convinced to do.
If I was already there, no chance whatsoever. It would not be close. I will not take an agreement I entered into in good faith and then decide partway through that I’m going to turn all my future efforts into making them regret it. Not unless the fate of the world is at stake with much higher confidence than this.
Because that’s a good way to lose every other agreement I’m part of. If that gets out once, everyone I’m dealing with needs to worry about whether they’ll be the second time. Even if they’ve seen me be trustworthy for years, if I decided they were evil, I might not just leave them in the lurch, but exploit every iota of the trust I’d earned to make them suffer and pay. Who wants to risk that?
And, maybe worse, how would that work for me, internally? The internal pivot once means that’s on the table in the future. People aren’t very good at compartmentalizing that kind of thing, at being trustworthy now and entirely shutting off the possibility they’ll be untrustworthy later. You’ll looking over your own shoulder in every honest deal, wondering if you should commit too hard, because you might change your mind later.
There’s some element of that in taking a job under false pretenses, Eastern Standard Tribe style. People will mistrust you, and should; you could be a very good actor. But not as much, because “very good actor intending to betray me” and “honest partner” are pretty much the only options. They don’t need to worry about “muddled half-committed possibly future betrayer,” which is good because that’s an enormous target and extremely hard to disprove. And you won’t fool yourself—you’ll knowing you’re putting on a false face, and keep straight what is false and what is true, and if they blur, you can catch it.
The difference is the same thing I just wrote about this week in Lie to me, but at least don’t Bullshit—reckless disregard for the truth vs. straightforward defiance of it. In one sense, lying directly with falsehoods is worse, but in terms of a rationalist’s attitude toward the truth, I would maintain it’s not half as bad as not knowing how much you are lying.
No, it’s screwing people over because you’ve committed to doing something, and that something screws over people. If there was no honorable way out, that would be a difficult moral question; benevolence vs. trustworthiness. But there’s an easy way out: employment is at will and you can simply leave.
There are two distinct desirable properties to have, ethically. (Probably more than two, but two that I can point to here.) One is benevolence: to do, and seek to do, things that are good for people; people in your circle of concern, typically, but that could be everyone who will ever live or just your family or in between. It’s also benevolent to expand your circle of concern on purpose. The other, trustworthiness, is to deal fairly with people when benevolence doesn’t require it. And you might say ‘all humans alive are in my circle, benevolence covers everyone the second one doesn’t matter.’ But this is not correct, because most of those people don’t trust that they are within your circle. They shouldn’t! You have not given them reason to. It’s very easy to say you care about all humanity—observe the case of 2015!Sam Altman! He even looked like he was paying costs for the belief! - and very hard to prove it—again, see how he did not, at all, care about it, and screwed over everyone who founded OpenAI with him.
Either one has to proved, and both are desirable even if you already proved one. Benevolence you prove by showing you keep paying real costs to help people, that you don’t benefit from except via their future benevolence, and by being predictably good for people with similar circles of concern. Then benevolent people see you as a kindred spirit and want to help you, because you’ll pass on help to others they also care about. Trustworthiness you prove by living up to your promises to people, maintaining commitments, and when you thought you could maintain them but can’t, winding down the commitments and paying recompense (monetary or not). Proving trustworthiness is most effective at proving you’re reliable when done with people you don’t much care about, because it shows you’re not up to lying to them, and so are less likely to be lying to those you call friends. Then people can trust that if they extend trust and credit to you—money, borrowed tools, vouching for you, anything—that you won’t take the money (or etc.) and run.
It’s very dangerous to have neither, and have people notice that. “It’s dangerous to go alone! Take this!” Nope, no one’s giving you anything. Society is not built to be navigated alone; it assumes you have a reasonable level of both types of honor, and wants to filter out people who don’t, to make it easier for those who do. “You made a contract with me, didn’t give to my demands, therefore die!” is a madman’s move, because now everyone who only wants to make contracts with reasonably honorable people knows you’re not one. (Someone very benevolent might pull it off. Benevolent, Pete Hegseth and the USG are not.)
And here’s the kicker: You are coming off as in that category too. Someone extremely untrustworthy, which you appear to proudly be, may still be benevolent. But if you’re sufficiently sloppy in your thinking you can convince yourself lots of selfish things are benevolent. (I could point to SBF, except that I think he was probably never particularly benevolent either.) And your thinking here’s pretty darn sloppy! So I don’t particularly trust that you’ll notice if you’re actually not benevolent, and I certainly don’t trust that you’ll admit it if you aren’t.
That’s, again, outside the X-risk model here. This is something that could succeed if entrusted to a trustworthy principal, unlike other methods where that would not be sufficient and there doesn’t seem to be a replacement of similar or lower difficulty that would be sufficient either. This is a method that, if it’s correct, reduces to a very difficult but ordinary human challenge.
But if I’m forced to give a name, Amanda Askell.
I’m very curious how this goes!
“In a way that seems reasonable to the judge” is hiding an enormous amount of complexity that cannot possibly be adjudicated fairly or with a reasonable expectation that your methods at the factory to test it will be approved in court.