I had assumed the first—they’re afraid of imperfect-values lock-in. I think it’s the “not to the problem of preventing complete disaster” phrase that tipped me off here.
David Udell
Karma: 1,060
The verdict that knowledge is purely a property of configurations cannot be naively generalized from real life to GPT simulations, because “physics” and “configurations” play different roles in the two (as I’ll address in the next post). The parable of the two tests, however, literally pertains to GPT. People have a tendency to draw erroneous global conclusions about GPT from behaviors which are in fact prompt-contingent, and consequently there is a pattern of constant discoveries that GPT-3 exceeds previously measured capabilities given alternate conditions of generation[29], which shows no signs of slowing 2 years after GPT-3’s release.
Making the ontological distinction between GPT and instances of text which are propagated by it makes these discoveries unsurprising: obviously, different configurations will be differently capable and in general behave differently when animated by the laws of GPT physics. We can only test one configuration at once, and given the vast number of possible configurations that would attempt any given task, it’s unlikely we’ve found the optimal taker for any test.
Reading this was causally responsible for me undoing any updates I made after being disappointed by my playing with GPT-3. Those observations weren’t more likely inside a weak-GPT world, because a strong-GPT would just as readily simulate weak-simulacra in my contexts as it would strong-simulacra in other contexts.
I think I had all the pieces to have inferred this… but some subverbal part of my cognition was illegitimately epistemically nudged by the manifest limitations of naïvely prompted GPT. That part of me, I now see, should have only been epistemically pushed around by quite serious, professional toying with GPT!
Reflexively check both sides of the proposed probability of an event:
“What do I think about P(DOOM) = 81%?”
and
“What do I think about P(~DOOM) = 19%?”
This can often elicit feedback from parts of you that would stay silent if you only considered one way of stating the probability in question.
When the sanity waterline is so low, it’s easy to develop a potent sense of misanthropy.
Bryan Caplan’s writing about many people hating stupid people really affected me on this point. Don’t hate, or even resent, stupid people; trade with them! This is a straightforward consequence of Ricardo’s comparative advantage theorem. Population averages are overrated; what matters is whether the individual interactions between agents in a population are positive-sum, not where those individual agents fall relative to the population average.
I note that Eliezer thinks that corrigibility is one currently-impossible-to-instill-in-an-AGI property that humans actually have. The sum total of human psychology… consists of many such impossible-to-instill properties.
This is why we should want to accomplish one impossible thing, as our stopgap solution, rather than aiming for all the impossible things at the same time, on our first try at aligning the AGI.
I hereby confer on you, reader, the shroud of epistemic shielding from predictably misleading statements. It confers irrevocable, invokable protection from having to think about predictably confused claims ever again.
Take those cognitive cycles saved, and spend them well!
I’ve noticed that part of me likes to dedicate disproportionate cognitive cycles to the question: “If you surgically excised all powerful AI from the world, what political policies would be best to decree, by your own lights?”
The thing is, we live in a world with looming powerful AI. It’s at least not consequentialist to spend a bunch of cognitive cycles honing your political views for a world we’re not in. I further notice that my default justification for thinking about sans-AI politics a lot is consequentialist… so something’s up here. I think some part of me has been illegitimately putting his thumb on an epistemic scale.
Fancy epistemic tools won’t override the basics of good epistemics:
You are embedded in a 3D spatial world, progressing in a time dimension. You want to get better at predicting events in advance, so you want to find the underlying generator for this 3D world’s events. This means that you’re rooting around in math space, trying to find the mathematical object that your observational trajectory is embedded in.
Some observations of yours are differentially more likely in some math objects than in others, and so it’s more likely that your world is the former math object than the latter. You start with some guess as to how relatively likely you are to live in all these different math objects, and eliminate all parts of that weighted possibility space that are inconsistent with what you observe. A world that anti-predicted what happened with 60% of its probability mass only keeps the remaining 40% of probability mass after that inconsistency. Always ask yourself: was that observation more consistent with one generator than with another generator? If so, then you “update” towards the first generator being your world, vs. the second generator—relatively more of the first generator was consistent with that observation than was the second generator.
Dath Ilan’s Views on Stopgap Corrigibility
Beautifully written story! It reminded me of (related stories spoilers)
John Barnes’ Kaleidoscope Century (1995), which similarly goes through a future-shock laden life trajectory, plus shares some other themes with your story. The progressive sense of future shock here is also like “The Big Sleep,” a short story about economics and cryogenics, and the vertiginous sense of your possibly being flung into the future at any moment reminds me of something in qntm says in Fine Structure:
”The simplest way to put it is this: once digitised, your mind could be sent anywhere, anytime. As you’ve mentioned yourself, it’s thought that within a few decades it will become possible to store an arbitrary amount of data in a single fundamental particle, itself stored in a device as small as a basketball… or a thumb… or a fingernail. You will be copied and copied and copied all over the world. Copies of your mind-state—the first digitised human mind-state in history, remember—could survive until the end of human civilisation. After you go to sleep this afternoon, one of you will wake up tomorrow morning. There is, let us say, a one in a million chance that you will wake up tomorrow morning. The rest of you are embarking on a subjectively instantaneous one-way journey into the uttermost unknown, where, beyond a few decades into the future, your single physical self will not be able to protect you. You will be completely without support or protection or preparation… This is the most dangerous thing anybody has ever done.”
Try pinging yourself:
What’s overdetermined by what you already know?
153
If you take each of the digits of 153, cube them, and then sum those cubes, you get 153:
1 + 125 + 27 = 153.
For many naturals, if you iteratively apply this function, you’ll return to the 153 fixed point. Start with, say, 298:
8 + 729 + 512 = 1,249
1 + 8 + 64 + 729 = 802
512 + 0 + 8 = 516
125 + 1 + 216 = 342
27 + 64 + 8 = 99
729 + 729 = 1,458
1 + 64 + 125 + 512 = 702
343 + 0 + 8 = 351
27 + 125 + 1 = 153
1 + 125 + 27 = 153
1 + 125 + 27 = 153...
Guidelines for Mad Entrepreneurs
There’s a rationality-improving internal ping I use on myself, which goes, “what do I expect to actually happen, for real?”
This ping moves my brain from a mode where it’s playing with ideas in a way detached from the inferred genre of reality, over to a mode where I’m actually confident enough to bet about some outcomes. The latter mode leans heavily on my priors about reality, and, unlike the former mode, looks askance at significantly considering long, conjunctive, tenuous possible worlds.
What’re the odds that we’re anywhere close to optimal in any theoretical domain? Where are our current models basically completed, boundedly optimal representations of some part of the universe?
The arguments for theoretical completion are stronger for some domains than others, but in general the odds that we have the best model in any domain are pretty poor, and are outright abysmal in the mindkilling domains.
“Ignorant people do not exist.”
It’s really easy to spend a lot of cognitive cycles churning through bad, misleading ideas generated by the hopelessly confused. Don’t do that!
The argument that being more knowledgeable leaves you strictly better off than being ignorant does relies you simply ignoring bad ideas when you spend your cognitive cycles searching for improvements on your working plans. Sometimes, you’ll need to actually exercise this “simply ignore it” skill. You’ll end up needing to do so more and more, to approach bounded instrumental rationality, the more inadequate civilization around you is and the lower its sanity waterline.
Note that a “distant entity” can be a computation that took place in a different part of your brain! Your thoughts therefore can’t perfectly track other thoughts elsewhere in your head—your whole brain is at all noisy, and so will sometimes distort the information being passed around inside itself.
You sometimes misspeak… and you sometimes misthink. That is, sometimes your cognitive algorithm a word, and the thought that seemed so unimpeachably obvious in your head… is nevertheless false on a second glance.
Your brain is a messy probabilistic system, so you shouldn’t expect its cognitive state to ever perfectly track the state of a distant entity.
Also, this reminds me of Bryan Caplan’s theory of the ideological spectrum, which is that thinking of yourself as ‘left-wing’ is essentially expressing anti-free-market sentiment, while thinking of yourself as ‘right-wing’ is essentially expressing anti-leftist sentiment.
Both are defined negatively on this account, and so aren’t ideologies especially composed of evidence for the efficacy of particular policies. Both would both fit a huge range of possible policies versus, e.g., the policy interventions implied by academic economics.
Back and Forth
Only make choices that you would not make in reverse, if things were the other way around. Drop out of school if and only if you wouldn’t enroll in school from out of the workforce. Continue school if and only if you’d switch over from work to that level of schooling.
Flitting back and forth between both possible worlds can make you less cagey about doing what’s overdetermined by your world model + utility function already. It’s also part of the exciting rationalist journey of acausally cooperating with your selves in other possible worlds.