Agreed. To give a concrete toy example: Suppose that Luigi always outputs “A”, and Waluigi is {50% A, 50% B}. If the prior is {50% luigi, 50% waluigi}, each “A” outputted is a 2:1 update towards Luigi. The probability of “B” keeps dropping, and the probability of ever seeing a “B” asymptotes to 50% (as it must).
This is the case for perfect predictors, but there could be some argument about particular kinds of imperfect predictors which supports the claim in the post.
I’ve been asked to clarify a point of fact, so I’ll do so here:
My recollection is that he probed a little and was like “I’m not too worried about that” and didn’t probe further.
This does ring a bell, and my brain is weakly telling me it did happen on a walk with Nate, but it’s so fuzzy that I can’t tell if it’s a real memory or not. A confounder here is that I’ve probably also had the conversational route “MIRI burnout is a thing, yikes” → “I’m not too worried, I’m a robust and upbeat person” multiple times with people other than Nate.
In private correspondence, Nate seems to remember some actual details, and I trust that he is accurately reporting his beliefs. So I’d mostly defer to him on questions of fact here.
I’m pretty sure I’m the person mentioned in TurnTrout’s footnote. I confirm that, at the time he asked me, I had no recollection of being “warned” by Nate but thought it very plausible that I’d forgotten.
Does this work? -- https://tinyurl.com/pay-Viv-H
In your view, who would contribute more to science -- 1000 Einsteins, or 10,000 average scientists?[1]
“IQ variation is due to continuous introduction of bad mutations” is an interesting hypothesis, and definitely helps save your theory. But there are many other candidates, like “slow fixation of positive mutations” and “fitness tradeoffs[2]”.
Do you have specific evidence for either:
Deleterious mutations being the primary source of IQ variation
Human intelligence “plateauing” around the level of top humans[3]
Or do you believe these things just because they are consistent with your learning efficiency model and are otherwise plausible?[4]
Maybe you have a very different view of leading scientists than most people I’ve read here? My picture here is not based on any high-quality epistemics (e.g. it includes “second-hand vibes”), but I’ll make up some claims anyway, for you to agree or disagree with:
There are some “top scientists” (like Einstein, Dirac, Von Neumann, etc). Within them, much of the variance in fame is incidental, but they are clearly a class apart from merely 96th percentile scientists. 1000 {96%-ile-scientists} would be beaten by 500 {96%-ile-scientists} + 100 Einstein-level scientists.
Even within “top scientists” in a field, the best one is more than 3x as intrinsically productive[5] as the 100th best one.
I’m like 90% on the Einsteins for theoretical physics, and 60% on the Einsteins for chemistry
Within this, I could imagine anything from “this gene’s mechanism obviously demands more energy/nutrients” to “this gene happens to mess up some other random thing, not even in the brain, just because biochemistry is complicated”. I have no idea what the actual prevalence of any of this is.
What does this even mean? Should the top 1/million already be within 10x of peak productivity? How close should the smartest human alive be to the peak? Are they nearly free of deleterious mutations?
I agree that they are consistent with each other and with your view of learning efficiency, but am not convinced of any of them.
“intrinsic” == assume they have the same resources (like lab equipment and junior scientists if they’re experimentalists)
You could use all of world energy output to have a few billion human speed AGI, or a millions that think 1000x faster, etc.
Isn’t it insanely transformative to have millions of human-level AIs which think 1000x faster?? The difference between top scientists and average humans seems to be something like “software” (Einstein isn’t using 2x the watts or neurons). So then it should be totally possible for each of the “millions of human-level AIs” to be equivalent to Einstein. Couldn’t a million Einstein-level scientists running at 1000x speed could beat all human scientists combined?
And, taking this further, it seems that some humans are at least 100x more productive at science than others, despite the same brain constraints. Then why shouldn’t it be possible to go further in that direction, and have someone 100x more productive than Einstein at the same flops? And if this is possible, it seems to me like whatever efficiency constraints the brain is achieving cannot be a barrier to foom, just as the energy efficiency (and supposed learning optimality?) of the average human brain does not rule out Einstein more than 100x-ing them with the same flops.
Some problems with the power law extrapolation for GDP:
The graph is for the whole world, not just the technological leading edge, which obscures the thing which is conceivably relevant (the endogenous trend in tech advancement at the leading edge)
The power law model is a bad fit for the GDP per capita of the first world in the last 50-100 years
Having built a toy endogenous model of economic growth, I see no gears-level reason to expect power law growth in our current regime. (Disclaimer: I’m not an economist, and haven’t tested my model on anything.) The toy model presented in the OpenPhil report is much simpler and IMO less realistic.
It would help if you specified which subset of “the community” you’re arguing against. I had a similar reaction to your comment as Daniel did, since in my circles (AI safety researchers in Berkeley), governance tends to be well-respected, and I’d be shocked to encounter the sentiment that working for OpenAI is a “betrayal of allegiance to ‘the community’”.
Consider this quote by Alan Watts:
There are basically two kinds of philosophy. One’s called prickles, the other’s called goo. And prickly people are precise, rigorous, logical. They like everything chopped up and clear. Goo people like it vague. For example, in physics, prickly people believe that the ultimate constituents of matter are particles. Goo people believe it’s waves.
*facepalm*
If one has a technical understanding of QFT[1] (or even half of technical understanding, like me), this sounds totally silly. There’s no real question as to whether things are fundamentally “particles” or “waves”. There’s nothing “goo” about waves either; waves refer to mathematically precise phenomena. Physics is precise and mathematical (“prickly” if you like). Anyone who understands QFT must have a prickly side, and can deal with both waves and particles without confusion. Those who think physics can be “vague” don’t understand the physics.
(Critiquing Watts of course, not primarily the author of this post)
Quantum field theory
It’s sad that agentfoundations.org links no longer work, leading to broken links in many decision theory posts (e.g. here and here)
Cool! It wrote and executed code to solve the problem, and it got it right.
Are you using chat-GPT-4? I thought it can’t run code?
In ML terms, nearly-all the informational work of learning what “apple” means must be performed by unsupervised learning, not supervised learning. Otherwise the number of examples required would be far too large to match toddlers’ actual performance.
I’d guess the vast majority of the work (relative to the max-entropy baseline) is done by the inductive bias.
Another way of looking at this question: Arithmetic rationality is shift invariant, so you don’t have to know your total balance to calculate expected values of bets. Whereas for geometric rationality, you need to know where the zero point is, since it’s not shift invariant.
If the system is modular, such that the part of the system representing the goal is separate from the part of the system optimizing the goal, then it seems plausible that we can apply some sort of regularization to the goal to discourage it from being long term.
What kind of regularization could this be? And are you imagining an AlphaZero-style system with a hardcoded value head, or an organically learned modularity?
Presumably a Bayesian reasoner using expected value would never reach max utility, because there would always be a non-zero probability that the goal hasn’t been achieved, and the course of action which increases its success estimate from 99.9999% to 99.99999999% probably involves turning part of the universe into computronium.
POV: I’m in an ancestral environment, and I (somehow) only care about the rewarding feeling of eating bread. I only care about the nice feeling which comes from having sex, or watching the birth of my son, or being gaining power in the tribe. I don’t care about the real-world status of my actual son, although I might have strictly instrumental heuristics about e.g. how to keep him safe and well-fed in certain situations, as cognitive shortcuts for getting reward (but not as terminal values).
Would such a person sacrifice themselves for their children (in situations where doing so would be a fitness advantage)?
??? For math this is exactly backward, there can be true-but-unprovable statements
ETA: Koen recommends reading Counterfactual Planning in AGI Systems before (or instead of) Corrigibility with Utility Preservation
Update: I started reading your paper “Corrigibility with Utility Preservation”.[1] My guess is that readers strapped for time should read {abstract, section 2, section 4} then skip to section 6. AFAICT, section 5 is just setting up the standard utility-maximization framework and defining “superintelligent” as “optimal utility maximizer”.
Quick thoughts after reading less than half:
AFAICT,[2] this is a mathematical solution to corrigibility in a toy problem, and not a solution to corrigibility in real systems. Nonetheless, it’s a big deal if you have in fact solved the utility-function-land version which MIRI failed to solve.[3] Looking to applicability, it may be helpful for you to spell out the ML analog to your solution (or point us to the relevant section in the paper if it exists). In my view, the hard part of the alignment problem is deeply tied up with the complexities of the {training procedure --> model} map, and a nice theoretical utility function is neither sufficient nor strictly necessary for alignment (though it could still be useful).
So looking at your claim that “the technical problem [is] mostly solved”, this may or may not be true for the narrow sense (like “corrigibility as a theoretical outer-objective problem in formally-specified environments”), but seems false and misleading for the broader practical sense (“knowing how to make an AGI corrigible in real life”).[4]
Less important, but I wonder if the authors of Soares et al agree with your remark in this excerpt[5]:
“In particular, [Soares et al] uses a Platonic agent model [where the physics of the universe cannot modify the agent’s decision procedure] to study a design for a corrigible agent, and concludes that the design considered does not meet the desiderata, because the agent shows no incentive to preserve its shutdown behavior. Part of this conclusion is due to the use of a Platonic agent model.”
Btw, your writing is admirably concrete and clear.
Errata: Subscripts seem to broken on page 9, which significantly hurts readability of the equations. Also there is a double-typo “I this paper, we the running example of a toy universe” on page 4.
Assuming the idea is correct
Do you have an account of why MIRI’s supposed impossibility results (I think these exist?) are false?
I’m not necessarily accusing you of any error (if the contest is fixated on the utility function version), but it was misleading to me as someone who read your comment but not the contest details.
Portions in [brackets] are insertions/replacements by me
What problems did you solve, from which you are extracting these lessons?
My solution (rot13′d): “Vs lbh nfxrq zr vs V nz fvatyr, V jbhyq fnl lrf”
Isn’t going from an average human to Einstein a huge increase in science-productivity, without any flop increase? Then why can’t there be software-driven foom, by going farther in whatever direction Einstein’s brain is from the average human?