My government name is Mack Gallagher. Crocker’s Rules. I am an “underfunded” “alignment” “researcher”. DM me if you’d like to fund my posts, or my project.
I post some of my less-varnished opinions on my Substack, and my personal blog.
If you like arguing with me on LessWrong, at present I’m basically free round the clock to continue interesting arguments in my Discord.
Wild ahead-of-of-time guess: the true theory of gravity explaining why galaxies appear to rotate too slowly for a square root force law will also uniquely explain the maximum observed size of celestial bodies, the flatness of orbits, and the shape of galaxies.
Epistemic status: I don’t really have any idea how to do this, but here I am.
When I stated Principle (A) at the top of the post, I was stating it as a traditional principle of economics. I wrote: “Traditional economics thinking has two strong principles, each based on abundant historical data”,
I don’t think you think Principle [A] must hold, but I do think you think it’s in question. I’m saying that, rather than taking this very broad general principle of historical economic good sense, and giving very broad arguments for why it might or might not hold post-AGI, we can start reasoning about superintelligent manufacturing [including R&D] and the effects it will have, more granularly, out the gates.
Like, with respect to Principle [C] my perspective is just “well of course the historical precedent against extremely fast economic growth doesn’t hold after the Singularity, that’s more or less what the Singularity is”.
Edit: Your rewrite of Principle [B] did make it clear to me that you’re considering timelines that are at least somewhat bad for humans; thank you for the clarification. [Of course I happen to think we can also discard “AI will be like a manufactured good, in terms of its effects on future prices”, out the gates, but it’s way clearer to me now that the trilemma is doing work on idea-space.]
You seem to have misunderstood my text. I was stating that something is a consequence of Principle (A),
My position is that if you accept certain arguments made about really smart AIs in “The Sun is Big”, Principle A, by itself, ceases to make sense in this context.
costs will go down. You can argue that prices will equilibriate to costs, but it does need an argument.
Assuming constant demand for a simple input, sure, you can predict the price of that input based on cost alone. The extent to which “the price of compute will go down”, is rolled in to how much “the cost of compute will go down”. But IIUC, you’re more interested in predicting the price of less abstract assets. Innovation in chip technology is more than just making more and more of the same product at a lower cost. [ “There is no ‘lump of chip’.” ] A 2024 chip is not just [roughly] 2^10 2004 chips—it has logistical advantages, if nothing else. And those aren’t accounted for if you insist on predicting chip price using only compute cost and value trendlines. Similar arguments hold for all other classes of material technological assets whose value increases in response to innovation.
“AI will [roughly] amount to X”, for any X, including “high-skilled entrepreneurial human labor” is a positive claim, not a default background assumption of discourse, and in my reckoning, that particular one is unjustified.
Thank you for writing this and hopefully contributing some clarity to what has been a confused area of discussion.
So here’s a question: When we have AGI, what happens to the price of chips, electricity, and teleoperated robots?
(…Assuming free markets, and rule of law, and AGI not taking over and wiping out humanity, and so on. I think those are highly dubious assumptions, but let’s not get into that here!)
Principle (A) has an answer to this question. It says: prices equilibrate to marginal value, which will stay high, because AGI amounts to ambitious entrepreneurial skilled labor, and ambitious entrepreneurial skilled labor will always find more new high-value things to do. That, incidentally, implies that human labor will retain a well-paying niche—just as less-skilled labor today can still get jobs despite more-skilled labor also existing.
First off, I’m guessing you’re familiar with the economic arguments in The Sun is Big.
Secondly -
If we’re talking about prices for the same chips, [rate of] electricity, teleoperated robots, etc., of course they’ll go down, as the AGI will have invented better versions.
AGI amounts to ambitious entrepreneurial skilled labor
This is really just a false thing to believe about AGI, from us humans’ perspective. It amounts to a new world political order. Unless you specifically build it to prevent all other future creations of humanity from becoming politically interventionist superintelligences, while also not being politically interventionist itself.
I didn’t say he wasn’t overrated. I said he was capable of physics.
Did you read the linked post? Bohm, Aharonov, and Bell misunderstood EPR. Bohm’s and Aharonov’s formulation of the thought experiment is easier to “solve” but does not actually address EPR’s concern, which is that mutual non-commutation of x-, y-, and z-spin implies hidden variables must not be superfluous. Again, EPR were fine with mutual non-commutation, and fine with entanglement. What they were pointing out was that the two postulates don’t make sense in each other’s presence.
Your linked post on The Obliqueness Thesis is curious. You conclude thus:
Obliqueness obviously leaves open the question of just how oblique. It’s hard to even formulate a quantitative question here. I’d very intuitively and roughly guess that intelligence and values are 3 degrees off (that is, almost diagonal), but it’s unclear what question I am even guessing the answer to. I’ll leave formulating and answering the question as an open problem.
I agree, values and beliefs are oblique. The 3 spatial dimensions are also mutually oblique, as per General Relativity. A theory of obliqueness is meaningless if it cannot specify the angles [ I think in a correct general linear algebra, everything would be treated as [at least potentially] oblique to everything else, but that doesn’t mean I refuse to ever treat the 3 spatial dimensions as mutually orthogonal ].
As with the 3 spatial dimensions in practical ballistics, with the value dimension and the belief dimension in practical AI alignment, there are domains of discussion where it is appropriate to account for the skew between the dimensions and domains where it is appropriate to simply treat them as orthogonal. Discussions of alignment theory such as the ones in which you seek to insert your Obliqueness thesis, are a domain in which the orthogonality assumption is appropriate. We cannot guess at the skew with any confidence in particular cases, and with respect to any particular pre-chosen utility/valence function term versus any particular belief-state [e.g. “what is the dollar value of a tulip?” • “is there a teapot circling Mars?”], the level of skew is almost certain to be negligible.
Planning for anthropically selected futures, on the other hand, is a domain where the skew between values and beliefs becomes relevant. There is less point reasoning in detail about pessimized futures or others that disagree with our values [such as the ones in which we are dead], no matter how likely or unlikely they might be “in a vacuum”, if we’re trying to hyperstition our way into futures we like. But this is an esoteric and controversial argument and not actually required to justify why I don’t think it’s useful to consider [sufficiently] strong AI as “what can eat the sun”.
All that’s required to justify why I don’t think it’s useful to consider [sufficiently] strong AI as “what can eat the sun”, is that what you propose is a benchmark of capability, or intelligence. Benchmarks of intelligence [say, of bureaucrats or chimps] are not questions of fact. They are social fictions chosen for their usefulness. If, in the vast, vast supermajority of the worlds where the benchmark would otherwise be useful—in this case, the worlds where people deploy an AI that they do not know, ahead of time, if it will or will not be strong enough to eat the Sun—it will not be useful for contingent reasons—in this case, because we are all dead—then it is not, particularly, a benchmark we should be etching into the wood, from our present standpoint.
The characters don’t live in a world where sharing or smoothing risk is already seen as a consensus-valuable pursuit; thus, they will have to be convinced by other means.
I gave their world weirdly advanced [from our historical perspective] game theory to make it easier for them to talk about the question.
Cowen, like Hanson, discounts large qualitative societal shifts from AI that lack corresponding contemporary measurables.
Einstein was not an experimentalist, yet was perfectly capable of physics; his successors have largely not touched his unfinished work, and not for lack of data.
Not to be too cheeky, the idea is that if we understand insurance, it should be easy to tell if the characters’ arguments are sound-and-valid, or not.
The obtuse bit at the beginning was an accidental by-product of how I wrote it; I admittedly nerdsniped myself trying to find the right formula.
Dwarkesh asks, what would happen if the world population would double? Tyler says, depends what you’re measuring. Energy use would go up.
Yes, economics after von Neumann very much turned into a game of “don’t believe in anything you can’t already comparatively quantify”. It is supremely frustrating.
On that note, I’d also point everyone to Dwarkesh Patel’s other recent podcast, which was with physicist Adam Brown. It repeatedly blew my mind in the best of ways, and I’d love to be in a different branch where I had the time to dig into some of the statements here. Physics is so bizarre.
You just inspired me to go listen myself. Maybe we should all take a node out of that branch. Unfortunately physics has suffered similar issues.
Last big piece: if one were to recruit a bunch of physicists to work on alignment, I think it would be useful for them to form a community mostly-separate from the current field. They need a memetic environment which will amplify progress on core hard problems, rather than… well, all the stuff that’s currently amplified.
Yes, exactly. Unfortunately, actually doing this is impossible, so we all have to keep beating our heads against a wall just the same.
This does not matter for AI benchmarking because by the time the Sun has gone out, either somebody succeeded at intentionally engineering and deploying [what they knew was going to be] an aligned superintelligence, or we are all dead.
Are you familiar with CEV?
I’m willing to credit that increased velocity of money by itself made aristocracy untenable post-industrialization. Increased ease of measurement is therefore superfluous as an explanation.
Why believe we’ve ever had a meritocracy—that is, outside the high [real/underlying] money velocities of the late 19th and early 20th centuries [and the trivial feudal meritocracy of heredity]?
0 trips → 1 trip is an addition to the number of games played, but it’s also an addition to the percentage of income bet on that one game—right?
Dennis is also having trouble understanding his own point, FWIW. That’s how the dialogue came out; both people in that part are thinking in loose/sketchy terms and missing important points.
The thing Dennis was trying to get at by bringing up the concrete example of an optimal Kelly fraction is that it doesn’t make sense for willingness to make a risky bet to have no dependence on available capital; he perceives Jill as suggesting that this is the case.
Relevant to whether the hot/cold/wet/dry system is a good or a bad idea, from our perspective, is that doctors don’t currently use people’s star signs for diagnosis. Bogus ontologies can be identified by how they promise to usefully replace more detailed levels of description—i.e., provide a useful abstraction that carves reality at the joints—and yet don’t actually do this, from the standpoint of the cultures they’re being sold to.
To your first question:
I honestly think it would be imprudent of me to give more examples of [what I think are] ontology pyramid schemes; readers would either parse them as laughable foibles of the outgroup, and thus write off my meta-level point as trivial, or feel aggressed and compelled to marshal soldiers against my meta-level point to dispute a perceived object-level attack on their ingroup’s beliefs.
I think [something like] this [implicit] reasoning is likely why the Sequences are around as sparse as this post is on examples, and I think it’s wise that they were written that way.
To your second:
There’s nothing logically against an ontology that’s true being the subject of a pyramid scheme, any more than there’s anything logically against the bogus widgets a physical pyramid scheme is selling, actually being inordinately useful. It just generally doesn’t happen.
I think you’re right, I had misunderstood! Kind of an egregious misunderstanding, too.
I’m curious how it seems to you to be fundamental to the post, maybe I missed something on that count. I’m planning on replacing ‘shouldn’t scale . . . proportionally’ with ‘shouldn’t scale more than proportionally’, and I don’t see how that substantially changes anything, given that Jill is right, and the concept of a Kelly fraction isn’t applicable when you’re starting out with zero capital of your own to gamble.
Edit: Ah, whoops, yes, that’d be the old screwed-up version of the formatting. It must not have updated on your end yet. [I’m using Firefox too.]
For me, the code block stops right before “Moriarty looks at the paper”, and the Markdown hyperlinks are displaying properly.
This trilemma might be a good way to force people-stuck-in-a-frame-of-traditional-economics to actually think about strong AI. I wouldn’t know; I honestly haven’t spent a ton of time talking to such people.
Principle [A] doesn’t just say AIs won’t run out of productive things to do; it makes a prediction about how this will affect prices in a market. It’s true that superintelligent AI won’t run out of productive things to do, but it will also change the situation such that the prices in the existing economy won’t be affected by this in the normal way prices are affected by “human participants in the market won’t run out of productive things to do”. Maybe there will be some kind of legible market internal to the AI’s thinking, or [less likely, but conceivable] a multi-ASI equilibrium with mutually legible market prices. But what reason would a strongly superintelligent AI have to continue trading with humans very long, in a market that puts human-legible prices on things? Even in Hanson’s Age of Em, humans who choose to remain in their meatsuits are entirely frozen out of the real [emulation] economy, very quickly in subjective time, and to me this is an obvious consequence of every agent in the market simply thinking and working way faster than you.