The hackernews discussion (https://news.ycombinator.com/item?id=44877076) is really disappointing. The top comment criticizes rationalists for the opposite I know them for.
Edit: Top comment changed. It was the one by JohnMakin.
osten
Karma: 74
An AI has just prevented its host from shutting it off and this incident is now part of the training data for future models. Solve for the equilibrium.
I agree, I’m probably not as sure about sufficient alignment but yes.
I suppose this also assumes a kind of orderly world where it actually is within the means of humanity, AGIs (within their Molochian frames), and trivial means of later superintelligences to preserve humans. (US office construction spending and data center spending are about to cross https://x.com/LanceRoberts/status/1953042283709768078 .)
Thanks for the reply, I have gripes with
analogy doesn’t by itself seem compelling, given that humanity as a whole (rather than particular groups within it or individuals) is a sufficiently salient thing in the world
etc. because don’t you think humanity from the point of view of ASI at the ‘branch point’ of deciding its continued existence may well be on the order of importance of an individual to a billionaire?
Agree but again, we don’t get to choose what existence means.
Yes and my reply to that (above) is humanity has a bad track record at that so why would AIs trained on human data be better? Think also of indigenous peoples, extinct species humans didn’t care enough about etc. The point also in the Dyson sphere parabel is not wanting something, it’s wanting something enough so that it happens.
since the necessary superintelligent infrastructure would only take a fraction of the resources allocated to the future of humanity.
I’m not sure about that and the surrounding argument. I find Eliezer’s analogy compelling here: When constructing a Dyson sphere around the sun, leaving just a tiny sliver of light enough for earth would correspond to a couple of dollars of the wealth of a contemporary billionaire. Yet you don’t get these couple of dollars.
(This analogy has caveats like Jeff Bezos lifting the Apollo 11 rocket motors from the ocean ground and giving them to the Smithsonian, which should be worth something to you. Alas it kinda means you don’t get to choose what you get. Maybe it is storage space for your brain scan like in AI 2027.)
Plus spelling out the Dyson sphere thing: The superintelligent infrastructure should highly likely by default get in the way of humanity’s existence at some point. At this point the AIs will have to consciously make a decision to avoid that at some cost to them. Humanity has a bad track record at doing that (not completely sure here but thinking of e.g. Meta’s effect on wellbeing of teenage girls). So why would AIs be more willing to do that?
Now I am (more) curious about that TheZvi Claude system prompt.
Maybe this is related: A crucial step in the workflow of Getting Things Done is to clarify the task. Many of the tasks you mention are not clearly specified. I suppose morphing them into questions means that the task becomes to first clarify the task.
A counterpoint may be that human civilization is just the observable of underlying optimization processes (individual and group selection on several levels). Obviously this observable has had variance whenever a new advantage got widely adopted and civilization morphed into the memeplex of whoever had the advantage. So from AI we shouldn’t expect this process to suddenly stop. However, the entities with the next advantage may well be agentic AIs.
Probably not saying something new here, but what is the probability of you living at a certain point in the evolution of the universe? Because of grabby aliens and the likelihood that grabby aliens don’t have human-like minds you’re not likely to live late in the history of the universe but rather early when the life-density of the universe is largest but the various life forms haven’t propagated their grabby variants yet.
Have we just discovered one of the actual reasons for the peer-review process sticking around? It forces CoTs to be legible unlocking secondary value?
Take into account exponential compute scaling and exponential efficiency scaling.
NP-hard problems have no known efficient algorithms, strong evidence suggests none exist
Can you back up this claim? I find the evidence I have seen pretty weak in a mathematical sense.
Yes, but it still makes sense to me. RL and ‘llm thinking’ are pretty basic heuristics at this point and I don’t expect them to circumvent exponential running time requirements for many problems they implicitly solve at the same time.
Ok, but why do you think that AIs learn skills at a constant rate? Might it be that higher level skills need more time to learn because compute scales exponentially with time but for higher level skills data is exponentially more scarce and needs linearly in task length more context, that is, total data processed scales superexponentially with task level?
I think this is relevant: https://slatestarcodex.com/2017/07/24/targeting-meritocracy/ . ‘Prevalence and tendency implies morality’ - I don’t think that’s an argument that people gestured at here try to make.
Orange peel is a standard ingredient in Chinese cooking. Just be careful with pesticides.
Why wouldn’t AI agents or corporations led by AIs develop their own needs and wants, the way corporations do that as well currently? An economy has no need for humans, it only needs needs and scarce potential to meet those needs.
They released the new models and updated apps in tranches.