I think I am actually more worried about bad actor humans that might leverage AI (well before any ASI exists) to make some type of nasty bio-agent (with or without a counter agent for their own protection or for extorting money from the targeted population). While tracking chemicals is hard I think that would be much easier than trying to track something to signal if someone is bioengineering some thing bad.
But yes, there are a lot of frames to but the question in.
I wonder if two other factors might also work against the 10x growth per year as a norm. One, standard S-curve growth patterns for start-up enterprises. What the time period for the high growth phase might be could be interesting—does rapid advancement of AI itself point to longer or shorter period of revenue growth? My quest is it might shorten it but have not really thought about that before.
The other is just opportunity costs. All that money is coming from somewhere, not thin air, and we have some real constraints on monetary velocity as well. So just how much of the money supply can AI revenue growth curves eat up?
Cheating on homework is somewhat difficult to define perhaps. Is using your calculator cheating when solving a math word problem? Is a lookup for the differential equation or integral cheating because you don’t derive it yourself?
Perhaps it’s a case of stupid homework assignments as one mentioned (my phrasing). I suspect there is also a case of just changes in the world and how one needs to learn to best interact with it. Engineers are taught a lot of math but generally work with both books of tables that provide the answers or with applications that do the same. You just really need to know how to use the tool rather than be able to replicate what the tool does (though if you can you survive without the tool where as the other guy is going to struggle). AI and getting answers to problems, ultimately is what the world is, so learning to work and use the tool correctly, while not yet part of the legacy program, seems to be something a bit different that cheating to me; even if the school or teacher might see it as cheating from a purely definitional perspective.
As a friend of mine back in high school quipped to me one day, being smart is not about what you know, but about knowing where to find what you need to know.
I liked this post. Strong upvote. I’m neutral on the funding/not funding issue but the points you make about not starting with the tool to solve some problem but with the problem and then selecting the tool(s) is so very important. I frequently see that as a problem in so much public debate and policy making.
Thanks for the comment. It does help me understand how some are thinking in terms of post ASI timelines. I think you offer something of a two extremes but also point to some clear aspect one could use for actually estimating end point once ASI starts whatever it’s going to do. The paving over has some physical constraints regardless of how powerful or fast ASI might be. Could be it figures out some material for paving that can be implemented faster than cement but if cement is used we know something about curing times, logistic times from extraction to pour and output rate for quantity flows. That’s a pretty straight calculation to model. Similarly we have some pretty good existing models for food/water requirements, arable land requirements to support life, eco system collapse so I could even probably take your bad case scenario and make a crude estimate of time to extinction.
I was much more interested in getting a human response but finally just asked an AI and got what I think was a good over view and better insight to why it’s not an aspect that is generally included in thinking about that timeline. But again, much more interested in getting a human perspective and thoughts so thanks for taking the time to jot something down.
If you do get around to writing the bit on fire code I would suggest shifting the title a bit. Fire code seems to fit in with a bunch of other regulatory cases, building code, FDA drug regulation, zoning code, etc. So seems hard to call fire code the root of all evil since the same analysis you’re starting with, applied to any of the others, will lead to a similar conclusion.
I also think it’s risky to point to how secondary and tertiary effects might result in off-sets to any welfare gain from the regulation without also offering the net impact or correctly assign origin of harm. For example, size of car is clearing a factor in the outcome of an accident but it’s not really the cause. One should control for other factors, like various source of distracted driving, driver training (I often see people trying to drive a big SVU as if it’s a sport car and know they have no clue about the vehicle dynamics or what the car is going to do if something unexpected happens). I do think it’s fair to say incentives towards larger vehicles results in need for other changes (this is the partial equilibrium versus general equilibrium problem) or impose a ceteris paribus constraint. But I’m not sure it’s fair to relax the immediate ceteris paribus constraint and then not follow through with the more generalized analysis to property locate the source of a follow on problem.
I think I am a prime example of one that has a serious bottleneck in terms of getting things out of my head and on paper when I try. As you say, developing that ability requires some type of “muscles”, just as being able to lift weights or do physically demanding activities. There is also the habit forming aspect. Like noted in one of the comments we forget to do things (often the new things we’re trying to do) because they just don’t fall into our normal behavior patterns[1].
I would also pose a question about slop concerns. Sure, not bad to avoid if one has a specific target they are aiming for I think but then if we look back as history, I wonder how many of the serendipitous discoveries or starting points wouldn’t fall into a slop bucket when views from what the discover was actually trying to do.
Which in the Thinkhaven vein of at least one new thought a day, seem to lead towards “What don’t I know?” What are the gaps in areas one does have some, even a lot, of knowledge about. Seems like a very positive behavioral pattern to cultivate.
My question was about a timeline. Has anyone made any predications about the speed of extinction once ASI exists? I have only see people making claims about the timeline to ASI.
If that was not clear, my bad.
So I should assume that it happens some time before the sun goes nova, as long as we don’t humans don’t escape the the inner solar system or come up with some energy sources that would allow survival after such an event?
Strikes me as a rather tautological type answer but if that is where the AI risk position is I honestly see no reason to get all worked up about the claims.
[This can be ignored—already answered well enough. Kind of a nit but the graph seems a bit confusing. How can GDP per capita continue on it’s trend, presumably with the 2.1% growth bust from the AI gains, while population is crashing? Or is the trend GDP just presuming the implied population changes that were historically embedded in the data series?]
But I would like to know more about the model for the extinction plot. I was just wondering about that views about the path between what you guys call the singularity and when extinction is suppose to occur. I think this is the first time I’ve seen that presented. (But don’t rule out it’s been a lot of placed I’ve just never looked.)
Interesting post and comments. I find myself musing about some adjacent aspects though.
1) Should I take this as a long-term problem of power concentration? If so, concentrated where? If I take much of the whole ASI is incorrigible (and the take-off speed as as some suggest rapid), then I’m not too sure I need to worry about the corporations or governments beyond the very near term. That puts me back in the more general frame of ASI vs humans.
2) It could be a pure power driven bit—Anthropic wants to control the power rather than have it out where anyone can use it which then puts Anthropic at the same risk as anyone. But maybe the announcement is more about business and legal negotiations. Obviously there is a lot of money at stake with Federal contracts and particularly Pentagon related contracts. If the potential for Mythos identifying and then building attack vectors that are very complex and hard to prevent (and possibly even notice) the current administration might be interested in rethinking the requirement to have a human make the final decision on an action that will result in people getting killed. So this might not be a good case study for the concentration of power inference.
I don’t think the “just want their short term returns” applies in many or most cases. I think the influential, large (as in meaningful voting power) tend to take longer term positions. I suspect it comes down more to what the various groups are trying to maximize. Founders and original insiders are probably maximizing a blend of pecuniary and non-pecuniary returns while the post IPO investors tent to be more exclusively focused on the pecuniary returns. But as that pecuniary focused group grows the founder find they have to make more and more compromises to retain control and pursue their initial vision.
But I think you’re right that after IPO the nature of the decision-making and the focus of what to pursue and maximize is going to shift and so relative priority regarding safety likely to shift—exactly how much and in which direction I don’t know but I would not be surprised if it moves in a way you seem concerned about.
Your situation sound a bit similar to my adopted son’s situation with G6PD. One of the things I’ve just learned is how non-uniform that condition is. He’s in the Philippines and they actually screen all new born children for it now. No actual treatments for his situation is available. But since frequency of occurrence in much higher in the Philippines than many other places in the world the Philippine medical infrastructure if pretty much on par with any 1st world county for this condition. And the current “treatment” is avoiding the triggers. If you talked with a Philippine doctor it is very likely they will know “current state” knowledge (maybe not cutting edge though). If I were to start asking doctors here in the USA (and I suspect it would be the same in EU) I suspect most will not really know much and possibly nothing at all.
So one thing to think about when “asking my doctor” is should I consider the question “mainstream” or “cutting edge/far tail of some distribution” and why should I think that.
Is this all something of a poor prompt getting a bad response?
I don’t think you should expect your doctor to know everything you might ask about, and on that agree that asking might not be of immediate value. However, doctors can and do actually research things for their patients (or at least mine seems to).
As I was not present I don’t know how the conversation proceeded but my approach to something like this is more I tell the doctor what I’ve learned and what information has lead to some initial opinion on something. Then ask if they know something more about the situation. I don’t do this often enough to say there is some strong patter but at least I provide a basic framing and common context into which I’m asking them to bring their expertise.
One thing does come to mind on this, but may well be way off target. It seems that a good model for most health and biology questions should fit with some pretty standard chemical reaction science. So ratios should matter a bit. Perhaps it’s as simple as at 1 pill the concentrations is just a poor ratio for the required reaction. As I understand it the current view is (per Gemini)
Inhibition of Sodium Channels: AM404 directly inhibits voltage-gated sodium channels (NaV1.7 and NaV1.8) in peripheral pain-sensing neurons (nociceptors). By blocking these channels, AM404 stops the generation of action potentials (pain messages) at the source before they are sent to the brain.
Perhaps the one pill simple does not allow the body (liver) to convert the paracetamol into enough AM404 to block enough neurons.
I would suggest considering both their naval build up and their position on the South China sea and its historical use of trade/economic and political pressure to advance it’s influence and boarders, with military might as a backup for and enforcement mechanism “By all accounts” is a bit far.
Can work well as long as the enemy don’t have good intel regarding that actual number of trained dogs, even if they cannot identify the specific dogs. But I suspect there are probably ways to get the trained dog to reveal itself without actually giving up the bombs.
The other day I was wondering to myself what might be important to be thinking about. I ended up posing a query to ChatGTP related to that to see what popped out. Two of the questions most interesting to me, and somewhat framed a new meta question for me. Those two questions were:
If knowledge becomes cheap, what becomes scarce?
If AI can do most of the thinking, which complementary human skills become more valuable?
Both questions relate to the deployment and integration of AI and what that might mean for people. The meta-question I had was “What to do in an AI world?” Hardly a new question but it got me thinking about how to try framing any more targeted questions that might yield some actual information or potential plans. I decided I could put the big question into 3 separate buckets, which I labelled base on some generally well known fictions (maybe the first shouldn’t be called fiction but....)
The “If we build it we all die” world
The Middle Earth world, and
The Giskard & Olivaw world
I found, at least for me, 1 and 3 were actually quite boring and generally implied I make, and have no need to make, any real changes in how I approach living my life. The Middle Earth world not only seems to beg answers to the the above two questions.
I jotted down some thought but hardly in an organized way and then though I might post to LW to get some reactions and additional insights/thought if any cared to voice them. Part of that interaction was ChatGTP offering to help refine and reorganize. With a small amount of back and forth I got to what is offered below. I chose to let the AI write it as I really do suck at writing. I don’t think ChatGTP fully grasped some of what I was aiming at. I may continue asking myself, and trying to answer, more on these questions. Dropping some of the suggested areas and maybe adding others.
In any case:
Below is the LLM “slop”, collapsed for those that hate seeing such things. While some of what was generated works well for me and does reflect what I was thinking about some is a bit off the mark. In part I think that is from the desire for some brevity and attempt to elicit other thoughts rather than really push my musing of views at the start.
However, starting with the two question, when knowledge get cheap what gets scarce and when AI can do a lot of thinking what what becomes valuable, I started from a largely economic production model. Things are scarce because few are produced relative to other things and to demand for the item. If some production is used in producing other items, as many are, as it become cheaper to produce more gets produced so one should be able to conclude that those production of the items depending on that now cheaper one will also increase. There is complementarity in production.
But “knowledge” (that’s a tricky term I think) is a bit different from something like steel or leather. What goods don’t have knowledge as a rather important input? So if knowledge is cheap one might say nothing is scarce. A bit simplistic, and incorrect, but also not completely wrong so the question itself needs a bit more examination.
I might put that in some inflation framed case. If all prices are rising at the same rate, then inflation doesn’t matter—or at least the claims are generally it doesn’t matter. In the real world inflation doesn’t manifest itself that was so inflationary periods always impact relative prices, disturb existing equilibria and largely case everyone, not just the lenders, headaches. But we also know that for better or worse people react to the nominal prices, and seem to have some difficulty in recognizing just what the relative price changes actually are. The numbers, not the ratios, have some effect. But that should not really matter for making good economic decisions.
So the cheap knowledge world seem a bit like the uniform inflation world*. If a lot of people are reacting to the nominal “price” it’s probably not the right thing to do. So while in today’s world where we don’t really face what are not largely nominal versus relative type impacts and decision is the this Middle Earth AI world one more like that? If so then the skills we don’t have or don’t often use that help us recognize cases of nominal type changes rather than relative type changes maybe matter.
That’s all rather abstract as I don’t think the nominal-relative economic metaphor does a great job illustrating the point but it’s the best I can come up with right now.
Going with the knowledge is an input to everything so that AI world is one a abundance, time probably becomes more scarce. Or perhaps more correct, assessing what to do with our time becomes more problematic and expensive (more scarce?) because we find we now have many more, and perhaps new unexperienced, options to spend our time on.
With that type of problem understanding oneself seems to become more and more important. It’s hard to make a good decision about what to do if you don’t really understand deeply what it is that is important and of value to you. That may be counter-intuitive or it may be incorrect. With fewer options we still benefit greatly by understanding what really drives us personally. But I might also find that if I don’t have too many options the ones that will be better for me may well be more obvious and not required much soul-searching so I can shirk a bit regarding effort to think about who and what I am. Does the knowledge we’re talking about in an AI world reduce the costs there or is that a bit outside what an AI can even offer? Perhaps the skills and effort towards introspection become more valuable and in a very real sense more scarce in that middle AI world.
[* The intuition here is that in a monetary case, central banks are the source of money that enables most inflation. Injection effects and transmission paths impact the relative observance of the increasing prices. So that uniform inflation world just cannot exist for us. But cheap knowledge from AI is different. The AI might be central (and perhaps even restricted which breaks this thought) but access is open. Everyone has easy access to the knowledge now. This significantly changes the dynamics and I think flattens the case—so becomes much more like the uniform inflation case.]
Below is the LLM output.
ChatGTP edit and rewrite from the chat
When thinking about AI’s long-term effects, it helps to distinguish between a few different possible futures. For the purposes of this discussion, I find it useful to divide them into three broad categories:
1. The “If We Build It We All Die” world
2. Middle Earth
3. The Giskard & Olivaw world
The names come from literature, though I’m not claiming the futures would resemble those stories in detail.
Interestingly, the two extremes are the least interesting for practical purposes.
If the first scenario is correct—if sufficiently powerful AI leads to human extinction—then there is not much to say about life in an AI world. In that case there *is no* meaningful AI world for humans. The only world that matters is the pre-AI world we live in now. And realistically there is little for most individuals to do about it except continue living their lives. Preventing AI entirely seems implausible. Work on safety and alignment may improve outcomes, but ordinary people have limited direct influence over that trajectory.
The opposite extreme is what I’ll call the Giskard & Olivaw world, after the robots in Isaac Asimov’s novels. Those robots could read minds and predict the future with remarkable accuracy, and they used those abilities to subtly guide humanity away from civilization-ending disasters. Humans retained a large degree of freedom, but the worst outcomes were quietly prevented and in general people lived very material rich and easier lives from AI and robot. In the extreme one need not do anything to improve oneself and still enjoy a very comfortable living. (Perhaps one needs to have some self motivation and curiosity about things but the costs on being lazy is very low and perhaps many follow that bad incentive structure.)
In a world like that, AI would effectively manage humanity’s largest risks while enabling enormous abundance. For most people, life would largely continue as it does today—except materially better.
In both extreme scenarios, the actions of ordinary individuals matter relatively little. Either AI destroys us, or it solves our largest problems.
The interesting case lies between these extremes.
I’ll call this Middle Earth: a world where AI becomes extremely powerful but does not completely determine humanity’s fate. Humans still make choices, compete for status, build institutions, and shape their societies—but they do so within an environment profoundly altered by AI.
Thinking about life in such a world raises many questions. Two in particular seem especially interesting:
1. If knowledge becomes cheap, what becomes scarce?
2. If AI can do most of the thinking, which complementary human skills become more valuable?
These questions may help illuminate what life in a “Middle Earth” AI world might actually look like.
If Knowledge Becomes Cheap, What Becomes Scarce?
One natural starting point is economic intuition. If a production process becomes dramatically cheaper, then the goods produced by that process generally become cheaper as well.
If AI drastically reduces the cost of producing knowledge—analysis, explanations, designs, plans, code—then many things that depend on knowledge production should also become cheaper.
At first glance this suggests a world of broad abundance. But that observation alone isn’t very informative. A more useful question may be:
Which things remain scarce even when knowledge production becomes cheap?
Several possibilities come to mind.
Time
No matter how much knowledge AI can produce, humans still face the same constraint: 24 hours in a day.
In fact, AI may increase the scarcity of time. As more activities become accessible—because AI can help us perform them—the number of possible ways to spend time expands. That increases opportunity cost. Choosing how to spend time may become more difficult precisely because more options become feasible.
This suggests that understanding our own preferences—what we actually want to do—may become more valuable rather than less.
Attention
Attention may become scarce for related reasons.
If AI dramatically expands the set of things we *could* pay attention to—ideas, projects, entertainment, opportunities—then managing attention becomes harder.
There is already a hint of this dynamic in modern politics. Participating meaningfully as an informed citizen has become more difficult as the complexity of society increases. People often respond by delegating attention—to institutions, media, or trusted intermediaries.
An AI world might amplify this pattern. One possibility is that people increasingly defer judgment to AI systems. Another possibility is that managing one’s attention becomes an increasingly important personal skill.
Trust and Verification
If AI systems can produce convincing text, images, and analysis at very low cost, then verification may become more valuable.
Knowing whether something is authentic—whether a claim, piece of information, or analysis is reliable—could become an important scarce resource.
Reputation systems, trusted institutions, and mechanisms for verification might become more important rather than less.
Taste and Curation
When knowledge production is expensive, expertise often lies in creating answers.
When knowledge production becomes cheap, expertise may shift toward deciding which answers matter. [Note: I think this is a weak framing by the AI, which answers one seeks in a high-cost knowledge world certainly is a very important problem that must be solve, and is only a little easier if you have fewer alternatives to consider.]
In other words, taste and curation.
Choosing what is worth doing, what is worth building, or what problems are actually important may become more valuable than producing solutions to well-defined problems.
This may be related to what some people have called *context engineering*: framing problems in ways that allow powerful AI systems to produce useful outputs.
Status
Another possibility is status.
Historically, knowledge has often been a path to status. Experts, scholars, advisors, and professionals derive part of their status from possessing knowledge that others lack.
If AI systems give everyone easy access to sophisticated analysis and explanations, that particular path to status may weaken.
But humans rarely stop caring about status simply because one route to it disappears. If knowledge ceases to differentiate people, status may shift toward other dimensions.
Possibilities might include:
* judgment (knowing what questions to ask)
* taste (choosing worthwhile goals)
* character and reliability
* leadership and coordination
* physical or embodied achievements
If this is correct, then the social role of expertise may change significantly even if knowledge itself becomes abundant.
Which Human Skills Become More Valuable?
The second question follows naturally from the first.
If AI systems can perform much of the cognitive work currently done by humans—analysis, planning, explanation—then what human abilities become more valuable in that environment?
One candidate is problem framing.
If AI systems are powerful tools for solving problems, then much of the value may lie in identifying which problems to solve and how to formulate them. The ability to translate messy real-world situations into well-posed problems might become increasingly important.
Another candidate is judgment under uncertainty.
Even if AI systems can generate sophisticated analyses, humans may still need to decide which analyses to trust, which goals to pursue, and which tradeoffs are acceptable. These decisions often involve values rather than purely technical reasoning.
Coordination and leadership may also become more valuable. Many important problems require groups of people to align around shared goals. AI may assist with analysis and planning, but motivating and organizing humans may remain a distinctly human domain.
Finally, there may be increasing value in understanding humans themselves—their motivations, values, and preferences. If AI systems can handle many technical tasks, the limiting factor in many projects may become human agreement about what should be done. [Note, here again I think the AI missed some of what I was thinking. To me what might be at least as important, and certainly so in terms of our individual lives, is understanding ourselves. Having seen the post this morning about noticing your own state of feeling seems it is at least close to some of what I was actually thinking. But still the social interactions will matter too—so you end up with things like specializing in context engineering or the like.]
Okay. That is something I didn’t know but fire codes are hardly the only standards that are generated via private organizations, but they all have to be implemented and enforced by governments. Without acceptance by the government with jurisdiction they are just words on paper.
But I don’t think that changes the claim that fire code is just one instances of a class and probably should not be singled out as something special unless it can be demonstrated as special. But one can look at the whole class and see similar problems of unintended consequences and special/inside interest incentives when trying to figure out if the over all results of any rule generation by some group is a new plus or negative. In fact I think knowing now that there are multiple layers between fire code design and implementation I have more concerns about labeling it as the root of all evil, seems more inflammatory than accurate.