AnthonyC

Karma: 3,428

AnthonyC 30 Sep 2025 2:32 UTC
2 points
0
in reply to: jchan’s comment on: How does the current AI paradigm give rise to the “superagency” that IABIED is concerned with?
But this is already presupposing the existence of the superintelligence whose feasibility we are trying to explain.
Strictly speaking I only presupposed an AI could reach close to the limits of human intelligence in terms of thinking ability, but with the inherent speed and parallelizability and memory advantages of a digital mind.
Do you have any examples handy of AI being successful at real-world goals?
In small ways (aka sized appropriately for current AI capabilities) this kind of thing shows up all the time in chains of thought in response to all kinds of prompts, to the point that no, I don’t have specific examples, because I wouldn’t know how to pick one. The one that first comes to mind, I guess, was using AI to help me develop a personalized nutrition/supplement/weight loss/training regimen.
Stepping back, I should reiterate that I’m talking about “the current AI paradigm”
That’s fair, and a reasonable thing to discuss. After all, the fundamental claim of the book’s title is about a conditional probability: IF it turns out that the anything like our current methods scale to superintelligent agents, we’d all be screwed.

AnthonyC 29 Sep 2025 17:28 UTC
7 points
3
on: How does the current AI paradigm give rise to the “superagency” that IABIED is concerned with?
I sincerely hope that if anyone has a concrete, actionable answer to this question, that they’re smart enough not to share it publicly, for what I hope are obvious reasons.
But aside from that caveat, I think you are making several incorrect assumptions.
1. “There is no massive corpus of such strategies that can be used as training data”
  1. The AI has, at minimum access-in-principle to everything that has ever been written or otherwise recorded, including all fiction, all historical records, and all analysis of both of those. This includes many, many, many examples and discussions of plans, successful and not, and detailed discussions of why humans believe they succeeded or failed.
2. “(a) doing real-world experiments (whereby generating sufficient data would be far too slow and costly, or simply impossible)”
  1. People have already handed substantial amounts of crypto to at least one AI, which it can use to autonomously act in the real world by paying humans. What do you see as the upper bound on this, and why?
  2. I think most people greatly overestimate how much of this is actually needed for many kinds of goals. What do you see as the upper bound for what can, in principle, be done with a plan that an army of IQ-180 humans (aka no better qualitative thinking than what the smartest humans can do, so that this is a strict lower bound on ASI capabilities) came up with over subjective millennia with access to all recorded information that currently exists in the world? Assume the plan includes the capability to act in parallel, at scale, and the ability to branch its actions based on continued observation, just like groups of humans can, but with much better coordination within the group.
3. “(b) a comprehensive world-model that is capable of predicting the results of proposed actions”
  1. See above—I’m not sure what you see as the upper bound for how good such a world model can or would likely be?
  2. One answer is “Because we’re going to have long since handed it thousands to billions bodies to operate in the world, and problems to come up with plans to solve, and compute to use to execute and revise those plans.” Without the bodies, we’re already doing this.
  3. Current non-superintelligent AIs already come up with hypotheses and plans to test them and means to revise them and checks against past data all the time with increasing success rates over a widening range of problems. This is synthetic data we’re already paying to generate.
  4. Also, have you ever run a plan (or anything else) by an LLM and asked it to find flaws and suggest solutions and estimate probabilities of success? This is already very useful at improving on human success rates across many domains.
4. “Plans for achieving such goals are not amenable to simulation because you can’t easily predict or evaluate the outcome of any proposed action. ”
  1. It’s actually very easy to get current LLMs to generate hypothetical actions well outside a narrow domain if you explain to them that there are unusually high stakes. We’re not talking about a traditional chess engine thinking outside the rules of chess. We’re about about systems whose currently-existing predecessors are increasingly broadly capable of finding solutions to open-ended problems using all available tools. This includes capabilities like deception, lying, cheating, stealing, giving synthesis instructions to make drugs, and explaining how to hire a hitman.
  2. Any plan a human can come up with without having personally conducted groundbreaking relevant experiments, is a plan that exists within or is implied by the combined corpus of training data available to an AI. This includes, for example, everything ever written by this community or anyone else, and everything anyone ever thought about upon reading everything ever written by this community or anyone else.

AnthonyC 28 Sep 2025 0:45 UTC
2 points
0
in reply to: dr_s’s comment on: An N=1 observational study on interpretability of Natural General Intelligence (NGI)
True, but I think in this case there’s at least no risk of an infinite regress. At one end, yes, it bottoms out in an extremely vague and inefficient but general hyperprior. I would guess from the little I’ve read that in humans these are the layers that govern how we learn from even before we’re born. I would imagine an ASI would have at least one layer more fundamental than this, which enable it to change various fixed-in-humans assumptions about things.
At the other end would be the most specific or most abstracted layer of priors that has proven useful to date. Somewhere in the stack are your current best processes for deciding whether particular priors or layers of priors are useful or worth keeping or if you need a new one.
I am actually not sure whether ‘prior’ is quite the right term here? Some of it feels like the distinction between thingspace and conceptspace, where the priors might be more about the expectations what things exist and where natural concept boundaries lie and how to evaluate and re-evaluate those?

AnthonyC 27 Sep 2025 13:14 UTC
3 points
0
on: Ranking the endgames of AI development
I equally hope to write “life in the day of” posts for each category soon as a better visualisation of what each of these worlds entails.
I think this would be really interesting and useful! For me, just reading the flowchart and seeing the list laid out makes me assume most people would seriously underestimate how broad these categories could actually be.
Exact placement would of course involve a number of value judgment calls. For example, I would probably characterize something like the outcome in Friendship is Optimal as an example of #7, but it could also be considered 8/10/11.
I’m also curious about your thoughts on the relative stability of each of these categories. To me, #6 seems metastable at best, for example, while #9 is an event, not a trajectory. AKA it is at least theoretically recoverable to some of the other states (or else declines into ¹⁰⁄₁₁).

AnthonyC 27 Sep 2025 12:46 UTC
2 points
0
on: An N=1 observational study on interpretability of Natural General Intelligence (NGI)
The ability to get to consciously decide when to discard or rewrite or call on the simple programs is a superpower evolution didn’t give humans. One that it seems would be the obvious solution for an AI that gets to call on an external, updatable set of tools. Or an ASI got got to rewrite the parts of itself that call the tools or notice (what it previously thought were) edge cases.
AKA, an ASI can go ahead and have a human-specific prior. It can choose to apply it until it meets entities that are alien, then stop applying it. Humans can’t really do that, in the same way that we can’t turn off our visual heuristics when encountering things we consciously know are weirdly constructed adversarial examples, even if we can sometimes override them with enough effort. The ASI, presumably, would further react to encountering aliens by reasoning from more basic principles (recurse as needed) as it learns enough to create 1) a new prior specific to those aliens, 2) a new prior specific to those aliens’ species, culture, world, etc.
Or at least, that’s my <4 minute human-level single attempt at guessing a lower bound on an ASI’s solution.

AnthonyC 27 Sep 2025 12:22 UTC
2 points
0
in reply to: StanislavKrym’s comment on: Economics Roundup #6
America would have to pay the subsidies off.
This is not necessarily true. At least not on any currently-human-relevant timescale. The ballooning can be a problem, especially when the money is spent very poorly. But if a reasonable fraction of it is spent on productive assets and other forms of growth, debt can grow for a long time. Longer than the typical lifespan of a country or currency.

AnthonyC 25 Sep 2025 11:52 UTC
2 points
0
in reply to: StanislavKrym’s comment on: AGI Companies Won’t Profit From AGI
The part of the reasoning where others use the AI to generate value does seem to underexplore the possibility that the AI companies themselves use the AI for that first.

AnthonyC 25 Sep 2025 11:50 UTC
2 points
0
in reply to: LTM’s comment on: AGI Companies Won’t Profit From AGI
What would you say to the idea that other kinds of capital retain value post-AGI? Like land, or mineral rights, or electricity generating capacity? I think those are also unlikely, but I do come across them once in a while.

AnthonyC 24 Sep 2025 22:48 UTC
5 points
3
on: AGI Companies Won’t Profit From AGI
Let’s consider the set of worlds where developing AGI does not destroy or permanently disempower humanity.
You have a good point, that in many such scenarios the investors in AI labs, or the AI companies, may not be able to capture more than a tiny fraction of the value they generate.
Does this make the investment a mistake?
Personally, I don’t think so. The people making these investments generally have a level of wealth where no amount of additional money can make more than a small additional improvement to their well being. In contrast, AGI and ASI could plausibly (within their lifetimes) render the world as a whole many OOMs richer, with capabilities that seem centuries or millennia away in a non-AI future. Not being able to claim all of that wealth may cost them status/dominance/control, but they would also gain the status/prestige of having enabled such radical improvement. And in any case, they might (very reasonably, IMO) be willing to trade status for immortality + godlike technological powers.
Also, in the proportion of worlds where the AI labs or those funding them do manage to retain control of the wealth created or to obtain and hold other forms of power, that’s quite the high payoff even at these valuations.

AnthonyC 23 Sep 2025 22:50 UTC
6 points
4
in reply to: Bronson Schoen’s comment on: Notes on fatalities from AI takeover
Yeah, my own instinct is to just see if the results are interesting in such a way that if I believed them, it would meaningfully change what I thought was the best strategy. In this case, don’t think so. Even what I see as a very optimistic set of assumptions still results in what I see as an unacceptably high risk of very bad outcomes. I do find the exploration itself interesting, though.

AnthonyC 22 Sep 2025 15:29 UTC
8 points
1
in reply to: MalcolmMcLeod’s comment on: This is a review of the reviews
Politics is the mind-killer, sure. But ASI is the planet-killer, and politics is the ASI-[possibility-thereof-]killer, so I am willing to let my mind take a few stray bullets.
This is an absolutely fantastic phrasing/framing.

AnthonyC 12 Sep 2025 11:55 UTC
3 points
1
on: The Techno-Pessimist Lens
Real solutions require deep, comprehensive understanding of the relevant problems and often involve trade-offs.
Agreed, though we usually don’t need to start with comprehensive understanding to start making things better. There are often institutional/organizational low-hanging-fruit choices that ameliorate particular harms at low or negative cost, which we nevertheless manage to just not do for many years or decades.
Most of these require some form of restraint on development, which has a cost
And most people do not know how to weigh costs against benefits, or to estimate either, or to evaluate the credibility of third party estimates of either. In many contexts, “Y has a cost, but will pay for itself in X years” (where X<10) is somehow not seen as a knockdown argument even in purely economic terms. Adding other positive non-economic effects sometimes, somehow makes Y look like a luxury good, even more out of reach, or somehow a scam.

AnthonyC 12 Sep 2025 11:42 UTC
4 points
0
on: The Eldritch in the 21st century
But the truth is that no one has power.
I agree with what you’re pointing at, but not with this statement. I think it’s worse than this. There are (groups of) people who (collectively) know how to make (some) things better. But The Power is held collectively by those you label The Powerless, and they lack the skills or drive to even know how to choose the right priesthood-holders to trust. As a result we spend an awful lot of time and effort tying our hands and shooting our feet, while competing would-be priests shout ineffectually at one another, regardless of who has a record of having made correct predictions before. Our ancestors may not have understood the natural world, but if someone showed up and made the right guesses about the behavior of some eldritch god vastly more often than anyone else could do, they would have been either elevated to leadership or feared/hated/condemned for black magic. We lost that skill, I think, in favor of playing social status games, the moment it sunk in that the natural-world-threats felt distant.
Aka: In practice we are sex-obsessed murder-monkeys and all of this is way above our pay grade.

AnthonyC 11 Sep 2025 0:16 UTC
3 points
0
in reply to: localdeity’s comment on: Childhood and Education #14: The War On Education
I had exactly one teacher, a professor in college, who understood and acted on this idea. His grading formula was such that acing the final always meant you got an A. But, the better you did on homework and tests before that (‘achievement points’), the less the final exam counted for. And the more effort you put in (homework, office hours, class participation), the more leniently your exams would be graded (‘effort points’). And the more effort the class as a whole put in, the more leniently everyone’s exams would be graded. But I think something like that only works if you’re willing to actually let students fail.

AnthonyC 11 Sep 2025 0:03 UTC
3 points
0
on: Childhood and Education #14: The War On Education
On grade level books and learning to read: Wow that’s some serious insanity I had not been aware of. When I was in 6th grade one of the vice principles had a bookshelf in his office full of (I think donated?) books. Anyone could borrow one, and keep it if they read it and wrote a report on it. The first one I borrowed was the a compilation of Plato’s dialogs and the Republic—that was my first real introduction to Philosophy as an academic subject. Also, my brother-in-law was reading chapter books at 3. And my whole first grade class was expected to be reading chapter books.
On even ‘gifted’ schools mostly not wanted kids to get far ahead: Many (in my limited personal experience, most) teachers are not experts in the subjects they’re teaching. A 7th grade math teacher is, if you’re lucky, an expert at teaching 7th grade math to typical 7th graders. They hopefully have a familiarity with 8th grade math and should be able to explain it, but I expect many would struggle to explain 10th grade math. Similarly, I wouldn’t expect them to be prepared to teach 7th grade math to even very bright and curious 4th graders—I would think doing that may require a deeper level of mathematical understanding and also social/psychological awareness in order to adapt the approach to where the students are.

AnthonyC 5 Sep 2025 16:47 UTC
3 points
0
on: My AI Vibes are Shifting
AI infrastructure seems really expensive. I need to actually do the math here (and I haven’t! hence this is uncertain) but do we really expect growth on trend given the cost of this buildout in both chips and energy? Can someone really careful please look at this?
This is not a really careful look, but: The world has managed extremely fast (well, trains and highways fast, not FOOM-fast) large-scale transformations of the planet before. Mostly this requires that 1) the cost is worth the benefit to those spending and 2) we get out of our own way and let it happen. I don’t think money or fundamental feasibility will be the limiters here.
Also, consider that training is now, or is becoming, a minority of compute. More and more is going towards inference—aka that which generates revenue. If building inference compute is profitable and becoming more profitable, then it doesn’t really matter how little of the value is captured by the likes of OpenAI. It’s worth building, so it’ll get built. And some of it will go towards training and research, in ever-increasing absolute amounts.
Even if many of the companies building data centers die out because of a slump of some kind, the data centers themselves, and the energy to power them, will still exist. Plausibly the second buyers then get the infrastructural benefits at a much lower price—kinda like the fiber optic buildout of the 1990s and early 2000s. AKA “AI slump wipes out the leaders” might mean “all of a sudden there’s huge amounts of compute available at much lower cost.”

AnthonyC 3 Sep 2025 20:47 UTC
2 points
0
on: Startup Roundup #3
Figuring out what a startup should say to investors is strangely useful for figuring out what it should actually do. Most people treat these questions as separate, but ideally they converge. If you can cook up a plausible plan to become huge, you should go ahead and do it.
If you’re not a software company, and what you want to do requires steel in the ground, then any workable plan to become huge will realistically require 3-4 years each in the lab, pilot, demo, and FOAK phases, largely in series, and will often benefit from the founders stepping down as CEO quite early in favor of someone with much more direct industry experience, and if you’re honest about that many VCs will run away.
As in, because you might have to raise at a lower number in the future, you should raise at a lower number than that now, so you don’t have a ‘down round.’ Or because you couldn’t handle having the cash.
As you explain later, the first part of this would be nonsense if the second part weren’t so important. AKA, if only the founders have the discipline to not increase spend rate beyond necessity and instead use the money to increase runway and still follow an optimal path to growth, instead of inefficiently chasing faster growth by spending more and just assuming more funding will be available when needed, this would not be such a problem.
Also it’s not about having a down round, necessarily. It’s sometimes about needing one at all. I’ve met people whose shareholders forced their companies to wind down instead of allowing a down round, even if the down round would likely have led to a successful exit later, because e.g. the shareholder was trying to raise their own next fund and a down round on their record would have made it harder.

AnthonyC 3 Sep 2025 18:20 UTC
2 points
0
in reply to: Gordon Seidoh Worley’s comment on: All Exponentials are Eventually S-Curves
Fair enough. If nothing else, it’s best to state where you think the min value, max value, and midpoint are, and ideally put error bars around those.
Or, at least, you can state outright you think the midpoint and max are far enough away as to be irrelevant distractions to some particular practical purpose. Or that you expect other factors to intervene and change the trend long before the later parts of the sigmoid shape become relevant.
To add: It is in principle very easy for people to make equivalent prediction errors in either direction about when a particular exponential will level off, and to be wrong by many orders of magnitude. In practice, I usually encounter a vocal minority who happily ignores the fact that the sigmoid even exists, and a larger group who thinks that leveling off must be imminent and the trend can possibly continue much longer. The cynic in me thinks the former group tends to start getting believed just in time to be proven wrong, while the latter group misses out on a lot of opportunities but then helps ensure the leveling off has less catastrophic consequences when it happens.
I’m curious: was there a particular (set of) sigmoid(s) you had in mind when writing this post? And particular opinions about them you haven’t seen reflected in discussions?
Most of the times I’ve used sigmoid in my own modeling/forecasting have been about adoption and penetration of new technologies. Often the width of the sigmoid (say the time to get from 1% to 99% of the way from min to max) is relatively easy to approximate, driven by forces like “how incumbent institutions are governed” (yes, this is critical even for most extremely disruptive innovations). The midpoint and maximum are much harder to anticipate.

AnthonyC 3 Sep 2025 17:48 UTC
20 points
2
on: All Exponentials are Eventually S-Curves
This is absolutely true. However, actually using it effectively requires having a sufficiently good, principled reason for thinking the limit is in some particular place, or will be approached on some particular timeline. When I’ve looked at many (most?) real-world attempts to forecast when some particular exponential will start looking like an s-curve, they’re usually really far off, sometimes in ways that ‘should’ be obvious, even if the forecasting exercise itself is instructive.

AnthonyC 1 Sep 2025 15:51 UTC
11 points
0
on: Generative AI is not causing YCombinator companies to grow more quickly than usual (yet)
I feel like this underestimates the difference between what you’re citing, valuation growth over 2 years post-YC, and the Garry Tan quote, which was about weekly growth during the 3 month YC program. I also wish the original Garry Tan claim were more specific about the metric being used for that weekly growth statistic. In principle, these aren’t necessarily mutually exclusive claims. In practice, I’d expect there’s some fudging going on.
I can imagine something like the following: Companies grow faster with less investment, reaching more revenue sooner because of GenAI. But, this also means the company has fewer defensible assets, and less of a lead over competitors, so the valuation is lower after 2 years. AKA potentially the cost of software innovation is going down, speed is going up, and there’s less of an advantage to being first because it’s getting easier to be a fast follower. In a world where it’s possible-in-principle to think about one person unicorns, then why should software companies ever have high valuations at all, once enough people know what they’re trying to build?
I’m curious what effects with would have, if true. If we end up in a place where an individual can build a $10-20M company, practically on their own, in months, with only a seed round, but can never get to $100M or $1B, how does this affect startup funding models? Pace of overall innovation? I could see this going really well (serial founders have time in their careers to build 50 companies, cost to access new software products drops) or really badly (VCs lose interest in software startups, innovation slows to a crawl), or any number of other ways.
Or, also not mutually exclusive with the above: Maybe GenAI-2023 is sufficiently different from GenAI-2025 that we shouldn’t really be comparing them, and the 2023 batches were growing less or slower due to dealing with the aftereffects of covid or something.