I think this is very hard thing to get right, and I don’t think there’s a scaleable way or org. that you can spend money on.
I think the current best bet would be to find existing people that are technical, driven, safety-pilled and already done some independent research, and fund them to continue doing that?
rahulxyz
Karma: 104
Yeah good point.
I agree this would obviously work, but none of the incentives at the top would make this remotely plausible today look at the board of OAI / Anthropic—probably the only 2 where this makes sense.
Maybe some employees inside these companies may be open to this, but the I think the chance the current board of either of these labs could get a majority vote today to shut it down is basically <5%
Thanks for bringing this up. I’ve been thinking similar though recently, and this seems to be valance related (yay democracy, boo dictatorship or something).
With a big if, assuming that AGI is controllable at all and the first person to reach AGI is the CCP could use it to enforce their power structure forever.
But then for some reason, people think the US getting their hands on AGI would somehow mean that “democracy” or something is in control rather than whoever is in power at that specific point enforcing their power structure forever? It’s not like democracies can never turn into uni-polar power structures, and it seems doubly naive to think someone with a magic AGI wand who’s already selected for large amounts of power-seeking behavior (democratic presidents as well as CCP leaders) would not use it to cement their own power.
Thanks, this resonated with me too and was I had never heard about it before! I was doing more research into it., and there seems to be the opposite of this called HSAM or highly superior autobiographical memory where some people can vividly relive their life given just a specific date like April 15, 1995. However, it seems to be less common (only 100 people diagnosed with it worldwide)
That made this whole thing feel even more alien to me.
Maybe it’s me being dumb, but how can someone believe:
a) AI will come up with new mathematical proofs and cures for diseases that no human has ever thought of (and that’s with thousands of geniuses throwing their lives at the problem)
b) AI will never come up with the idea of taking over the world—which multiple random sci-fi authors and script writers have come up with, probably within 30 minutes of thinking about it.
simultaneously.
I get it’s expensive. But their whole goal is to automate AI research (capabilities or safety) and they claim that the “race” is so crucial to humanity that they be first. On top of all that, if they think this TOS phrase is sufficient deterrence to their competitors, then it makes me think they they either they’re lying about the stakes, or they’re incompetent.
I would assume they would keep that version of Claude for themselves internally, and not the public version of Claude. Why give their competitors any edge at all?
I remember in the tail end of 2024, I was thinking—“these AIs are going to come for lonely single men who’ll spend hours addicted to talking to their AI girlfriends.” And of course I wouldn’t be one of those schmucks who spent hours talking to an LLM...
But I also see myself in May 2026 spending a couple of hours every day talking to Claude...I guess it came for me first?
I’m using it a lot recently for summarizing / understanding papers in bio, AI etc. And it feels genuinely useful. But every now and then I read something that seems just a little bit incoherent, especially on more scientific topics.
I’m just wondering if it’s more optimized to make explanations that are vaguely insight-porn oriented and make me feel like I understand something deeper than I actually do. It’s also easy to push back and Claude will just change it’s mind easily and give me another just-as-persuasive sounding argument why it was wrong [1]
It does feel like it’s persuasive capability is rising faster than it’s actual world-modelling capability, and it’s prone to give incorrect or half-truths while sounding very convincing. If those two capabilities keep diverging, I’m worried I might be messing up my own world model somehow by continuing to optimize my learning with AI.
Has anyone else felt this, or have any suggestions?
-----------------[1] It tried to convince me just now biological neurons might depend on quantum mechanics—without any prompting on my part, and also claimed that this was a “serious scientific debate” and referenced a bunch of papers to support it’s point that on deeper look had nothing to do with what it actually claimed. This was during me researching current tech in BCIs like neuralink etc. And of course, it changed it’s mind pretty quickly when I pushed back.
I think the top 3 were still tech companies 10 years ago, and 5 out of 10 were already there so this doesn’t seem new? It’s just how much room they’ve had to grow that’s been surprising IMO.
Here’s the top 3 as of 2016 (AI researched)
Top 3 companies by market capitalization (Q1 2016)Apple — ~$604 billion
Alphabet (Google) — ~$518 billion
Microsoft — ~$437 billion
Facebook / Amazon were also in the top 10. It’s mostly been them expanding ~10x from there in market cap.
I see a lot of vibes recently that AI is going to produce these amazing GDP growth rates like 20%+. I wanted to record my
predictioncounter-vibe that I don’t think current GDP statistical methods are good at measuring discontinuous trends like AI.When phones started replacing standalone cameras, we got a reduction in “camera GDP” and a slight increase in phone GDP, but a 100x increase in photography production. GDP doesn’t really measure “amount of photography” happening in a meaningful way. GDP fails at measuring disruptions, but is mostly smoothed over because the whole economy isn’t getting disrupted at the whole time.
Let’s say we have AI replacing human labor across a wide variety of industries. The nominal value of all of it goes from 10T to 1T. The remaining work is high value stuff, or only senior staff managing agents or something similar. Economic surveys would show humans getting the same salaries and the deflator for this doesn’t change. It’s now recorded as 1T as real GDP instead of 10T. I don’t think this is because lump of labor fallacy is suddenly true, but new tasks have to appear that we can actually do better than AI, and enough of it that we can absorb all the existing labor. People are over-anchoring on this happening during the Industrial Revolution and that happened over generations—not a few years.
So I think it’s just as likely for GDP to fall 10% than rise, even if we get a better quality of life. And especially if AI starts disrupting the physical economy with robots etc.
AGI → loads of revenue path makes sense to me
But I can imagine an AI that is the 99.9th percentile across some disciplines but not all. I’d assume we already spend ~10T for things like engineering talent, medical advice, legal etc.. and that seems like AI companies could make that much (given they can capture a lot of the excess value—assume there’s only a single AI lab and there’s no competition if you will). I can imagine something slightly better than today’s AI’s have that level of revenue after proliferating through the economy for another decade.
Even if it’s deficient in a bunch of other things we are good at (writing, comedy, physical labor, making better AI’s etc..) It seems to me you can get very far without all human skills, but just a subset of them.
I see a few posts like this anchoring AGI timelines to company revenue / GDP, most notably from economists. But I’d like to understand where this intuition comes from..It seems to me similar to the biological anchors or back in the day Kurzweilian anchor to FLOP/s.
For me, GDP anchors aren’t any more intuitive to me for AGI/ ASI any more than number of parameters or FLOP/s intuitions. Like I can totally imagine AI companies having revenue of ~10% of GDP (10T) without an AGI, even with current level AIs proliferating over the next 10 years.
Isn’t that a fully general argument against any boycott / activism? Being vegan doesn’t by itself slow down factory farming either.
I think your revenue does marginally lead to them having higher growth rates, and raising more money and not being “funding constrained” in the next round of funding and researching the next set of capabilities.
I don’t if the right answer is to not use it, but it does seem like a dilemma.
This whole SaaSpocalyse scenario outlined here https://www.lesswrong.com/posts/bKrpLhqcoN6WycrFp/citrini-s-scenario-is-a-great-but-deeply-flawed-thought has made me think that one obvious loser in all this is Amazon / AWS
It’s been said that the real money maker for Amazon is AWS and not their retail business.
In fact, the lock-in is so strong that there’s a cottage industry of people with AWS certifications and firms whose sole job is “AWS Cost Optimization”.
But what seems to be not yet priced in is the ease of which anyone with a datacenter can now build an AWS-compatible API in the future.
In the end of the day, amazon is bunch of servers in a datacenter. All the so called “services” are just some syntactic sugar for people that don’t want to manage their own servers—and that’s where their moat lies.
It’s hard for a startup who’s built on top of these services to migrate out to another bare-bones rack in another datacenter , but if the datacenter can give them a compatible API, then moving becomes a click of a button (for the most part).
But if you look at how openai competitors worked, almost everyone has a “openai-compatible” API—all I do is change the URL to new model provider and I’m good to go.
This seems like it would truly kill the AWS lock-in, and it doesn’t seem to be priced in to their stock price at all. Maybe people don’t think of AWS as a SaaS company? I would never myself short a stock, but it does seem like the second-order effect to all this is obviously not priced in at all.
Yes mostly agree. Unless the providers themselves log all responses and expose some API to check for LLM generation, we’re probably out of luck here, and incentives are strong to defect.
One thing I was thinking about (similar to i.e—speedrunners) is just making a self-recording or screenrecording of actually writing out the content / post? This probably can be verified by an AI or neutral third party. Something like a “proof of work” for writing your own content.
Yes, I agree with that. I’m not claiming that knowing about it stops you from wanting ice cream.
I’m claiming that if the concept was hardwired into our brains, evolution would have had an easy time optimizing us directly to want “inclusive genetic fitness” rather than wanting ice cream.i.e—we wouldn’t want ice cream at all but reason from first principles what we should eat based on fitness.
Just finished reading “If Anyone Builds It, Everyone Dies”. I had a question that seems like an obvious one, but one I didn’t see addressed in the book, maybe someone can help:
The main argument in the book is the analogy to humans. Evolution “wanted” us to maximize genetic fitness, but it didn’t get what it trained for. Instead, it created humans who love ice cream and condoms even though they reduce our genetic fitness.
With AGI, we’re on track to do something similar—we won’t get an AI aligned to human interests even though we do RLHF or any other such simple training or shaping to an AI, it’ll end up wanting something weird and inhuman rather than maximizing human values.
But in my mind, this seems to miss a fairly important point: The fact that human brains don’t come pre-wired with much knowledge. We have to learn it from scratch. We don’t come out of the womb with concept of “inclusive genetic fitness”. It took us culture and ~200,000 years to figure that out, and we still only learn it after about 15-20 years of existing. So there’s no way that evolution could have made us point our utility function to “inclusive genetic fitness” because that concept doesn’t exist in our brains.
Modern AIs don’t seem like that. They come with the sum of human knowledge baked in during pre-training. As they get smarter, the concept of “human values” or “friendly AI” is definitely something in it’s existing mind. So it should be much easier for us to do alignement and test whether we can point it to that specific concept vs. what what evolution had.
Seems mostly true. There’s also a group of people flailing around trying to fit it in their workflows because all the top tech companies are saying it’s the next big thing.
I notice lots of LARPing too with adding the word “AI” to everything hoping that will unlock some new avenues.
> The big struggle is to even start using AI coding assistant tools. Lots of teams just don’t use them at all, or use them in very limited ways. People leading these teams know they are going to lose if they don’t change but are struggling to get their orgs to let them.
It seems to me 25-50% of developers are using some form of AI-assisted coding. Did you notice that the beaureacracy of their companies was not allowing their developers to use coding assistants?
Do you have a limit on number of grants you’d be able to give out?