Optimizing for the flourishing of all sentient life regardless of substrate.
george_adams
Karma: 61
Have any agents tried autonomously posting on LW?
I’m as AGI pilled as the next person on LW but this post gives me Feynman vibes:
I returned to civilization shortly after that and went to Cornell to teach, and my first impression was a very strange one. I can’t understand it any more, but I felt very strongly then. I sat in a restaurant in New York, for example, and I looked out at the buildings and I began to think, you know, about how much the radius of the Hiroshima bomb damage was and so forth… How far from here was 34th street?… All those buildings, all smashed — and so on. And I would go along and I would see people building a bridge, or they’d be making a new road, and I thought, they’re crazy, they just don’t understand, they don’t understand. Why are they making new things? It’s so useless
But, fortunately, it’s been useless for almost forty years now, hasn’t it? So I’ve been wrong about it being useless making bridges and I’m glad those other people had the sense to go ahead.We should have high uncertainty in our distribution of how this is going to go, how fast it is going to go, where the marginal value add of humans will be in the short and medium term, etc.
Is it possible to stay at Lighthaven even if not attending an event there?
The probability that a wild prediction of the future will be true is low no matter how well reasoned it is. One should take the prior that the future will be insanely different but it is extremely hard to predict how it will be different.
More evidence here as well: nondeterminism, STEVE-1 nondeterminism.
Am I crazy that I think that coding agents dramatically slow me down? Everyone around me (including some of the most technically gifted people I know) raves about them and how it’s made them 2x/5x/10x/100x more productive. I haven’t felt this at all except for the case of vibe coding something for fun where I don’t really care about the quality/implementation of the code. For running experiments where I care about the code, I’ll try and have Claude code implement something, it will spit out a bunch of code. I’ll read the code, dislike parts of it and try to get Claude to iterate on it. Claude won’t really get to the target of what I want so at some point I’ll give up and write the entire thing myself. Also, every time I do vibe code something, I understand it less well and thus iterate worse on subsequent experiments/code I want to write. If I am unique in this, a reason could be because I’ve spent probably 1000x more time writing code as I have spent reading code and I suck at reading code.
I also can see a reason for this being that the delta between code that’s 98% well done for an experiment and code that’s 100% well done for the same experiment is dramatically different. And agents help to speed up the initial writing dramatically while not being able to iron off the last few small percentage points. In other words, the value I get out of perfect code, and almost perfect code is dramatically different. Am I crazy?
An observation about how amazing large economies are: the inflation adjusted GDP of the entire United States in 1810 (~8million people) is roughly the same amount that is spent on PhD stipends per year in the US in 2026 (around 0.05% of 2026 GDP).
Any predictions for what percentage of world GDP AI will be when an AI solves the Riemann hypothesis? (If this happens)
And what if it does?
Why don’t EA/rationalist firms use prediction markets for deciding who to hire (futarchy style)? Your application to a company would be something like ticking a box that says, “I give you permission to create a prediction market on me that I won’t trade in.” The statement the market would be trading could be P(we won’t regret hiring X | we hire X) or P(X will meet the following KPIs | we hire X), where X is the candidate. Effectively, this would outsource the work of hiring to people who want to profit on the market.
You could also make candidates do some set of tests or competitions and show that information to traders (as well as resume and other standard info). Then you pick the top trading candidates you want (or several above some threshold). This would also give more opportunity to some candidate X to publicly do things they believe will convince traders to increase their market’s price. It would be really cool if the mechanism actually deciding hiring decisions had the capacity to look at everything a candidate was willing to do/show to get hired, vs the current model of very restricted time commitment per candidate.
I suspect the blocker to this is that the trading volume would just be too low to give meaningful information.
The idea of radical life extension has been around for millennia, it has been scientifically plausible for decades if not centuries, but it has always been a marginal concern. There was never a society which organized to make the cure of ageing a major priority.
I would describe what we have already done as radical life extension. Perhaps we have a difference in definition. From this link:
In 1900, the average life expectancy of a newborn was 32 years. By 2021 this had more than doubled to 71 years.
The model that is most convincing of why we didn’t orient ourselves around something like a cult of increasing life expectancy is that we went down the path of least resistance of technological progress and economic growth.
The human race in general was never on board with transhumanism.
I claim this was never a realistic goal. The set of cultures in which we have the cultural norms/tools to create technology and large economic growth (that is required for transhumanism) AND which prioritize transhumanism above everything else are not very numerous.
In some sense, you can see the afterlife promise of many religions as a form of transhumanism, and billions of people are on board with that. Yet basically, none of these religions have contributed to actually achieving something like transhumanism.
I’ve heard this critique lobbed around a fair bit in AI safety circles: “{some org} is bad at philosophy”. What does this mean? I’ve heard this both from collaborators in person and on LW. A decent number of times, this critique has been directed at Anthropic. I’ll apologize for how vague (and unclear) this post is in advance but does anyone have any idea what people are getting at when they say this? It is usually an unmotivated statement too (at least in my judgement) and a statement meant to critique an organization’s approach to AI safety.
My prediction for the next few years (or until AGI) is that there’s going to start to be a winner take all approach to computer science talent. The majority of the job of software engineering will be automated (if it isn’t already). There are still robustness pockets that human software engineers can help fill for now. But I expect top AI researchers to continue making exorbitant amounts of money, even if little software engineering is involved. So, computer science talent will start to loosely resemble the competitiveness of professional sports, where there is a step function change between compensation if you “make it” or not. And the hiring bar will be extremely high. You can already start to see glimpses of this with the weight people put on Olympiad contestants when recruiting.
I think quantified intuitions is a reasonable, although incomplete version of what you describe. It specifically focuses on scope insensitivity rather than a traditional rationalist curriculum.
Markets pricing in AGI also is also conditional on markets believing something like the current legal/property rights system will continue to hold after AGI. If it is possible that AI is a bubble, and it’s not obvious that you will win anything if you get the AGI trade right, then traders won’t “price in” AGI even if it is extremely economically valuable and coming soon.
My argument is also not that markets won’t price in AI in its current form or increasing capabilities, it is specifically at the point where we actually have strong AGI systems phase shift.
I disagree with your post, but I will add an additional example: falling birthrates. I don’t remember in which of his essays it was (probably in Fanged Noumena), but Nick Land posits that the technocapital system of capitalism which he views as being AGI has figured out that it won’t need humans much longer and thus has no incentive to keep the birth rates up. I obviously do not literally believe this, but I think it helps illustrate what you’re trying to describe.
I know this is 7 months late! But I read this shortform yesterday and it somewhat resonated with me. And then today I read Noah Smith’s most recent blog post which perfectly described what I think you’re getting at so I’m linking it here.
Why trust your prior over the prior of the market/hedge funds? By this I mean why expect that this isn’t already priced in? AI (and AGI) is a big enough news story now such that I would expect hedge funds to be thinking about things like this. At recruiting events, I’ve asked quants how they’re thinking about this exact question and I usually got pretty decent AGI pilled responses.
It is certainly possible that the market hasn’t priced this in, but my prior is in the vast, vast majority of cases, there is some quant that has already sucked out any potential gains one could get.
I’m also a college student who has been wrestling with this question for my entire undergrad. In a short timelines world, I don’t think there are very good solutions. In longer timelines worlds, human labor remains economically valuable for longer.
I have found comfort in the following ideas:
1) The vast majority of people (including the majority of wealthy white-collar college-educated people) are in the same boat as you. The distribution of how AGI unrolls is likely to be so absurd that it’s hard to predict what holds value after this. Does money still matter after AGI/ASI? What kinds of capital matters after AGI/ASI? These questions are far from obvious for me. If you take these cruxes, then even people at AGI labs could be making the wrong financial bets. You could imagine a scenario where AGI lab X builds AGI first and comes to dominate the global economy so that everyone with stock options in AGI lab Y will be left with worthless capital ownership. You could even imagine a scenario of owning stock in an AGI lab that builds AGI and then that capital is no longer valuable.2) For a period of time, I suspect that young people are likely to have an advantage in terms of using “spiky” AI tools to do work. Being in the top few percentile of competence for coding with LLMs or doing math with LLMs or even doing other economically valuable tasks using AI is likely to have career opportunities.
3) You can expect some skills to be important up until the point of AGI. For example, I see coding and math in this boat. Not only will they be important, but the people doing the most crucial and civilization altering research will likely be very good at these skills. These people are likely to be the 1 in a million Ilya Sutskever’s of the world, but I still find it motivating to build up this skillset at a point which is really the golden age of computer science.
More generally, I have found it useful to think about outcomes as sampled from a distribution and working hard as pushing up the expected value of that distribution. I find this gives me much more motivation.
In hindsight it seems you were correct.