titotal

Karma: 2,071

titotal 23 Oct 2025 11:48 UTC
69 points
54
on: The Doomers Were Right
Last month, doomers predicted that the Rapture would happen. The doomers were wrong, as they have been all the other dozens of notable times they predicted this.
Doomers predicted that the Y2K bug would cause massive death and destruction. They were wrong.
Moving to your social examples, doomers predicted that the legalisation of gay marriage would lead to the legalisation of bestiality. They were wrong.
You even provide an example yourself where people claim that D&D leads to satanism. This didn’t happen! Having an orc hero is not satanism! satanism has remained a niche religion. The doomers were wrong.
Cherrypicking examples where doomerist predictions were wrong and examples where they were right is a very poor way to figure out whether a particular doomerist prediction is wrong or right.
Skeptics are not arguing that no technology or social movement has ever had a negative effect, they are arguing that humans are biased towards apocalyptic thinking and overestimating the threat of new stuff. If you want to rebut this, you have to actually collect some unbiased data to test the claim.

titotal 24 Jun 2025 9:00 UTC
13 points
2
in reply to: elifland’s comment on: A deep critique of AI 2027’s bad timeline models
I’m leaving the same comment here and in reply to daniel on my blog.
First, thank you for engaging in good faith and rewarding deep critique. Hopefully this dialogue will help people understand the disagreements over AI development and modelling better, so they can make their own judgements.
I think I’ll hold off on replying to most of the points there, and make my judgement after Eli does an in-depth writeup of the new model. However, I did see that there was more argumentation over the superexponential curve, so I’ll try out some more critiques here: not as confident about these, but hopefully it sparks discussion.
The impressive achievements in LLM capabilities since GPT-2 have been driven by many factors, such as drastically increased compute, drastically increased training data, algorithmic innovations such as chain-of-thought, increases in AI workforce, etc. The extent that each contributes is a matter of debate, which we can save for when you properly write up your new model.
Now, let’s look for a second at what happens when the curve goes extreme: using median parameters and starting the superexponentional today, the time horizon of AI would improve from one-thousand work-years to ten-thousand work-years In around five weeks. So you release a model, and it scores 80% on 1000 work year tasks, but only like 40% on 10,000 work year tasks (the current ratio of 50% to 80% time horizons is like 4:1 or so). Then five weeks later you release a new model, and now the reliability on the much harder tasks has doubled to 80%.
Why? What causes the reliability to shoot up in five weeks? The change in the amount of available compute, reference data, or labor force will not be significant in that time, and algorithmic breakthroughs do not come with regularity. It can’t be due to any algorithmic speedups from AI development because that’s in a different part of the model: we’re talking about three weeks of normal AI development, like it’s being done by openAI as it currently stands.. If the AI is only 30x faster than humans, then the time required for the AI to do the thousand year task is 33 years! So where does this come from? Will we have developed the perfect algorithm, such that AI no longer needs retraining?
I think a mistake could be made in trying to transfer intuition about humans to AI here: perhaps the intuition is “hey, a human who is good enough to do a 1 year task well can probably be trusted to do a 10 year task”.
However, if a human is trying to reliably do a “100 year” task (a task that would take a team of a hundred about a year to do), this might involve spending several years getting an extra degree in the subject, read a ton of literature, improving their productivity, get mentored by an expert in the subject, etc. While they work on it, they learn new stuff and their actual neurons get rewired.
But the AI equivalent to this would be getting new algorithms, new data, new computing power, new training. ie, becoming an entirely new model, which would take significantly more than a few weeks to be built. I think there may be some double counting going on between this superexp and the superexp from algo speedups.

titotal 12 Sep 2024 13:23 UTC
6 points
6
in reply to: Jeremy Gillen’s comment on: james oofou’s Shortform
List of lethalities is not by any means a “one stop shop”. If you don’t agree with Eliezer on 90% of the relevant issues, it’s completely unconvincing. For example, in that article he takes as an assumption that an AGI will be godlike level omnipotent, and that it will default to murderism.

titotal 12 Sep 2024 9:37 UTC
5 points
4
on: Is this a Pivotal Weak Act? Creating bacteria that decompose metal
building a bacteria that eats all metals would be world-ending: Most elements on the periodic table are metals. If you engineer a bacteria that eats all metals, it would eat things that are essential for life and kill us all.
Okay, what about a bacteria that only eats “stereotypical” metals, like steel or iron? I beg you to understand that you can’t just sub in different periodic table elements and expect a bacteria to work the same. There will always be some material that the bacteria wouldn’t work on that computers could still be made with. And even making a bacteria that only works on one material, but is able to spread over the entire planet, is well beyond our abilities.
I think list of lethalities is nonsense for other reasons, but he is correct in that trying to do a “pivotal act” is a really stupid plan.

titotal 11 Sep 2024 9:44 UTC
3 points
1
in reply to: Adam Scholl’s comment on: AI forecasting bots incoming
Under peer review, this never would have been seen by the public. It would have incentivized CAIS to actually think about the potential flaws in their work before blasting it to the public.

titotal 10 Sep 2024 11:07 UTC
28 points
11
on: AI forecasting bots incoming
I asked the forecasting AI three questions:
Will iran possess a nuclear weapon before 2030:
539′s Answer: 35%
Will iran possess a nuclear weapon before 2040:
539′s Answer: 30%
Will Iran posses a nuclear weapon before 2050:
539′s answer: 30%
Given that the AI apparently doesn’t understand that things are more likely to happen if given more time, I’m somewhat skeptical that it will perform well in real forecasts.

titotal 14 Aug 2024 13:08 UTC
16 points
13
on: Rabin’s Paradox
The actual determinant here is whether or not you enjoy gambling.
Person A who regularly goes to a casino and bets 100 bucks on roulette for the fun of it will obviously go for bet 1. In addition to the expected 5 buck profit, they get the extra fun of gambling, making it a no-brainer. Similarly, bet 2 is a no brainer.
Person B who hates gambling and gets super upset when they lose will probably reject bet 1. The expected profit of 5 bucks is outweighed by the emotional cost of doing gambling, a thing they are upset by.
When it comes to bet 2, person B still hates gambling, but the expected profit is ridiculously high that it exceeds the emotional cost of gambling, so they take the bet.
Nobody is necessarily being actually irrational here, when you account for non-monetary costs.

titotal 28 May 2024 15:20 UTC
7 points
7
on: Real Life Sort by Controversial
I believe this is important because we should epistemically lower our trust in published media from here onwards.
From here onwards? Most of those tweets that chatgpt generated are not noticeably different from the background noise of political twitter (which is what it was trained on anyway). Also, twitter is not published media so I’m not sure where this statement comes from.
You should be willing to absorb information from published media with a healthy skepticism based on the source and an awareness of potential bias. This was true before chatgpt, and will still be true in the future.

titotal 22 Apr 2024 9:04 UTC
2 points
0
in reply to: gilch’s comment on: Any evidence or reason to expect a multiverse / Everett branches?
No, I don’t believe he did, but I’ll save the critique of that paper for my upcoming “why MWI is flawed” post.

titotal 10 Apr 2024 16:35 UTC
6 points
2
in reply to: RHollerith’s comment on: Any evidence or reason to expect a multiverse / Everett branches?
I’m not talking about the implications of the hypothesis, I’m pointing out the hypothesis itself is incomplete. To simplify, if you observe an electron which has a 25% chance of spin up and 75% chance of spin down, naive MWI predicts that one version of you sees spin up and one version of you sees spin down. It does not explain where the 25% or 75% numbers come from. Until we have a solution to that problem (and people are trying), you don’t have a full theory that gives predictions, so how can you estimate it’s kolmogorov complexity?
I am a physicist who works in a quantum related field, if that helps you take my objections seriously.

titotal 10 Apr 2024 11:33 UTC
5 points
4
in reply to: tangerine’s comment on: Any evidence or reason to expect a multiverse / Everett branches?
It’s the simplest explanation (in terms of Kolmogorov complexity).
Do you have proof of this? I see this stated a lot, but I don’t see how you could know this when certain aspects of MWI theory (like how you actually get the Born probabilities) are unresolved.

titotal 16 Mar 2024 16:35 UTC
13 points
−1
on: ‘Empiricism!’ as Anti-Epistemology
The basic premise of this post is wrong, based on the strawman that an empiricist/scientist would only look at a single piece of information. You have the empiricist and scientists just looking at the returns on investment on bankmans scheme, and extrapolating blindly from there.
But an actual empiricist looks at all the empirical evidence. They can look the average rate of return of a typical investment, noting that this one is unusually high.They can learn how the economy works and figure out if there are any plausible mechanisms for this kind of economic returns. They can look up economic history, and note that Ponzi schemes are a thing that exists and happen reasonably often. From all the empirical evidence, the conclusion “this is a Ponzi scheme” is not particularly hard to arrive at.
Your “scientist” and “empricist” characters are neither scientists nor empiricists: they are blathering morons.
As for AI risk, you’ve successfully knocked down the very basic argument that AI must be safe because it hasn’t destroyed us yet. But that is not the core of any skeptics argument that I know.
Instead, an actual empiricist skeptic might look at the actual empirical evidence involved. They might say hey, a lot of very smart AI developers have predicted imminent AGI before and been badly wrong, so couldn’t this be that again? A lot of smart people have also predicted the doom of society, and they’ve also been wrong, so couldn’t this be that again? Is there a reasonable near-term physical pathway by which an AI could actually carry out the destruction of humanity? Is there any evidence of active hostile rebellion of AI? And then they would balance that against the empirical evidence you have provided to come to a conclusion on which side is stronger.
Which, really, is also what a good epistemologist would do? This distinction does not make sense to me, it seems like all you’ve done is (perhaps unwittingly) smeared and strawmanned scientists.

titotal 14 Jan 2024 13:39 UTC
9 points
1
on: Against most AI risk analogies
I think some of the quotes you put forward are defensible, even though I disagree with their conclusions.
Like, Stuart Russell was writing an opinion piece in a newspaper for the general public. Saying AGI is “sort of like” meeting an alien species seems like a reasonable way to communicate his views, while making it clear that the analogy should not be treated as 1 to 1.
Similarly, with Rob wilbin, he’s using the analogy to get across one specific point, that future AI may be very different from current AI. He also disclaims with the phrase “a little bit like” so people don’t take it too seriously. I don’t think people would come away from reading this thinking that AI is directly analogous to an octopus.
Now, compare these with Yudkowsky’s terrible analogy. He states outright “The AI is an unseen actress who, for now, is playing this character.”. No disclaimers, no specifying which part of the analogy is important. It directly leads people into a false impression about how current day AI works, based on an incredibly weak comparison.

titotal 29 Nov 2023 20:00 UTC
2 points
0
in reply to: Charlie Steiner’s comment on: Thoughts on teletransportation with copies?
Right, and when you do wake up, before the machine is opened and the planet you are on is revealed, you would expect to see yourself in planet A 50% of the time in scenario 1, and 33% of the time in scenario 2?
What’s confusing me is with scenario 2: say you are actually on planet A, but you don’t know it yet. Before the split, it’s the same as scenario 1, so you should expect to be 50% on planet A. But after the split, which occurs to a different copy ages away, you should expect to be 33% on planet A. When does the probability change? Or am I confusing something here?

titotal 27 Nov 2023 11:36 UTC
8 points
4
on: Wikipedia is not so great, and what can be done about it.
While Wikipedia can definitely be improved, I think it’s still pretty damn good.
I really cannot think of a better website on the internet, in terms of informativeness and accuracy. I suppose something like Khan academy or so on might be better for special topics, but they don’t have the breadth that Wikipedia does. Even google search appears to be getting worse and worse these days.

titotal 26 Nov 2023 0:25 UTC
8 points
2
on: The two paragraph argument for AI risk
Okay, I’m gonna take my skeptical shot at the argument, I hope you don’t mind!
an AI that is *better than people at achieving arbitrary goals in the real world* would be a very scary thing, because whatever the AI tried to do would then actually happen
It’s not true that whatever the AI tried to do would happen. What if an AI wanted to travel faster than the speed of light, or prove that 2+2=5, or destroy the sun within 1 second of being turned on?
You can’t just say “arbitrary goals”, you have to actually explain what goals there are that would be realistically achievable by an realistic AI that could be actually built in the near future. If those abilities fall short of “destroy all of humanity”, then there is no x-risk.
As stories of magically granted wishes and sci-fi dystopias point out, it’s really hard to specify a goal that can’t backfire
This is fictional evidence. Genies don’t exist, and if they did, it probably wouldn’t be that hard to add enough caveats to your wish to prevent global genocide. A counterexample might be the use of laws: sure, there are loopholes, but not big enough that the law would let you off on a broad daylight killing spree.
Current AI systems certainly fall far short of being able to achieve arbitrary goals in the real world better than people, but there’s nothing in physics or mathematics that says such an AI is *impossible*
Well, there is laws of physics and maths that put limits on available computational power, which in turn puts a limit on what an AI can actually achieve. For example, a perfect Bayesian reasoner is forbidden by the laws of mathematics.

titotal 22 Nov 2023 18:59 UTC
16 points
4
on: OpenAI: The Battle of the Board
If Ilya was willing to cooperate, the board could fire Altman, with the Thanksgiving break available to aid the transition, and hope for the best.
Alternatively, the board could choose once again not to fire Altman, watch as Altman finished taking control of OpenAI and turned it into a personal empire, and hope this turns out well for the world.
Could they not have also gone with option 3: fill the vacant board seats with sympathetic new members, thus thwarting Altman’s power play internally?

titotal 21 Nov 2023 13:06 UTC
7 points
1
on: “Why can’t you just turn it off?”
Alternative framing: The board went after Altman with no public evidence of any wrongdoing. This appears to have backfired. If they had proof of significant malfeasance, and presented it to their employees, the story may have gone a lot differently.
Applying this to the AGI analogy would be be a statement that you can’t shut down an AGI without proof that it is faulty or malevolent in some way. I don’t fully agree though: I think if a similar AGI design had previously done a mass murder, people would be more willing to hit the off switch early.

titotal 20 Nov 2023 18:50 UTC
1 point
−2
on: For Civilization and Against Niceness
Civilization involves both nice and mean actions. It involves people being both nice and mean to each other.
From this perspective, if you care about Civilization, optimizing solely for niceness is as meaningless and ineffective as optimizing for meanness.
Who said anything about optimizing solely for niceness? Everyone has many different values that sometimes conflict with each other, that doesn’t mean that “niceness” shouldn’t be one of them. I value “not killing people”, but I don’t optimize solely for that: I would still kill Mega-Hitler if I had the chance.
Would you rather live in a society that valued “niceness, community and civilization”, or one that valued “meanness, community and civilization”? I don’t think it’s a tough choice.
I think that being mean is sometimes necessary in order to preserve other, more important values, but that doesn’t mean that you shouldn’t be nice, all else being equal.

titotal 15 Nov 2023 12:03 UTC
8 points
7
in reply to: Michael Tontchev’s comment on: When did Eliezer Yudkowsky change his mind about neural networks?
Partially, but it is still true that Eliezer was critical of NN’s at the time, see the comment on the post:
I’m no fan of neurons; this may be clearer from other posts.