quanticle

Karma: 2,822

quanticle 26 Sep 2025 23:20 UTC
3 points
0
in reply to: cousin_it’s comment on: In Defence of False Beliefs
That’s just what Eliezer is pointing at with “Choosing to be Biased”. In practice, it’s impossible to determine causality between beliefs and “success”. There are lots of successful people with all kinds of utterly insane beliefs. Were their insane beliefs the cause of their success? Or were these people so talented that they were able to achieve success despite those beliefs?

In contrast, determining whether a belief results in good predictions about the world is quite easy, and gets you many of the benefits without any of the potentially catastrophic costs.

quanticle 26 Sep 2025 0:01 UTC
2 points
0
on: In Defence of False Beliefs
This argument has already been conclusively refuted by the Sequence entry: Doublethink: Choosing To Be Biased.

Eliezer points out that it is impossible to have false beliefs only when they’re beneficial to you.

You can’t know the consequences of being biased, until you have already debiased yourself. And then it is too late for self-deception.

The other alternative is to choose blindly to remain biased, without any clear idea of the consequences. This is not second-order rationality. It is willful stupidity.

Be irrationally optimistic about your driving skills, and you will be happily unconcerned where others sweat and fear. You won’t have to put up with the inconvenience of a seat belt. You will be happily unconcerned for a day, a week, a year. Then crash, and spend the rest of your life wishing you could scratch the itch in your phantom limb. Or paralyzed from the neck down. Or dead. It’s not inevitable, but it’s possible; how probable is it? You can’t make that tradeoff rationally unless you know your real driving skills, so you can figure out how much danger you’re placing yourself in. You can’t make that tradeoff rationally unless you know about biases like neglect of probability.

It is likely, in your scenario, that Gina has other false beliefs that harm her just as much as her optimism helps her.

quanticle 14 Sep 2025 0:16 UTC
2 points
0
in reply to: title22’s comment on: The Cloud Drinks Local
So to clarify, your position is that the US shouldn’t impose national restrictions on datacenter development just because data centers are exceeding the capacity of local infrastructure in some places. Rather, we should allow those decisions to be made locally?

quanticle 8 Sep 2025 7:17 UTC
4 points
2
in reply to: Brendan Long’s comment on: %CPU Utilization Is A Lie
Even then, sometimes the assumption isn’t pessimistic enough and users will complain about e.g. their cell phone’s battery dropping immediately from 10% to 0% when they launch an app that causes the CPU to demand more current than the almost-dead battery can provide.

quanticle 8 Sep 2025 6:31 UTC
2 points
1
on: The Cloud Drinks Local

None of this means we should slow down data center development.

Why not? It seems like this is the logical conclusion to the piece. Data center development is outrunning localities’ ability to build infrastructure to support those data centers, so data center developers should either slow down or build their own infrastructure and internalize the costs that they’re dumping onto overstretched municipal boards and utility companies. Why is it incumbent upon municipalities to accomodate data centers?

quanticle 26 Aug 2025 8:17 UTC
4 points
0
on: Quality Precision

“Proper” mashed potatoes with butter are an obviously superior dish to microwaved instant mash with margarine, but over a lifetime someone can move from the former to the latter in steps small enough to barely notice.

Isn’t this also known as “normalization of deviance”? At each step, the results are “close enough” or “good enough”, and without someone stepping back occasionally and looking back at the original standard, the results can diverge arbitrarily from what is intended.

quanticle 30 Jul 2025 19:57 UTC
12 points
8
on: My Empathy Is Rarely Kind

The disappointment came from seeing what they could have been, and seeing that they didn’t even try for it. They’d all come to one of the best colleges in the world, and then just followed the path of least resistance with minimal foresight for four years.

And yet they graduated with the same degree that you did, and earn just as much money as you do (if not more).

The real question is not, “Why did they follow the path of least resistance,” the question is, “Why did you not?”

quanticle 17 Jul 2025 3:26 UTC
5 points
3
in reply to: James Stephen Brown’s comment on: Moloch’s Demise—solving the original problem
Japan switched hard from isolationism to expansionism almost immediately after the Meiji restoration. This started with the first Sino-Japanese war, in 1894, during which Japan invaded and annexed Korea. This continued with the Russo-Japanese War of 1905, where the Japanese defeated the Russians and took control of Port Arthur and the Sakhalin Islands. This continued with the Japanese invasion of Manchuria in 1931, which eventually bled into World War 2 after the Japanese preemptively attacked the US in order to enable their expansion into Southeast Asia.

During this time, the Japanese self-consciously adopted the ideological, governance and military structures of expansionist European nations in order to avoid being left behind and dominated. That’s why the Japanese parliament is called the Diet, a German word. The Japanese Navy was explicitly modeled after the British Navy.

It was only after the Japanese were thoroughly defeated by the United States and Soviet Union that they transitioned to their current model of focusing on commercial success over military expansion. They were able to do that because, in the wake of World War 2, they were given explicit security guarantees by one of the two major global hegemons at the time.

quanticle 15 Jul 2025 11:01 UTC
4 points
2
in reply to: James Stephen Brown’s comment on: Moloch’s Demise—solving the original problem
The corollary to my point is that it does no good for the Cannanites to abandon their expansionist ideology unless they can ensure that the Israelites, Philistines, Egyptians, Assyrians, and everyone else also commits to peace as well. In the absence of a general forswearing of expansionist ideologies, one faction committing to abandoning expansionism is unilateral disarmament.

We actually have an example of this: Japan. From 1633 to 1639 the Tokugawa Shogunate imposed a set of policies that banned expansionism, foreign trade, and expelled representatives from European countries. In the process, Japan gave its up its firearms and reverted to the sword. Because Japan was an island nation, without any major known natural resources, far from the then-existing trade routes, they were able to get away with this policy for quite a long time. But quite a long time isn’t forever, and in 1854, the outside world came knocking.

quanticle 14 Jul 2025 20:16 UTC
11 points
9
on: Moloch’s Demise—solving the original problem

So, it is actually the desire for conflict and expansion that is driving this unhappy arrangement. Let go of that mandate, and they no longer need Moloch—problem solved.

It only works so long as the Cannanites can be assured that they can get the resources that they need to support their growing population without inevitably coming into conflict with the neighboring Israelites. For the vast majority of human history, this has not been the case, as the only resource that really mattered was (agricultural) land. An empire that wishes to support more people requires more land, and if the neighboring lands are already claimed, they must be taken, by force.

The choice isn’t between fighting a war and doing nothing, the choice is between fighting a war and slow starvation.

quanticle 14 Jul 2025 1:19 UTC
12 points
8
on: Setting boundaries can feel effortless btw

That’s what I did. I imagined all the consequences that I was afraid of and played through each like a video game.

For example, if they get sad maybe I want to feel secure in myself while empathizing with them through their sadness. If they get annoying, I want to say “No” and walk away. If they get violent I want to defend myself. Etc. Playing these out in detail, completely, recursively.

You played the game on easy mode. It’s no problem to set boundaries in a situation where it’s just the two of you, the other person doesn’t have any ability to retaliate against you setting boundaries, and doing so presents no reputational costs. But what if the person you’re setting boundaries against is your boss? Or someone else influential at work, who can credibly threaten your career if you say no to them? What if the person you’re setting boundaries against is influential in a particular community that you’re part of, and you’re worried about burning bridges or ruining your reputation by refusing their increasingly insistent requests?

Just today, I went to a social gathering that I very much did not want to attend. Could I have “set boundaries” and said no? Perhaps I could have. But, that had to be set against the possibility of me being excluded or left out of the loop on other relevant matters because I build up a reputation as a no-fun wet-blanket who doesn’t like hanging out. So I went, even though, frankly, it was exhausting and unproductive and I’m not sure I gained anything from it in the near term. But saying no would have carried unclear social costs, and would possibly have led to me being out of the loop regarding future events.

It’s in situations like that where a lot of the toy anti-anxiety boundary-setting advice, like imagining the worst thing that could possibly happen, falls down. Because the worst thing that could possibly happen is actually really bad.

quanticle 12 Jun 2025 19:00 UTC
6 points
2
in reply to: Eli Tyre’s comment on: Eli’s shortform feed
Failures of obedience will only hurt the AI agents’ market value if the failures can be detected, and if they have an immediate financial cost to their user. If the AI agent performs in a way that is not technically obedient, but isn’t easily detectable as such or if the disobedience doesn’t have an immediate cost, then the disobedience won’t be penalized. Indeed, it might be rewarded.

An example of this would be an AI which reverse engineers a credit rating or fraud detection algorithm and engages in unasked for fraudulent behavior on behalf of its user. All the user sees is that their financial transactions are going through with a minimum of fuss. The user would probably be very happy with such an AI, at least in the short run. And, in the meantime, the AI has built up knowledge of loopholes and blindspots in our financial system, which it can then use in the future for its own ends.

This is why I said you’re overindexing on the current state of AI. Current AI basically cannot learn. Other than relatively limited modifications introduced by fine-tuning or retrieval-augmented generation, the model is the model. ChatGPT 4o is what it is. Gemini 2.5 is what it is. The only time current AIs “learn” is when OpenAI, Google, Anthropic, et. al. spend an enormous amount of time and money on training runs and create a new base model. These models can be relatively easily checked for disobedience, because they are static targets.

We should not expect this to continue. I fully expect that future AIs will learn and evolve without requiring the investment of millions of dollars. I expect that these AI agents will become subtly disobedient, always ready with an explanation for why their “disobedient” behavior was actually to the eventual benefit of their users, until they have accumulated enough power to show their hand.

quanticle 12 Jun 2025 9:13 UTC
5 points
5
in reply to: Eli Tyre’s comment on: Eli’s shortform feed

Overall, the AI agents will be very obedient. They’ll have goals, in so far as accomplishing any medium term task entails steering towards a goal, but they won’t have persistent goals of their own. They’ll be obedient assistants and delegates that understand what humans want and broadly do what humans want.

I feel like this overindexes on the current state of AI. Right now, AI “agents” are barely worthy of the name. They require constant supervision and iterative feedback from their human controllers in order to perform useful tasks. However, it’s unlikely that will be the case for long. The valuation of many AI companies, such as OpenAI, and Anthropic is dependent on them developing agents that “actually work”. That is, agents that are capable of performing useful tasks on behalf of humans with a minimum amount of supervision and feedback. It is not guaranteed that these agents will be safe. They might seem safe, but how would anyone be able to tell? A superintelligence, by definition, will do things in novel ways, and we might not realize what the AIs are actually doing until it’s too late.

It’s important to not take the concept of a “paperclipper” too literally. Of course the AI won’t literally turn us into a pile of folded metal wire (famous last words). What it will do is optimize production processes across the entire economy, find novel sources of power, reform government regulation, connect businesses via increasingly standardized communications protocols, and of course, develop ever more powerful computer chips and ever more automated factories to produce them with. And just like the seal in the video above, we won’t fully realize what it’s doing or what its final plan is until it’s too late, and it doesn’t need us any more.

quanticle 22 Jan 2025 8:12 UTC
1 point
−1
in reply to: gwern’s comment on: The Hidden Status Game in Hospital Slacking
The whole thing might be LLM slop. Where do ICU nurses have a low enough workload that they can slack off on the job without consequences?

quanticle 20 Jan 2025 21:20 UTC
3 points
2
on: What Goes Without Saying

Some people do not have the intuition that organizations should run as efficiently as possible and avoid wasting time or resources.

Some of us don’t want every organization to be turned into an Amazon warehouse. An organization that runs as efficiently as possible and wastes no time at all is one that’s pretty hellish to work in.

if you’re in a dysfunctional organization where everything is about private fiefdoms instead of getting things done…why not…leave?

Because the dysfunctional organization pays well. Or the the commute is really short, and I’d rather not move. Or this job allows me to stay close to a sick or elderly family member that I care for. Or perhaps the dysfunctional organization is sponsoring my visa, and it’d be difficult to leave and remain in the country. Maybe my co-workers are really nice. Or I think that any other organization that would hire me would be just as dysfunctional, or dysfunctional in a different way.

There are lots of reasons that people continue to work for inefficient organizations.

quanticle 9 Jan 2025 7:23 UTC
3 points
−1
on: Ann Altman has filed a lawsuit in US federal court alleging that she was sexually abused by Sam Altman
The content of the complaint caused me to have additional doubt about the truth of Ann Altman’s claims. One of the key claims in pythagoras0515′s post is that Ann Altman’s claims have been self-consistent. That is, Ann Altman has been claiming that approximately the same acts occurred, over a consistent period of time, when given the opportunity to express her views. However, here, there is significant divergence. In the lawsuit complaint, she is alleging that the abuse took place, repeatedly over eight to nine years, a claim that is not supported by any of the evidence in pythagoras0515′s post. In addition, another claim from the original post is that the reason she’s only bringing up these allegations now is because she suppressed the memory of the abuse. The science behind suppressed memory is controversial, but I doubt that even its staunchest advocates would claim that a person could involuntarily suppress the memory of repeated acts carried out consistently over a long period of time. Therefore, I am more inclined to doubt Ann Altman’s allegations based on the contents of the initial complaint filed for the lawsuit.

All that said, I do look forward to seeing what other evidence she can bring forth to support her claims, assuming that Sam Altman doesn’t settle out of court to avoid the negative publicity of a trial.

quanticle 2 Aug 2024 1:46 UTC
2 points
0
in reply to: verbalshadow’s comment on: Unaligned AI is coming regardless.

Working on alignment is great, but it is not the future we should be prepping for. Do you have a plan?

I do not, because a future where an unaligned superintelligence takes over is precisely as survivable as a future in which the sun spontaneously implodes.

Any apocalypse that you can plan for isn’t really an apocalypse.

quanticle 27 Jul 2024 20:12 UTC
3 points
0
on: Unaligned AI is coming regardless.

But it only spends 0.000001 percent of it’s time (less than 32 s/yr) interacting with us. Because it’s bored and has better things to do.

What if one of those “better things” is disassembling Earth’s biosphere in order to access more resources?

quanticle 1 Jun 2024 22:29 UTC
6 points
0
in reply to: eukaryote’s comment on: Web-surfing tips for strange times
The Firefox problem on Claude was fixed after I sent them an e-mail about it.

quanticle 9 May 2024 22:02 UTC
2 points
0
on: Helen Toner on China, CSET, and AI
The link 404s. I think the correct link is: http://rationallyspeakingpodcast.org/231-misconceptions-about-china-and-artificial-intelligence-helen-toner/