Mis-Understandings

Karma: 315

Mis-Understandings 29 Mar 2026 0:30 UTC
2 points
0
on: Nick Bostrom: How big is the cosmic endowment?
If there is weirder physics, such that FTL or relaxations to the laws of thermodynamics are possible, I assume that the estimation increases. Then again, under those conditions there may not be a finite upper bound.

Mis-Understandings 7 Mar 2026 20:35 UTC
1 point
0
on: Can governments quickly and cheaply slow AI training?
There is another two verification routines, which is looking over shoulders at internal documents, and banning new releases.
There is also just checking for the presense of the registered weights in the gpu’s local rap, which produces a memory tax big enough that there is not space to fit the intermediate values of the gradient. This requires that the monitors have code on the running machines, but periodic memory dumps can be a surveilance mechanism by verifying that all models on gpus match a model known to be already trained, which stops new initialization and thus new runs.

Mis-Understandings 1 Mar 2026 22:03 UTC
1 point
0
in reply to: Mis-Understandings’s comment on: I’m Bearish On Personas For ASI Safety
That is to say, this is assuming that there is a difference in type between the sorts of heuristics that a pretrained and not superhuman LLM will reach for, and those necessary to be superinteligent. There is always the chance that you just select for regular engineering, but you always reach for the right branch first. Since the right branch is also one that the regular persona would have generated, then the number of bits of selction towards danger is at most the number of bits of selection between a safe and a RLed persona.
This model has personas as moral up until the RL step that makes them sufficiently inhuman.

Mis-Understandings 1 Mar 2026 21:51 UTC
1 point
0
on: I’m Bearish On Personas For ASI Safety
the base LLM has never seen a superintelligence in its pre-training corpus
Is “The LLM, but lucky on sampling”, something not in the corpus. It seems that that is exactly the corpus GRPO generates.

Mis-Understandings 24 Feb 2026 1:48 UTC
3 points
1
in reply to: Josh Snider’s comment on: The 2028 Global Intelligence Crisis—a finance-oriented vignette
They could. In the scenario where they do not, you get Japan like problems of not enough inflationary firepower. To do that with confidence requires the tax base to support it, which means as labor share of income drops, taxing the capital share of income as well.
Taxing the capital share of income is not the end of the world, it is a relatively normal policy, but we would have to generate the political will to actually do so, the government must be able to credibly collect those taxes.
If we think the funds for helicopter money come from government borrowing, and the bond market breaks due to unexpected deflation, the government cannot get the money to pay for the inflation to get the market to get the money (this is what is meant by systemic risk), or default
If we think the funds come from future taxes, if the capital share of income increases and the willingness to consume from accumulated capital drops, the government must increase taxes on the income streams of capital or labor to make up consumption
If you think MMT is correct, and the government can just do helicopter, they will have to do that to keep equilibrium.
All of those cases (helicopter money and capital taxation) as specific policies are major policy shits. There is always a chance that does not happen.
There is a big difference between “The Fed Can” and “The Fed Will”, and if you are at 95% odds of the feds getting it right, that is still 5% of not fixing it. That is what is meant by left tail risk.

Mis-Understandings 24 Feb 2026 1:46 UTC
4 points
0
in reply to: Noosphere89’s comment on: The 2028 Global Intelligence Crisis—a finance-oriented vignette
Remember the top line, where they talked about this being a model of left tail risk (for investors).
They are predicting specifically that there is reasonable odds of a case where the stimulus is underbaked and too late, causing significant deflation, and a large number of past claims on future economic activity do not survive the restructure, and so every actor taking interest rate risk is at risk of getting blown out by deflation, or taking huge nominal losses from a default due to their counterparties getting blown out by deflation.

Mis-Understandings 24 Feb 2026 1:06 UTC
5 points
1
in reply to: Rochambeau’s comment on: The 2028 Global Intelligence Crisis—a finance-oriented vignette
They are explicitly predicting that if there is massive AI productivity growth, there is no guarantee that the government will take the opportunity to print and spend to generate enough inflation to prevent the breakdown of basically all loans to extreme deflation. Somebody who found a way to exit into something safe from extreme deflation was fine, but everyone who misidentified it is soaked.
it might be that the only safe asset is datacenter shares, or something, and everybody who is not in that has whatever they are holding deflated to nothing.
The deal they predict looks like
1. We play musical chairs with all the claims on future production
2. There is way more future production to claim.
Even with a fairness guarantee, that can still be scary.

Mis-Understandings 12 Feb 2026 21:20 UTC
3 points
0
in reply to: Bronson Schoen’s comment on: 1a3orn’s Shortform
The counterargument against continous tokens being passed forwards is that if you want to use neuralese, you have to give up sampling, since the big idea of latent reasoning is to not pass through the random discretization of sampling a token. But random discretization is itself powerful, especially with the possibility of a useful bias. If you give it up, the model becomes deterministic, so it can’t use Best of N. If Best of N or tree search on chains of thoughts is really important, either in training or in deployment, that is something that is not really compatible with the latent paradigm, in addition to the difficulty of training data.
The argument against semantic drift/Thinkish is extremely weak, and we should expect semantic drift when training with self play without countermeasures.

Mis-Understandings 5 Feb 2026 20:44 UTC
9 points
7
on: The nature of LLM algorithmic progress
For running out of optimizations, there is another factor. And that is that some amount of re-optimization is needed each time you change hardware. Consequentially it is really hard to separate hardware progress (the new NVIDIA chips are better) from optimization of the software that runs on them (Because they have new features, the use of which is an optimization)
What links here?
- Petropolitan's comment on The nature of LLM algorithmic progress (v2) by Steven Byrnes (6 Feb 2026 20:08 UTC; 3 points)

Mis-Understandings 30 Jan 2026 23:10 UTC
3 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform
This analysis assumes that the government of Taiwan has no power, that the US is the one to negotiate with. If you are thinking of this, the question is not what deals the US would agree to, but what deal Taiwan would agree to. The US is not actually in a position to veto a reunification agreed between both governments, though they likely would not be able to backstop it either (which creates a little negotiating friction).
All of these concession are concessions to the US. What really matters for a peaceful settlement (since Taiwan can destroy the surplus) is whether the CCP can give meaningful concessions on the terms of reunification with Taiwan.
That is to say, there is no “process to gradually and peacefully hand over Taiwan to China” as an option for the US. It might exist for the government of Taiwan, but because the US does not actually control any of it’s partners among middle powers.
as for “China care more about Taiwan than anybody else. ”
This is false
The US might care less, but the government of Taiwan probably cares more.

Mis-Understandings 26 Jan 2026 17:53 UTC
46 points
8
on: Ada Palmer: Inventing the Renaissance
The big problem with pre-x LLMs, especially into the deep past, is sampling bias. That is to say, the set of texts that survive is not the set of texts that were present, which is an extra bias on the already biased data collection for regular LLMs. There is also the temptation to support (especially for placing a LLM as a strategic) with analysis and data that comes after the period, which was collected with a hindsight bias for what is interesting.

Mis-Understandings 21 Jan 2026 20:20 UTC
6 points
3
on: Claude Codes #3
So far, I have not had Claude Code refuse a request from me, not even once.
When Claude reinterprets the request to be “the request, but safe”, this is invisible, and good on the margin.
I think doing this is mostly fine.

Mis-Understandings 16 Jan 2026 21:14 UTC
1 point
0
in reply to: Elizabeth’s comment on: Elizabeth’s Shortform
The problem with spontaneous agreement is that explicit agreement gives a chance for parties A and B to exchange hidden information without letting third parties know.
The reason for agreements being formalized is also because since you need to negotiate to reach the efficient settlement, since you have to reveal secret information, but publicly implementing the results of secret negotiations leads to suspiscion of secret alliances. If your public allies think you have secret allies, they are not happy, since that is bad for them. So publicizing agreements makes you more predictable, which is good for you.

Mis-Understandings 1 Jan 2026 4:38 UTC
1 point
0
on: Chromosome identification methods
I am assuming the first step is to count the chromosomes, and isolate that you have exactly a collection of full sets, which came from the same cells, so that the property, (there are k of each exactly) holds. I would need to look in literature to see the sucess rate of isolating all chromosomes, not just (at least one)
It seems for this method you must get 22 sucessful sequences in a row, which is hard if sequencing fails even sometimes, or you lose a chromosome even sometimes.

Mis-Understandings 20 Dec 2025 20:36 UTC
1 point
0
in reply to: Chr Cz’s comment on: Chr Cz’s Shortform
For 1. resampling is not (asking the AI to try again) it is forcibly rewriting it’s history so that it made a different one of its plausible decisions. Every resampled continuation plausibly could have been the original continuation, given a different rng seed, and there is no nonshared place to put a scratchpad that would pass the information of the resampling event. It would probably increase the risk that you resample into a less suspicious bad path, but it is outweighed by the fact that the AI has to mostly bide it’s time (in order to not get caught in the fraction with good monitoring done in the first requrests) (this is a previous important result), so on net it is a gain.
For 2. The model used in this paper is investigating that the untrusted model is behaving optimally given it’s ability to actually do sneaky bad stuff. This requires calculating the optimum knowing the monitoring protocol, so the untrusted AI must know about the monitor. There is not in this model (does not know about the monitoring setup), because if the model actually thinks that there is no monitoring, it just tries immediately and then loses. If it thinks there is monitoring, it is really hard to trick it about the monitoring strategy.
for 3. No, you want to use the same model, since you can prevent the resampled branches from knowing about each other (basically they only share the shared history (history before resampling) and weights). This is basically what you would want from another “instance”. The modeling assumption is that there is exactly one untrusted AI, which wins if any path breaks out, so there is not any meaningfully different untrusted AI to ask to try from that starting point.

Mis-Understandings 2 Nov 2025 15:53 UTC
1 point
0
in reply to: quetzal_rainbow’s comment on: Decision theory when you can’t make decisions
One box P1 and Two box P1 are not the only strategies. You can also try to find some source of randomness to take a mixed strategy from. If you are not able to take mixed strategies, Game theory and decision theory are not guaranteed to converge.

Mis-Understandings 2 Nov 2025 1:31 UTC
7 points
2
on: Decision theory when you can’t make decisions
How does the typical construction of Newcomb’s problem handle mixed strategies?
1. The predictor can read your RNG, so your strategy is not really mixed. But this is a violation of game theory assumptions about the possibility of mixed strategies. If you are banned from mixed strategies, Game theory does not work so well, as I understand it.
2. The predictor has some utility distribution over (One box, two box) (Predict one box, Predict two box), and you can solve the Nash equilibrium, maybe to a mixed strategy equilibrium (where you pick one box the percentage) (IE. you want to one box often enough that the Predictor predicts that you will, but no more than that, based on the relative payouts)
3. The predictor has some policy on P(Place in box) for each P (onebox), and there the maximum utility might not be at P(onebox)

Mis-Understandings 14 Oct 2025 0:54 UTC
2 points
−1
on: If Anyone Builds It Everyone Dies, a semi-outsider review
The question of why no “might kill us” as a class is simple. There is such a class, but if it lost the fight to kill us, it obviously was not ASI (picking a fight with the world and losing is pretty dumb), or it might win, at which case it won, we die. And then we will be in the same scenario for every AI stronger than it, and for AI weaker than it that might yet get lucky, just as we might get lucky and win at bad odds. The next AI we make will also want to fight us for the same reasons, and we will need to either fight it to (including preemptively, e.g. turning it off because a dumber model did something), or get a reason to believe that we will never fight it. And if you know you will fight your AI eventually, and you will win now, fight now.

Mis-Understandings 6 Oct 2025 14:04 UTC
11 points
9
on: LLMs are badly misaligned
if it were competent enough to reflect and actively reconcile its own inner drives
Why do we think that reflection is neccesary for competence. That is, competence does not seem to imply coherence, unless I missed an argument.

Mis-Understandings 3 Oct 2025 11:26 UTC
36 points
2
on: No, That’s Not What the Flight Costs
This analysis is slightly incorrect, since if rewards programs are treated as assets, they contribute to net enterprise value, not to market cap. Market cap mostly subtracts loans outstanding from the total expected returns of the enterprise. American, for instance, has 37 billion dollars out in debt, and the market cap is the expected value above the expected payout to the loans.
So in this analysis, American is actually worth 47 billion-ish dollars. 37 billion of those are owned by the creditors, and 10 billion is owned by the shareholders. 22 billion of that worth is the rewards program, and the rest is the rest of the business.
If american’s rewards program disappeared, the creditors would take a haircut, but they would not get nothing.
You can do the same for all of these.
As an aside, every time I have seen someone saying this, they neglect to model debt. I think it might be systematically confusing.
What links here?
- Halfhaven Digest #1 by Taylor G. Lunt (8 Oct 2025 14:24 UTC; 15 points)