LTM

Karma: 52

LTM 16 Mar 2026 16:23 UTC
2 points
0
in reply to: MichaelDickens’s comment on: AI Safety Needs Startups
This is a sentiment that I’ve heard often when discussing AI safety for-profits; that they often abandon their original mission in favour of a neutral or actively harmful objective. I am not aware of enough examples to treat this as an established pattern.
Anthropic is the strongest example, starting explicitly as a safety company and now clearly accelerating the frontier. However, they have used their position to recently advocate for restricted use of autonomous weapons and a curtailment of mass surveillance. Anthropic shows that this kind of influence is possible (although whether they have been net positive is a more complicated question). I am not aware of enough examples of AI safety for profit startups being derailed to agree that this is a structural flaw we should treat as disqualifying.
By some estimates the ratio of AI safety to capabilities funding is roughly 1:250. In this kind of situation, it seems less important to find approaches which are likely not net negative (such as many AI safety research orgs) and more important to find approaches which could be strongly net positive. As I outlined, I think that poor alignment exists in research and advocacy orgs as well and that for-profits are not uniquely pre-disposed to harmful effects. I believe that there is a significant cost to being so risk-averse about a whole category of intervention.
I’m curious about your thoughts on:
1. Evidence for frequent failures of AI safety startups. Many people I have spoken to believe this, although I don’t see the examples I would expect to. (Depending on your views of frontier labs, these may be a small number of failures which nevertheless give the for-profit structure a strongly net negative impact. However, this specific accelerationist failure mode might be unique to the lab structure rather than companies in aggregate?)
2. How these ideas translate to research and advocacy orgs. What bar for integrity and clarity of plan should we demand from them? Non-profits have their own misalignment pressures, and I’m not sure the bar applied to them is equally demanding.

LTM 16 Mar 2026 16:02 UTC
1 point
0
in reply to: cousin_it’s comment on: AI Safety Needs Startups
I’m not convinced that the labs are able to justify their race behaviour because they have access to alignment research. It seems to me that the race is justified by money, gestures to national security, or a vague bias towards building rather than some guarantee that this technology helps humanity. If AI stops making obvious mistakes, it might become slightly harder to galvanise public support for a pause, but I doubt overall it would have a high impact on the trajectory of the technology.
I think of the push towards AI safety companies as improving access to capital, information, and integration for safety work. The ability to integrate monitoring solutions into sensitive industries, catch jailbreaks at runtime, or continually evaluate models is extremely valuable, and I claim that this is easier to do using a for-profit model.

LTM 6 Oct 2025 15:59 UTC
3 points
0
in reply to: AnthonyC’s comment on: AGI Companies Won’t Profit From AGI
This seems very difficult to predict to me. Whether assets recorded on paper or digitally retain any meaning post-TAI depends on how well information can be shared, whether the courts continue to function as they did before TAI’s development, and whether power centralises enough for widespread theft to be sustained.
My intuition is that the value of such assets will spike as AI capabilities increase pre-AGI. The labour impacts will push more people afraid of being made unemployed to invest in capital driving up asset prices. Accelerated R&D from AI will presumably result in the discovery of more economically efficient uses of existing resources, driving up the prices of land, minerals, electricity etc. up even more.

Whether capital retains value post-AGI seems to come down to whether you can track your ownership effectively. I’ve written arguing that reality-collapse leads to an effective end to property rights here—https://www.lesswrong.com/posts/nFrNEMFi2EwodEXXz/keeping-capital-is-the-challenge. In summary, if digital artefacts become trivial to forge, you may not be able to trace the theft of your assets or seek retribution meaningfully. Capital retains value, but you do not retain your capital.

LTM 25 Sep 2025 11:48 UTC
1 point
0
in reply to: StanislavKrym’s comment on: AGI Companies Won’t Profit From AGI
I think you are correct in that success results in the automation of labour and increasing the capital share of economic output to functionally 100%.
I do not think it follows that this grants investors in these AI labs part of the lightcone. If they are locked in perfect competition, there is no surplus to return to investors. Investors in natural resources, land, or perhaps data collection companies could expect everything they own to become vastly more valuable. But the investors in AI labs see their investment locked in a situation where profit margins are zero, and fail to claim the portion of the lightcone that capital might have otherwise offered them.

LTM 25 Sep 2025 11:44 UTC
1 point
0
in reply to: AnthonyC’s comment on: AGI Companies Won’t Profit From AGI
I think this is broadly correct. Investors in AI labs early on likely viewed their investment more like a donation than a traditional financial transaction.
My point is more narrow. That RSI → homogeneous product → low profit margins in the long run assuming similar hardware and data availability. The data and feedback signals these labs are able to collect post-AGI might even be enough for them to provide a large return on investment for their investors who are financially minded.
I think this kind of model is worth bearing in mind for individuals who believe capital will retain relevance post-AGI, and who are currently trying to grow their capital by investing in AI labs or companies dependent on them.

LTM 4 Feb 2025 22:58 UTC
3 points
0
in reply to: Nathan Helm-Burger’s comment on: Keeping Capital is the Challenge
I agree that in a takeover scenario where AI capabilities rush wildly ahead of human understanding or control, the ability of the world’s second species to retain exclusive resource access will be limited. This is a plausible future, but it is not the only one. A lot of effort is, and very likely will continue to be, directed towards controlling frontier AI and making it as economically beneficial for its owners as possible.
If this work bears fruit, a world where AI is made by people with capital for people with capital seems very likely.
The French nobility had their capital taken by force by revolutionaries in a violent takeover—the kind which AI may execute. An analogy for the kind of situation I am concerned about would be some portion of the French nobility making the laws so complex, the financial structure of the kingdom so riddled with loopholes, that the other nobles were disenfranchised by due process rather than violence.
Taking this analogy further, the French king had to care about the nobility, and the nobility about the peasantry, in part because of their productive capacity. As described in The Intelligence Curse — LessWrong, when AGI renders your work valueless the incentive for the state to invest in your welfare greatly decreases.

LTM 4 Jan 2025 15:53 UTC
1 point
0
on: Preference Inversion
I think this post is interesting, although I don’t particularly agree with the conclusions. I think it is helpful to think about the formation of your mind and goals—a tradition which I know goes back to Rousseau and most likely goes back further (I am not very knowledgeable on the topic).
I think a lot of the difficulty goes back to the distinction between ‘real’/‘intrinsic’ and goals and those people proport to believe in. Looking at your example of a Christian sexual prude, nothing about their behaviour implies to me that these virtues of chastity and distance from sex are false at all. Perhaps had they been raised in the absence of society, they would think differently. But if that were the case their views on fashion would probably vastly differ also, and yet we don’t consider their desire to dress a certain way to be falsified.
To be clear, I am not claiming that values cannot be gained societally. More that the issue at play is not the creation and adoption of these values from a childhood of coercive exposure, but that these values are hard to satisfy. Sexual prudishness makes it difficult to fulfil inbuilt sexual desires, and so combining both into a single person leaves them less overall satisfiable.
But this goes far beyond societal impacts. In a world of scarce resources, our many inbuilt physical desires as well as our entirely inbuilt desire to behave altruistically conflict in a similar way. It is difficult to be both feared and loved, as some people seem to want to be. Your concern might be better framed at this kind of conflict dragging down one’s maximum available utility than a conflict between inbuilt and constructed desires.
Nice post! I’d be interested to see what you think about the above.
As a side note, you mention that men are apparently better at aligning their sexual preferences to those of society than women. This could be evidence of men being hardwired to adopt the expectations of sexual prudishness to their core. But it could also be that men shaped those expectations to begin with through generations of societal dominance. Or a myriad of other factors which mean that for a reason entirely distinct from inbuilt preference falsification, men’s desires just line up better to those of society’s than women’s.

LTM 15 Dec 2024 16:51 UTC
7 points
2
on: Why Don’t We Just… Shoggoth+Face+Paraphraser?
I’m really excited about this, but not because of the distinction drawn between the shoggoth and the face. Applying a paraphraser such that the model’s internal states are repeatedly swapped for states which we view as largely equivalent could be a large step towards interpretability.
This reminds me on the concept that CNNs work as they are equivariant under translation. The models can also be made (approximately) rotationally equivariant by applying all possible rotations for a given resolution to the training data. In doing this, we create a model which does not use absolute position or orientation which turn out to be excellent priors for generalisable image categorisation among many other tasks.
We decided upon transformations which our model should be invariant under, and applied them to training data. Doing the same to internal states periodically across the model’s activations could force the internal messaging to not only be interpretable to us, but share our approach to irrelevant details.
There would likely be a negative alignment tax associated with this. However, this seems to be a broadly applicable approach to improving interpretability in other contexts.
Consider an image generation model which is configured to output progressively higher resolution and fleshed out generations. If asked to generate a dog, perhaps our ‘paraphraser’ could swap out parts of the background or change the absolute position of the dog in the image, making fewer changes as the product is filled in. If our model works well, this should give us greater diversity of outputs. If it is failing, a change of scenery from a grassy field to a city block could cause the model to entirely diverge from the prompt by generating a telephone box rather than a dog. This could show erroneous connections the model has drawn and unveil details about its functioning which impede proper generalisation.

LTM 17 Nov 2024 13:38 UTC
5 points
0
in reply to: Seth Herd’s comment on: Making a conservative case for alignment
I agree with your points about avoiding political polarisation and allowing people with different ideological positions to collaborate on alignment. I’m not sure about the idea that aligning to a single group’s values (or to a coherent ideology) is technically easier than a more vague ‘align to humanity’s values’ goal.
Groups rarely have clearly articulated ideologies—more like vibes which everyone normally gets behind. An alignment approach from clearly spelling out what you consider valuable doesn’t seem likely to work. Looking to existing models which have been aligned to some degree through safety testing, the work doesn’t take the form of injecting a clear structured value set. Instead, large numbers of people with differing opinions and world views continually correct the system until it generally behaves itself. This seems far more pluralistic than ‘alignment to one group’ suggests.
This comes with the caveat that these systems are created and safety tested by people with highly abnormal attitudes when compared to the rest of their species. But sourcing viewpoints from outside seems to be an organisational issue rather than a technical one.

LTM 13 Oct 2024 17:47 UTC
5 points
0
on: My theory of change for working in AI healthtech
One method of keeping humans in key industrial processes might be expanding credentialism. Individuals remaining control even when the majority of the thinking isn’t done by them has always been a key part of any hierarchical organisation.
Legally speaking, certain key tasks can only be performed by qualified accountants, auditors, lawyers, doctors, elected officials and so on.
It would not be good for short term economic growth. However, legally requiring that certain tasks be performed by people with credentials machines are not eligible for might be a good (though absolutely not perfect) way of keeping humans in the loop.

LTM 16 Jun 2024 12:25 UTC
1 point
0
on: Safety isn’t safety without a social model (or: dispelling the myth of per se technical safety)
Broadly agree, in that most safety research expands control over systems and our understanding of them, which can be abused by a bad actor.
This problem is encountered by for-profit companies, where profit is on the lines instead of catastrophe. They too have R&D departments and research directions which have the potential for misuse. However, this research is done inside a social environment (the company) where it is only explicitly used to make money.
To give a more concrete example, improving self-driving capabilities also allows the companies making the cars to intentionally make them run people down if they so wish. The more advanced the capabilities, the more precise they can be in deploying their pedestrian-killing machines onto the roads. However, we would never expect this to happen as this would clearly demolish the profitability of a company and result in the cessation of these activities.
AI safety research is not done in this kind of environment at present whatsoever. However, it does seem to me that these kinds of institutions that carefully vet research and products, only releasing them when they remain beneficial, are possible.

LTM 4 Apr 2024 13:11 UTC
2 points
0
on: My PhD thesis: Algorithmic Bayesian Epistemology
Really fascinating stuff! I have a (possibly answered) question about how using expert updates on other expert prediction might be valuable.
You discuss the negative impacts of allowing experts to aggregate themselves, or viewing one another’s forecasts before initially submitting their own. Might there be value in allowing experts to submit multiple times, each time seeing the submitted predictions of a previous round? The final aggregation scheme would be able to not only assign a credence to each expert, but also gain a proxy for what credence the experts give to one another. In a more practical scenario where experts will talk if not collude, this might give better insight into how expert predictions are being created.
Thanks for taking the time to distill this work into a more approachable format—it certainly made the thesis more manageable!