ColeG

Karma: 7

ColeG 30 May 2026 10:35 UTC
1 point
0
on: Cognitive Security as an AI Safety Cause Area
I agree that Cognitive Security will be a major issue going forward, but I think it would be useful to decompose the problem into distinct branches. Information, attention, and moral/values manipulation are related, but not ultimately the same attack vector. They likely require different defensive strategies, and their relative threat levels may change differently over time as new capabilities emerge.

Here I argue that information manipulation may diminish as the dominant long-term attack vector as AI, data storage, sensors, and simulations improve, provided these capabilities remain broadly accessible and decentralized. Attention manipulation and moral/values manipulation seem more pernicious to me. We have limited time, attention, and compute for evaluating possible futures, while the relevant state space grows explosively with longer time horizons. And because morality remains an unsolved problem, there may be no simple analogue of “fact-checking” for these forms of influence.

Separate from governance and oversight, I think there may also be technical or infrastructural defenses worth considering. Information manipulation can, I think, be addressed largely by improving access to information and the tools for parsing it, while keeping these capabilities decentralized and broadly accessible. Attention manipulation and moral/values manipulation seem to require a different class of defenses (i.e., using AI against AI). This could include AI Debate or related methods that expose misleading reasoning and adversarial persuasion. Another direction would be to develop tools that can detect measurable persuasion or sycophancy cues and distinguish them from dialectical cues associated with non-manipulative truth-seeking.

ColeG 5 Apr 2026 11:33 UTC
2 points
0
in reply to: anaguma’s comment on: Academic Proof-of-Work in the Age of LLMs
I agree. “I worked really hard on it” is neither necessary nor sufficient for research quality. We already know that lots of careful-looking, labor-intensive, neatly written work can still be wrong or non-replicable. Meanwhile, some valuable insights emerge from relatively simple “aha” moments, and some deep ideas are developed more clearly outside the formal journal pipeline (ex: The Bitter Lesson).

Instead of reverting back to the old imperfect proof-of-work proxy for truth, we should try figuring out how to use these new AI tools to help assess research merit more efficiently.

Granted, some research work will require expensive experiments or other forms of “hard work”, in which case proof-of-work can still function as a useful initial filter.

ColeG 4 Apr 2026 19:43 UTC
1 point
−2
on: Democracy Dies With The Rifleman
I think it’s possible that extreme centralization could occur here (not necessarily so), but even if it does happen, it will likely not be long-term maintainable. A few familiar failure modes come to mind. 1.) Highly centralized systems suppress dissent leading to sycophancy and its associated delusional thinking. 2.) They also tend to hollow out the intellectual capital they need for innovation, adaptation, and maintenance. 3.) Maintaining rule requires delegation. The more it delegates to compensate for degraded performance (see 1 and 2) the greater the risk of creating powerful misaligned subordinates. 4.) Even if AI helps extend the life of a central committee, succession remains a problem. If the bus factor of key members is too high, the transition will be brittle and may fail outright. These failure modes are pretty timeless, and I don’t see how AI would prevent them still being applicable here. Maybe AI delays these feedback mechanisms to some degree, but my suspicion is that it will accelerate them. So, even if AI weakens the dependence on distributed coercive power, there are other mechanisms that will likely limit the extent and / or duration of centralization.

This isn’t to say great atrocities can’t occur during the “relatively short” lifetime of a centralized rule. The 20th century has provided a number of examples wherein short-lived authoritarian regimes have caused mass death and suffering. So all reasonable efforts should still be made to prevent the emergence of such a highly centralized regime.

ColeG 2 Apr 2026 19:00 UTC
4 points
0
on: Intelligence Dissolves Privacy
I actually wonder whether information asymmetries will even be sustainable in the long run, or whether we’ll eventually approach a world where access to information is relatively equalized (that will all depend on how centralized things become). If things do become fairly equalized, the issue may shift from information asymmetries and how to control them to attention asymmetries. I argued something like that here.

I think you hit on this some by shifting the argument from “what can be known?” to “what should be done with what is known?” But I think the next problem may be more than just good attention-allocation decisions. Rather, it may be more so establishing proper defenses against AI-powered attention manipulation, since that seems a likely next locus of control.