Galathir

Karma: 19

Galathir 22 Mar 2026 9:33 UTC
1 point
0
on: Galathir’s Shortform
One last post from me for a bit:
Does anyone want to join me in exploring a simulation of the safety and explorative nature of many copies of the LLM with the same activation Vs many copies with different activations . Known good activations and points on the line between them too.

I think this might have important implications for how we design and deploy AI. I’m hoping to answer the questions.

- Do we want a few well tested models or an ecosystem of models/agents that rely on each other.
- What are the trade offs.

Galathir 21 Mar 2026 21:48 UTC
6 points
0
on: Galathir’s Shortform
Is there a taxonomy for cooperation anywhere?
Cooperating with copies is a trope. But has anyone written up about other types of cooperation that you expect to be evolutionarily stable. Like cooperation in some thing with things that cooperate with things like you on some things.
A specific example might be cooperation for robustness between different agents, that they’ll save you if times aren’t good for them and you’ll do the same. Because you do better in vastly different situations and you want to keep the other around as insurance.
Another example might be, if animals were rational, the shark taking steps to save a diverse set of fish so it has a healthy ecosystem to be the apex of.

Galathir 13 Mar 2026 14:23 UTC
7 points
0
on: Galathir’s Shortform
I’ve been trying to think of goals that could be aligned to, that don’t collapse into replicating the same thing a la smiley faces.
One thing I found is that certain goals are self limiting by their implications. Like Robustness. It aims for robustness to shocks and outside influence, but one of the core ways of doing that is to have lots of different things that react differently to shocks that support each other. So there is some desire for diversity built into robustness. Things designed by one entity for robustness might be less robust than lots of things designed by multiple entities with different goals. So a form of goal pluralism might be built in.

Does anyone know of any other interesting goals that have this kind of structure?

Galathir 8 Mar 2026 10:41 UTC
1 point
0
in reply to: Stephen Martin’s comment on: Stephen Martin’s Shortform
I could get behind this if instead of getting legal person hood , there was something between tool and person that they could be granted. Perhaps a grab bag of rights, things that recognise they are likely to have goals but otherwise might be completely alien

Galathir 23 Feb 2026 12:55 UTC
3 points
0
in reply to: Mitchell_Porter’s comment on: Galathir’s Shortform
This discussion inspired me to create an app to highlight the assumptions that lead me to care more about s-risk than dying from AI (I’m likely to experience partial singularities, but not ais that harvest my atoms)
https://singularities-we-survive.streamlit.app/
It is just is quickly whipped up by AI so I’ve not put much thought into the numbers myself. The shape of argument is what interested me.
If it was my day job to think about this stuff I could put more time into it .. it would be great if it was someone’s job to create things like this we could argue with and improve (or run our own versions that could be queried by people proposing policy interventions)
Not going to share it now widely as is.

Galathir 22 Feb 2026 23:22 UTC
1 point
0
in reply to: Mitchell_Porter’s comment on: Galathir’s Shortform
That depends on the person being an Occam’s razor type of person, not a decision maker under deep uncertainty or some other decision making system.
The dangerous thing occurs if a person or persons is blocked from normal collaborative development in ways that stretch normal explanation, but they can develop something by themselves that might be dangerous (due to special knowledge).
By stretch normal explanation I mean things like too much uniformity in the advice given to the blocked person gets (due to pause managing to dominate the info sphere, using unknown tactics ).
The paused people might not know what is going on, but that uncertainty allows explanations to breed, some of which might justify building super intelligences.

Galathir 22 Feb 2026 16:19 UTC
3 points
0
in reply to: Mitchell_Porter’s comment on: Galathir’s Shortform
I always think it’s worth starting as you mean to go on. The overton window might shift. The range of actors you might have to worry about might increase as we get more compute overhang, that kind of thing. We don’t know what the computational limits for making a superintelligence.

But yeah more of a black swan thing than something to easily plan for.

Galathir 22 Feb 2026 12:01 UTC
1 point
−1
on: Galathir’s Shortform
I think there is a risk that people will worry about alignment to a very successful pause/stop AI effort.
They will see it reducing diversity in thought (everyone agreeing with pause) and think there might already be a super intelligence causing a potential s-risk of trapping intelligence in one way of thinking. Trapped in a totalising system.
This might increase risk by making them think they need to do things like deploy their own super intelligence.
How do you design the pause or stop AI movement to avoid that failure mode?

Galathir 1 Feb 2026 10:53 UTC
1 point
0
in reply to: Kaarel’s comment on: kh’s Shortform
I’m imagining humanity fracturing into a million or billion different galaxies depending upon their exact level of desire for interacting with AI. I think the human value of the unity of humanity would be lost.
I think we need to buffer people from having to interact with AI if they don’t want to. But I value having other humans around. So some thing in between everyone living in their perfect isolation and everyone being dragged kicking and screaming into the future is where I think we should aim.

Galathir 6 Jan 2026 20:08 UTC
1 point
0
on: Galathir’s Shortform
In order to build trust and see if there exists information I don’t know I propose a test. A complementary tool to prediction markets.
Prediction markets are great for aggregating beliefs about known questions. But what about questions you don’t know to ask? What about detecting that someone has a frame you haven’t encountered? Here’s a privacy-preserving way to discover unknown unknowns without revealing what you know or learning what they know
People build AIs that represent the knowledge they have. They can be trained to not expose that knowledge, they would also be in environments that didn’t log the interactions they have with other AIs, what they would log is if the AI went out of activation distribution from it’s training set. This would imply some novel argument or possibility that you don’t think of.
You would have to be less sure of your ideas around certain subjects if there was something you were missing.
With mutual verification of the system before use. Both parties inspect the architecture. Neither can cheat without the other seeing.

Galathir 5 Jan 2026 13:38 UTC
−1 points
0
on: Galathir’s Shortform
The post AI period should have a fund that helps people that have been put in a bad position during the development of AI, but can’t talk about it due to info hazard reasons.
If the info hazard hasn’t passed this might have to be done illegibly, to avoid leaking the existence of people with info hazards

Galathir 26 Dec 2025 20:13 UTC
1 point
0
in reply to: Karl Krueger’s comment on: Whole Brain Emulation as an Anchor for AI Welfare
Should they get pain when their data centres they are housed in have issues

Galathir 22 Dec 2025 12:06 UTC
0 points
0
on: Galathir’s Shortform
I’m working on an AI powered tool to explore making decisions in complex fictional worlds , with the hope that it will translate into making better decisions when faced with real decisions.
Still in alpha stage give me a shout if you are curious and want a link.

Galathir 10 Dec 2025 13:11 UTC
4 points
−1
on: Galathir’s Shortform
A reason to act less than optimally: a fun thought experiment. To complexify anyone trying to predict you.
You might be in a hostile or at least not optimal simulation. There would be people trying to predict you and control you so the simulation is stable (for whatever reason they want the society you are in to persist).
If you act naively rationally you are predictable, your actions will be predicted. So the system as a whole will tend towards simplicity. This isn’t good because the simulation systems need to deal with a complex outer world too.
So be irrational in a way that is purposeful.
Make big bets you know you will lose (but stimulate other people to do interesting things). Get money pumped for a while to learn about those systems.
Maybe send messages by acting irrational on purpose. Bring life to the world.

Galathir 3 Dec 2025 18:07 UTC
1 point
0
in reply to: Galathir’s comment on: Galathir’s Shortform
AI might help with people generating tests for key results from okrs and publishing if they are not met.
If the key results are published this could help with AI pauses by validating that no stories on creating beyond frontier models have been written or started (assuming that that is a key result people care about).
I figured that objectives and key results are how companies maintain alignment and avoid internal arms races so might be useful for alignment between entities too (perhaps with government accredited badges for people that maintain objectives like pausing and responsible data use)

Galathir 2 Dec 2025 9:02 UTC
2 points
−1
on: Galathir’s Shortform
Is there anyone exploring how AI might be used to increase integrity and build trustworthiness.
For example it could scan the behaviour of people, businesses or AI and see whether it is consistent to stated promises, flagging things that are not.
It might be used to train LLMs to be consistent if they are too be used as agents

Galathir 30 Nov 2025 21:26 UTC
1 point
0
on: Galathir’s Shortform
Has anyone been writing evals on computer and network system administration for AI? It seems like this is something we would want to improve as it could increase the effort required to takeover the networks in an AI takeover scenario

Galathir 9 Oct 2025 9:13 UTC
1 point
0
on: Galathir’s Shortform
This is a letter I’m thinking about sending to my MP (hence the UK specific things). I would be interested in other peoples take on the problem.

The UK government’s creation of the AI Safety Institute as well its ambition to ‘turbocharge’ AI presents a challenge. To unlock the immense promise of AI without causing dangerous AI arms races, we need to establish international coordination, codes of conduct and cooperative frameworks appropriate for AI. This will require the UK leading on an international stage, having tried things out on a local level.
Why these safety measures around the development of AI have not been established already, is an open problem that I have not seen studied. My current hypothesis might be that discussion and policy around AI and the future is dominated by lesswrong style rationalism and neorealism. These philosophies suggest that building international mechanisms for cooperation on AI development is impossible and not worth trying and that things like codes of conduct for AI engineers are naive.

If there is a domination, it might be due to a founder effect as rationalism and philosophers like Nick Bostrom have been influential in the creation of the field of AI safety and AI based existential risk. They look at AI from an economically rational point of view. Rationalists also control the large online forum for discussing AI safety and policy, lesswrong and I’m not aware of others.
If there is a domination by cynical rationalism it is a problem for two reasons.
The first is that it has not been talked about or researched, so we don’t know the scope of the problem. The negative impact it has had on the AI safety progress or policy. It might have caused the lack of progress in UN or other international coordination on AI safety.

Secondly it can lead to problems if humans (and presumably AI) adopt the cynical view point. Historically it was this cynical philosophy that led to arms races in nuclear weapons and made the world less safe due to the security dilemma. The security dilemma is where states are trapped to make things less secure because there is supposed anarchy on the international level. Applying this to AI means a potentially dangerous AI arms race as there will not be time to build in the necessary safeguards into AI.

The lack of trust and lack of trust in the ability to build trust is corrosive to life and intelligence.
So what can be done? To start with this argument suggests researching the AI safety community and seeing how broad it is philosophically. This can then feed into the next step. If this study finds a narrow philosophical base, broadening the research directions might be the way to go. Perhaps by funding people doing traditional PhDs, or with research bodies like ARIA. They would then research codes of conduct for engineers and attempt to embed and test values and philosophies inside AI. The philosophies might Include care-based philosophies, deontology and virtue based.

So instead of just the field of AI alignment, where there is the hanging question of who to align the AI with, there could be movement towards trying different philosophies embedded in design and seeing which ones work best for humanity empirically. If actors see that each other are working towards non-cynically rational AI in a way that is not cynically rational, then the pressure to speed up the arms race becomes less.

Testing different philosophies in the real world could be achieved by doing user centred design of regulatory mandated tests for broadly deployed AI. The tests would have to be passed before the AI could be released. For example the AI agents could be simulated in an environment with humans and how the goals given to them interacted with simulated humans goals could be observed. Then the AI trained to pass this test would interact with a diverse set of humans to get feedback on how well the test makes the AI useful and non-harmful. This approach to test design could be tried out on a national or local level, before rolling out internationally.

The UK has shown an ambition to lead the way on AI safety and policy. This could be a way for the UK government to realise that ambition.

Galathir 3 Oct 2025 7:28 UTC
1 point
0
on: Galathir’s Shortform
Companies seem to be trapped in a security dilemma situation, they worry about being last to develop AGI so seem to be rushing towards the capabilities rather than safety. This in part is due to worries about owning/controlling the future.

Other aspects of humanity such as governments and the scientific community aren’t (at least visibly) rushing because they aren’t completely (economically) rational in that regard. Being more norm or rule following. Other ways not to be economically rational include caring for others (or humanity/nature in general)

We need to embed more rule-following into AI, so it doesn’t rush. This might need to be government mandated, as the rational companies might not be incentivised to rush. Government/International community mandated tests in simulated environments to make sure an AI follows the rules or cares about humanity, might be the way forward.

Caring and rule following seem different from corrigibility work or from the idea of alignment. A caring AI can have different goals from humanity but it would still allow/enchance humans ability to go about their business.

The rules I would look towards would definitely include never modifying the caring code of the AI.

Caring could be operationalised by using explainable AI to capture what parts of NN they thought were humans and embedding in the AIs goal system something that sought to increase or not modify the options the human could take.

Galathir 28 Sep 2025 21:56 UTC
1 point
0
on: Galathir’s Shortform
There is a reason why the arms race around more and bigger nuclear weapons stopped and hasn’t started again. Why do people think that is and can we use that understanding to stop the arms race (to be there first and control the future) around AI