Hi is there any update on this work in this area? I’ve had some thoughts about trying to make multipolar things work with agreed upon minimal floors (so humans would agree that all AI would have something like respect for life and other AI built in, to avoid some multipolar traps)
Would love to find people to talk to about this.
Thinking about recursive self improvement. It seems likely that people will do it soon, no matter what people worry about. Why hasn’t there been more effort to keep humans part of the loop that improves?
Making AIs that help optimise human skills , so they can continue to do alignment research and other things that we can’t or don’t want to automate.
Relatedly we should look to see if there could be training, AI or otherwise, against AI psychosis for people in those positions.
I’ve been thinking about recursive self improvement. But especially the likelihood that it will be important soon. Soon means with current LLMs that might just give up or delete the test set or otherwise detach from reality. Call the probability of that happening P. You can estimate P by looking at research tasks and how much they need to get helped along by humans to stay on task.
Goedel machines (that prove that the next step is better) try to use maths as an oracle. Which relies on the mathematical foundations being true. Other oracles I’ve theorised could help are oracles from the future that can say whether the change detaches the system from reality or not, a sort of go/no-go signal for potentially detaching changes.
Are there any other classes of oracles?
Edit: There might be some complex quantum computation that I’m not thinking of, due to classical computer bias.
Just a quick question: is anyone tracking or writing about the window we have where young people aren’t dependent on AI for their thinking and it’s impact on the future of alignment work
One last post from me for a bit:
Does anyone want to join me in exploring a simulation of the safety and explorative nature of many copies of the LLM with the same activation Vs many copies with different activations . Known good activations and points on the line between them too.
I think this might have important implications for how we design and deploy AI. I’m hoping to answer the questions.
- Do we want a few well tested models or an ecosystem of models/agents that rely on each other.
- What are the trade offs.
Is there a taxonomy for cooperation anywhere?
Cooperating with copies is a trope. But has anyone written up about other types of cooperation that you expect to be evolutionarily stable. Like cooperation in some thing with things that cooperate with things like you on some things.
A specific example might be cooperation for robustness between different agents, that they’ll save you if times aren’t good for them and you’ll do the same. Because you do better in vastly different situations and you want to keep the other around as insurance.
Another example might be, if animals were rational, the shark taking steps to save a diverse set of fish so it has a healthy ecosystem to be the apex of.
I’ve been trying to think of goals that could be aligned to, that don’t collapse into replicating the same thing a la smiley faces.
One thing I found is that certain goals are self limiting by their implications. Like Robustness. It aims for robustness to shocks and outside influence, but one of the core ways of doing that is to have lots of different things that react differently to shocks that support each other. So there is some desire for diversity built into robustness. Things designed by one entity for robustness might be less robust than lots of things designed by multiple entities with different goals. So a form of goal pluralism might be built in.
Does anyone know of any other interesting goals that have this kind of structure?
I could get behind this if instead of getting legal person hood , there was something between tool and person that they could be granted. Perhaps a grab bag of rights, things that recognise they are likely to have goals but otherwise might be completely alien
This discussion inspired me to create an app to highlight the assumptions that lead me to care more about s-risk than dying from AI (I’m likely to experience partial singularities, but not ais that harvest my atoms)
https://singularities-we-survive.streamlit.app/
It is just is quickly whipped up by AI so I’ve not put much thought into the numbers myself. The shape of argument is what interested me.
If it was my day job to think about this stuff I could put more time into it .. it would be great if it was someone’s job to create things like this we could argue with and improve (or run our own versions that could be queried by people proposing policy interventions)
Not going to share it now widely as is.
That depends on the person being an Occam’s razor type of person, not a decision maker under deep uncertainty or some other decision making system.
The dangerous thing occurs if a person or persons is blocked from normal collaborative development in ways that stretch normal explanation, but they can develop something by themselves that might be dangerous (due to special knowledge).
By stretch normal explanation I mean things like too much uniformity in the advice given to the blocked person gets (due to pause managing to dominate the info sphere, using unknown tactics ).
The paused people might not know what is going on, but that uncertainty allows explanations to breed, some of which might justify building super intelligences.
I always think it’s worth starting as you mean to go on. The overton window might shift. The range of actors you might have to worry about might increase as we get more compute overhang, that kind of thing. We don’t know what the computational limits for making a superintelligence.
But yeah more of a black swan thing than something to easily plan for.
I think there is a risk that people will worry about alignment to a very successful pause/stop AI effort.
They will see it reducing diversity in thought (everyone agreeing with pause) and think there might already be a super intelligence causing a potential s-risk of trapping intelligence in one way of thinking. Trapped in a totalising system.
This might increase risk by making them think they need to do things like deploy their own super intelligence.
How do you design the pause or stop AI movement to avoid that failure mode?
I’m imagining humanity fracturing into a million or billion different galaxies depending upon their exact level of desire for interacting with AI. I think the human value of the unity of humanity would be lost.
I think we need to buffer people from having to interact with AI if they don’t want to. But I value having other humans around. So some thing in between everyone living in their perfect isolation and everyone being dragged kicking and screaming into the future is where I think we should aim.
In order to build trust and see if there exists information I don’t know I propose a test. A complementary tool to prediction markets.
Prediction markets are great for aggregating beliefs about known questions. But what about questions you don’t know to ask? What about detecting that someone has a frame you haven’t encountered? Here’s a privacy-preserving way to discover unknown unknowns without revealing what you know or learning what they know
People build AIs that represent the knowledge they have. They can be trained to not expose that knowledge, they would also be in environments that didn’t log the interactions they have with other AIs, what they would log is if the AI went out of activation distribution from it’s training set. This would imply some novel argument or possibility that you don’t think of.
You would have to be less sure of your ideas around certain subjects if there was something you were missing.
With mutual verification of the system before use. Both parties inspect the architecture. Neither can cheat without the other seeing.
The post AI period should have a fund that helps people that have been put in a bad position during the development of AI, but can’t talk about it due to info hazard reasons.
If the info hazard hasn’t passed this might have to be done illegibly, to avoid leaking the existence of people with info hazards
Should they get pain when their data centres they are housed in have issues
I’m working on an AI powered tool to explore making decisions in complex fictional worlds , with the hope that it will translate into making better decisions when faced with real decisions.
Still in alpha stage give me a shout if you are curious and want a link.
A reason to act less than optimally: a fun thought experiment. To complexify anyone trying to predict you.
You might be in a hostile or at least not optimal simulation. There would be people trying to predict you and control you so the simulation is stable (for whatever reason they want the society you are in to persist).
If you act naively rationally you are predictable, your actions will be predicted. So the system as a whole will tend towards simplicity. This isn’t good because the simulation systems need to deal with a complex outer world too.
So be irrational in a way that is purposeful.
Make big bets you know you will lose (but stimulate other people to do interesting things). Get money pumped for a while to learn about those systems.
Maybe send messages by acting irrational on purpose. Bring life to the world.
AI might help with people generating tests for key results from okrs and publishing if they are not met.
If the key results are published this could help with AI pauses by validating that no stories on creating beyond frontier models have been written or started (assuming that that is a key result people care about).
I figured that objectives and key results are how companies maintain alignment and avoid internal arms races so might be useful for alignment between entities too (perhaps with government accredited badges for people that maintain objectives like pausing and responsible data use)
A more self consistent basis for rationality
Logically you can’t derive what you ought to do from what is.
This applies to being calibrated or caring about the survival of the human race.
A logically consistent rationalist is someone who happens to care about logical consistency and being calibrated where possible and logic dictates.
They state there axioms and don’t look down on people that are not logically consistent because that is is just a different way to be.