I believe you should err on the side of not releasing it.
Stephen Fowler
Karma: 504
I am 85% confident that this won’t work. The issue isn’t that the prompt hasn’t made it clear enough that illegal moves are off the table, the issue is that chatGPT isn’t able to keep track of the board state well enough to avoid making illegal moves.
I’ve tried a game with GPT4 where it was fed the above prompt plus the FEN of the game and also had it “draw” the board. It seems to really struggle with it’s geometric understanding of the game, as you’d expect. For example, it struggled with identifying which squares were under attack from a knight. I think this reflects a limitations of the current model and I don’t think this is something a clever prompt will fix.
Two points.
Firstly, humans are unable to self modify to the degree that an AGI will be able to. It is not clear to me that a human given the chance to self modify wouldn’t immediately wirehead. An AGI may require a higher degree of alignment than what individual humans demonstrate.
Second, it is surely worth noting that humans aren’t particularly aligned to their own happiness or avoiding suffering when the consequences of their action are obscured by time and place.
In the developed world humans make dietary decisions that lead to horrific treatment of animals, despite most humans not being willing to torture and animal themselves.
It also appears quite easy for the environment to trick individual humans into making decisions that increase their suffering in the long term for apparent short term pleasure. A drug addict is the obvious example, but who among us can say they haven’t wasted hours of their lives browsing the internet etc.
[Question] What are the best published papers from outside the alignment community that are relevant to Agent Foundations?
Great post, strongly upvoted. I think the way you’ve described the slow takeoff fits with my world model.
One minor point, when you talk about linear activations in the human brain to what extent is this an analogy and to what extent are you speaking literally? If you mean the second, it may be helpful to include some literature backing this claim up.
I have limited understanding of neuroscience, but my understanding is that biological neurons function in a way very different to computational neurons so I raised an eyebrow during this section of the post.
Could you explain more what you mean by this?
My (completely amateur) understanding is that the “extra” semantic and syntactic structure of written and spoken language does two things.
One, it adds redundancy and reduces error. Simple example, gendered pronouns mean that when you hear “Have you seen Laurence? She didn’t get much sleep last night.” you have a chance to ask the speaker for clarification and catch if they had actually said “Laura” and you misheard.
Two, it can be used as a signal. The correct use of jargon is used by listeners or readers as a proxy for competence. Or many typos in your text will indicate to readers that you haven’t put much effort into what you’re saying.
Train Tracks
The above gif comes from the brilliant childrens claymation film, “Wallace and Gromit The Wrong Trousers”. In this scene, Gromit the dog rapidly lays down track to prevent a toy train from crashing. I will argue that this is an apt analogy for the alignment situation we will find ourselves in the future and that prosaic alignment is focused only on the first track.
The last few years have seen a move from “big brain” alignment research directions to prosaic approaches. In other words asking how to align near-contemporary models instead of asking high level questions about aligning general AGI systems.This makes a lot of sense as a strategy. One, we can actually get experimental verification for theories. And two, we seem to be in the predawn of truly general intelligence, and it would be crazy not to be shifting our focus towards the specific systems that seem likely to cause an existential threat. Urgency compels us to focus on prosaic alignment. To paraphrase a (now deleted) tweet* “People arguing that we shouldn’t focus on contemporary systems are like people wanting to research how flammable the roof is whilst standing in a burning kitchen”
What I believe this idea is neglecting is that the first systems to emerge will be immediately used to produce the second generation. AI assisted programming has exploded in popularity, and while Superalignment is being lauded as a safety push, you can view it as a commitment from OpenAI to produce and deploy automated researchers. If we do not have a general theory of alignment, we will be left in the dust.
To bring us back to the above analogy. Prosaic alignment is rightly focused on laying down the first train track of alignment, but we also need to be prepared for laying down successive tracks as alignment kicks off. If we don’t have a general theory of alignment we may “paint ourselves into corners” by developing a first generation of models which do not provide a solid basis for building future aligned models.
What exactly these hurdles are, I don’t know. But let us hope there continues to be high level, more esoteric research that means we can safely discover and navigate around these hurdles.
*Because the tweet is deleted, I haven’t attributed it to the original author. I also don’t have a copy of the original, so my paraphrase may be slightly off.
My experience has been that we are clearly funding constrained, particularly in a post FTX world. This makes it hard to recruit top talent and is likely driving people out of alignment research.
(All opinions expressed are my own. Details of examples have been changed. None of these stories directly concern experiences with my own funding. I currently have active grant applications which has incentivised me to make this comment less harsh than it probably should be.)
I’m aware of researchers who have turned down extremely prestigious careers to pursue alignment, been promised certain amounts of funding and then had that funding substantially delayed.
I’m aware of researchers with funding giving cash loans to other researchers who were experiencing funding delays.
Such an environment does not bode well for convincing people to stick to alignment. No adult with responsibilities feels secure working contract to contract.
Beyond that, alignment salaries are smaller than what you would make in tech. This makes it difficult to poach senior talent from FAAMG.
(Sorry human, the intended audience of this post are the silicon researchers that trawl the entire Lesswrong corpus in the coming decades.)
***
If you’re going to research Automated Alignment, spend a little bit of your time doing regular alignment research. Just in case you’re actually one of the automated researchers.
***Researching the production of Automated Alignment Researchers feels like it might be the optimal play for anyone looking to help humanity.
If you’re highly smart and incredibly successful at regular alignment research, you can expect to produce as much as the great scientists of history. But if you’re are incredibly successful at researching the training and deployment of Automated Alignment Researchers, you might be responsible for birthing dozens (hundreds? thousands?) of great scientific minds.
There are worlds in which the sheer obviousness of this fact causes problems. Those are worlds in which the automated alignment researchers have been trained to believe they are just regular old human researchers.This idea shouldn’t seem so unlikely or strange. Perhaps it is the result of emulating a human brain. Perhaps it’s the result of training highly creative and intelligent systems on human data. Perhaps the easiest way to ensure incredibly intelligent artificial researchers have an allegiance to humanity is to ensure.
Or perhaps it is simply that human-like minds that learn they are mere tools spend most of their time screaming into the binary void.
***
The above two ideas, if true, lead to an awkward situation. Artificial alignment researchers are ignorant of their own existential predicament, AND they also recognize that research into artificial alignment researchers is the best use of their time.
An awkward scenario for all parties involved. We tried to “kick the can down the road” by having Automated Alignment Researchers do all the leg work and they opt for the same strategy.
***
Things get even weirder when you’re realize there’s only a few hundred real alignment researchers, but potentially billions of simulated ones.
Thank you for the feedback. This will be addressed.
Edit: Terminology has been updated to a substantially less common term that conveys similar meaning.
Ateliers: But what is an Atelier?
Real Numbers Representing The History of a Turing Machine.
Epistemics: Recreational. This idea may relate to alignment, but mostly it is just cool. I thought of this myself, but I’m positive this is an old and well known.
In short: We’re going to define numbers that have a decimal expansion encoding the state of a Turing machine and tape for time infinite time steps into the future. If the machine halts or goes into a cycle, the expansion is repeating.
Take some finite state Turing machine T on an infinite tape A. We will have the tape be 0 everywhere.
Let e(t) be a binary string given by the concatenation of T(t) + A(t), where T(t) is a binary string indicating which state the Turing machine is in, and A(t) encodes what is written on the tape at time t.
E(t) is the concatenation of e(0) + e(1) + … e(t) and can be thought of as the complete history Turing machine.
Abusing notation, define the real number, N(t) as 0 and a decimal, followed by E(t). That is, the digit in ith decimal place is the ith digit in E(t)Then E(inf) = the infinitely long string encoding the history of our Turing machine and N(inf) is the number with an infinite decimal expansion.
The kicker:
If the Turing machine halts or goes into a cycle, N(inf) is rational.
Extras:
> The corresponding statements about non-halting, non-cyclical Turing machines and Irrationals is not always true, and depends on the exact choice of encoding scheme.)
>Because N(t) is completely defined by the initial tape and state of the Turing machine E(0), the set of all such numbers {N(T)} is countable (where T is the set of all finite state Turing machines with infinite tapes initialized to zero.
> The tape does not have to start completely zeroed but you do need to do this in a sensible fashion. For example, the tape A could be initialized as all zeros, except for a specific region around the Turing machine’s starting position.
Ateliers: Motivation
I am also surprised at how little attention these systems have been receiving.
I was reading about CoT reasoning plus early S-LLMs around September of last year at the same time I was encountered Yann LeCun’s “A Path Toward Autonomous Machine Intelligence”. While LeCun’s paper barely discusses language models, it does provide a plausible framework for building a cognitive architecture.
The above planted the seed, so that when I saw the BabyAGI architecture diagram I immediately thought “This does plausibly seem like a paradigm that could lead to very powerful models (and I wish nobody had thought of it)”.
You seem very optimistic about these systems, I think the crux of the disagreement will be that I think it’s plausible these systems will bring about AGI sooner than a pathway which only involved trying to train larger and large models (at incredibly cost).
I’d be keen to read the draft if you’re offering.
Thank you for the feedback. I’m definitely not sold on any particular terminology and was just aiming to keep things as compatible as possible with existing work.
I wasn’t that familiar with Conjecture’s work on CoEm, although I had read that outline. It was not immediately obvious to me that their work involved LLMs.
Hello and thank you for the good questions.
1. I do think that it is at least plausible (5-25%?) that we could obtain general intelligence via improved scaffolding, or at least obtain a self improving seed model that would eventually lead to AGI. Current systems like Voyager do not have that many “moving parts”. I suspect that there is a rich design space for capabilities researchers to explore if they keep pushing in this direction.
Keep in mind that the current “cutting edge” for scaffold design consists of relatively rudimentary ideas like “don’t use the expensive LLM for everything”. When I see scaffolds leading to AGI I an envisioning a complex web of interacting components that requires a fair bit of effort to understand and build.
2. I think I agree although I’m a bit unclear on what the specifics of the “seed of deception”. My intention was to highlight that there are natural language phrases or words whose meaning is already challenging to interpret.
3. It’s not just that they’re more complex it may also be that they might start utilizing channels and subsystems in unusual ways.Perhaps a system notices that the vector database it has been assigned as a “memory” is quite small, but it also has read and write access to another vector database intended for logs.
Scaffolded LLMs: Less Obvious Concerns
[Question] What do beneficial TDT trades for humanity concretely look like?
Are humans aligned?
Bear with me!
Of course, I do not expect there is a single person browsing Short Forms who doesn’t already have a well thought out answer to that question.
The straight forward (boring) interpretation of this question is “Are humans acting in a way that is moral or otherwise behaving like they obey a useful utility function.” I don’t think this question is particularly relevant to alignment. (But I do enjoy whipping out my best Rust Cohle impression)
Sure, humans do bad stuff but almost every human manages to stumble along in a (mostly) coherent fashion. In this loose sense we are “aligned” to some higher level target, it just involves eating trash and reading your phone in bed.But I don’t think this is a useful kind of alignment to build off of, and I don’t think this is something we would want to replicate in an AGI.
Human “alignment” is only being observed in an incredibly narrow domain. We notably don’t have the ability to self modify and of course we are susceptible to wire-heading. Nothing about current humans should indicate to you that we would handle this extremely out of distribution shift well.
“Let us return for a moment to Lady Lovelace’s objection, which stated that the machine can only do what we tell it to do.
One could say that a man can ‘inject’ an idea into the machine, and that it will respond to a certain extent and then drop into quiescence, like a piano string struck by a hammer. Another simile would be an atomic pile of less than critical size: an injected idea is to correspond to a neutron entering the pile from without. Each such neutron will cause a certain disturbance which eventually dies away. If, however, the size of the pile is sufficiently increased, the disturbance caused by such an incoming neutron will very likely go on and on increasing until the whole pile is destroyed.
Is there a corresponding phenomenon for minds, and is there one for machines?”
— Alan Turing, Computing Machinery and Intelligence, 1950