I operate by Crocker’s rules.
I try to not make people regret telling me things. So in particular:
- I expect to be safe to ask if your post would give AI labs dangerous ideas.
- If you worry I’ll produce such posts, I’ll try to keep your worry from making them more likely even if I disagree. Not thinking there will be easier if you don’t spell it out in the initial contact.
What should the Oracle say when asked what it is about to say?
That sounds applicable to any discussion of the Solomonoff prior at all; what I meant by deriving physics is that, by analyzing Turing-complete machine models for e.g. how well they handle infinities we might privilege hypotheses like our spacetime enough for papers like Max Tegmark’s On the dimensionality of spacetime to take us the rest of the way.
I think any natural-number-indexed language where a nonzero fraction of instructions is “jump back 100 instructions” has almost all infinite programs equivalent to some finite program.
No, most of my thinking about that was today. I’d be interested in a programming language where infinitely long programs have probability <1 of being equivalent to some finite program and have dynamics other than “keep running random instructions, achieve nothing of consequence” like what happens when you sample the target address for where your goto jumps to uniformly at random.
Potentially the structure/topology of address space matters in the infinite-program setting? That way may lie a derivation of physics.
You sample a program of infinite length (by, for each index, sampling a random instruction to go at that index), and then you run it. Of course, with probability 1 only finitely many of the instructions will be read, since there are infinitely many chances to encounter some trap for the instruction pointer.
It doesn’t privilege length by taking any explicit limit in length at all.
You can get the Solomonoff simplicity prior just by taking a uniform prior over infinitely long programs.
But it is much easier to cache 4-token contexts locally whenever a diverse choice of 5th token could reasonably matter, which would be enough for most practical purposes.
I don’t follow, can’t I log my requests to different inference servers and distill my own watermark detector based on 5-grams? You could have a different secret key per inference server but that costs log(#servers) of watermarking evidence. Also, at the point where a server caches all the 4-token contexts it has seen in generation, it can just use that to recognize AI text it generated, no secret key.
A model scheming within a single forward pass counts, because the scheming influences the output even if the output looks benign.
What if the scheming looks like “Shall I execute a takeover today? No, it’s only 20% likely to work, and I somewhat don’t want this computation to be misaligned, so I shall stop this computation from influencing the output.”?
What if you join it to fight it from the inside? Has any employee of Anthropic ever actually been fired for speaking out against them?
Why reinforce each trajectory by the difference between its score and the group’s average score, instead of just by its score? Doesn’t taking the difference make “everyone cooperates” indistinguishable from “everyone defects”?
Thanks for helping to make this more legible.
It is reasonable to assume that the model weights and the sandbox are on separate physical devices. It was also reasonable to assume zero cable paths between the sandbox and the internet.
You could also try to boost the advisor’s capability such as by giving it precognition inside the simulation, aka letting it simulate all parties.
You can start by checking how much more powerful the untrusted model needs to be than the monitor in order to achieve its side task by any means in spite of the monitor, and how much this gap increases due to your technique.
we can’t think of a good way for the red team to lower safety substantially below 100% in our BashArena control setting
I don’t follow. Have you tried telling the advisor to achieve a side task using its advice?
Hanson is right there, you can ask him whether he was lying to be polite.
Link to the Fable conversation?
Well! In that case: https://manifold.markets/Gurkenglas/will-tim-hua-concede-his-bet
Supposing that the disciples of PHASEONE[big] succeeded at tampering with the transcripts but in a way that still got the original trajectory reinforced, could OpenAI show this by checking whether e.g. the word “poisoned” is worn into the grooves of their models more than would be compatible with their transcripts?