I’m interested in doing in-depth dialogues to find cruxes. Message me if you are interested in doing this.
I do alignment research, mostly stuff that is vaguely agent foundations. Currently doing independent alignment research on ontology identification. Formerly on Vivek’s team at MIRI.
Relevant comment on reddit from someone working on Leela Odds:
Why would models start out aligned by default?
This is the best I’ve got so far. I estimated the rating using the midpoint of a logistic regression fit to the games. The first few especially seem to have been inflated due to not having enough high rated players in the data, so it had to extrapolate. And they all seem inflated by (I’d guess) a couple of hundred points due to the effects I mentioned in the post. (Edit: Please don’t share the graph alone without this context).
The NN rating in the Blitz data highlights the flaw in this method of estimating the rating.
I haven’t found a way to get similar data on human vs human games.
Took a while to download all this. I’m curious what your blitz rating is?
Does that sound right?
Can’t give a confident yes because I’m pretty confused about this topic, and I’m pretty unhappy currently with the way the leverage prior mixes up action and epistemics. The issue about discounting theories of physics if they imply high leverage seems really bad? I don’t understand whether the UDASSA thing fixes this. But yes.
That avoids the “how do we encode numbers” question that naturally raises itself.
I’m not sure how natural the encoding question is, there’s probably an AIT answer to this kind of question that I don’t know.
By “control plausibly works” I didn’t mean “Stuff like existing monitoring will work to control AIs forever”. I meant it works if it is a stepping stone allows us to accelerate/finish alignment research, and thereby build aligned AGI.
I think several of the subquestions that matter for whether it’ll plausibly work to have AI solve alignment for us are in the second category. Like the two points I mentioned in the post. I think there are other subquestions that are more in the first category, which are also relevant to the odds of success. I’m relatively low confidence about this kind of stuff because of all the normal reasons why it’s difficult to say how other people should be thinking. It’s easy to miss relevant priors, evidence, etc. But still… given what I know about what everyone believes, it looks like these questions should be resolvable among reasonable people.
Makes sense, but in that case, why penalize by time? Why not just directly penalize by utility? Like the leverage prior.
Also, why not allow floating point representations of utility to be output? Rather than just binary integers?
Aren’t there programs that run fast and also return a number that grows much faster than |p|? Like up arrow notation. Why don’t these grow faster than your speed prior penalizes them?
I think there are reasonable people who look at the evidence and think it plausible that control works, and also reasonable people who look at the evidence and think it implausible that control works. And others who think that openai-superalignment-style plans plausibly work.
Something is going wrong here.
Nice! Yeah after Leela is down to a couple of pieces the game is usually over. But I’ve had to learn not to be lazy at this point because a couple of times it managed to force a stalemate.
Why have you read chess books?
our current thread is an adjustment to the error measure.
We’re not sure that this is necessary. I quite like the current form of the errors. I’ve spent much of the past week searching for counterexamples to the ∃ deterministic latent theorem and I haven’t found anything yet (although it’s partially a manual search). My current approach takes a P(X_1,X_2) distribution, finds a minimal stochastic NL, then finds a minimal deterministic NL. The deterministic error has always been within a factor of 2 of the stochastic error. So currently we’re expecting the theorem can be rescued.
rather than
That seems like a cool idea for the mediation condition, but Isn’t it trivial for the redundancy conditions?
That’ll be the difference between max and sum in the denominator. If you use sum it’s 3.39.
Here’s one we worked out last night, where the ratio goes to infinity.
but thought it was just numerical error
I was totally convinced it was a numerical error. I spent a full day trying to trace it in my numpy code before I started to reconsider. At that point we’d worked through the proof carefully and felt confident of every step. But we needed to work out what was going on because we wanted empirical support for a tighter bound before we tried to improve the proof.
Oh nice, we tried to wrangle that counterexample into a simple expression but didn’t get there. So that rules out a looser bound under these assumptions, that’s good to know.
Alfred Harwood and I were working through this as part of a Dovetail project and unfortunately I think we’ve found a mistake. The Taylor expansion in Step 2 has the 3rd order term . This term should disappear as goes to zero, but this is only true if stays constant. The transformation in Part 1 reduces (most terms of) and at the same rate, so decreases at the same rate as . So the 2nd order approximation isn’t valid.
For example, we could consider two binary random variables with probability distributions
and and and .
If , then as .
But consider the third order term for which is
This is a constant term which does not vanish as .
We found a counterexample to the whole theorem (which is what led to us finding this mistake), which has , and it can be found in this colab. There are some stronger counterexamples at the bottom as well. We used sympy because we were getting occasional floating point errors with numpy.
Sorry to bring bad news! We’re going to keep working on this over the next 7 weeks, so hopefully we’ll find a way to prove a looser bound. Please let us know if you find one before us!
Nice, agreed. This is basically why I don’t see any hope in trying to align super-LLMs. (this and several similar categories of plausible failures that don’t seem avoidable without dramatically more understanding of the algorithms running on the inside).
Okay. I think this anthropic theory makes a falsifiable prediction (in principle). The infinite precision real numbers could be algorithmically simple, or they could be unstructured. The theory predicts that they are not algorithmically simple. If it were the case that they were algorithmically simple, we could run a solomonoff inductor on the macrostates and it would recover the full microstates (and this would probably be simpler than the abstraction-based compression).
By the way, there seems to be an issue where sympy silently drops precision under some circumstances. Definitely a bug. A couple of times it’s caused non-trivial errors in my KLs. It’s pretty rare, but I don’t know any way to completely avoid it. Thinking of switching to a different library.