# Oleg Trott

Karma: 118

Columbia PhD, co-winner of the most well-funded ML competition ever, creator of the most cited molecular docking program: olegtrott.com

• many of which will allow for satisfaction, while still allowing the AI to kill everyone.

This post is just about alignment of AGI’s behavior with its creator’s intentions, which is what Yoshua Bengio was talking about.

If you wanted to constrain it further, you’d say that in the prompt. But I feel that rigid constraints are probably unhelpful, the way The Three Laws of Robotics are. For example, anyone could threaten suicide and force the AGI to do absolutely anything short of killing other people.

• Quoting from the CEV link:

The main problems with CEV include, firstly, the great difficulty of implementing such a program—“If one attempted to write an ordinary computer program using ordinary computer programming skills, the task would be a thousand lightyears beyond hopeless.” Secondly, the possibility that human values may not converge. Yudkowsky considered CEV obsolete almost immediately after its publication in 2004.

Neither problem seems relevant to what I’m proposing. My implementation is just a prompt. And there is no explicit optimization (after the LM has been trained).

Has anyone proposed exactly what I’m proposing? (slightly different wording is OK, of course)

• Technically true. But you could similarly argue that humans are just clumps of molecules following physical laws. Talking about human goals is a charitable interpretation.

And if you are in a charitable mood, you could interpret LMs as absorbing the explicit and tacit knowledge of millions of Internet authors. A superior ML algorithm would just be doing this better (and maybe it wouldn’t need lower-quality data).

# Align­ment: “Do what I would have wanted you to do”

12 Jul 2024 16:47 UTC
13 points
• A variation on this:

Any expression should be considered for replacement by a slightly bigger or smaller one. For example

z = f(x**2 * y)

should be replaceable by

z = f((x**2 - 1) * y)

The generated programs are quite short. So I would guess that this multiplies their number by 100-1000, if you consider one perturbation at a time.

• If you have locations that you want to perturb, then if you try a single off-by-one perturbation at a time, this adds programs. With two at a time, this adds programs.

There’s a possible optimization, where you only try this on tasks where no unperturbed program was found (<28%)

EDIT: Ironically, I made an off-by-one error, which Program Dithering would have fixed: This should be

# Fix sim­ple mis­takes in ARC-AGI, etc.

9 Jul 2024 17:46 UTC
9 points
• This looks similar, in spirit, to Large Language Models as General Pattern Machines:

BTW, there’s some discussion of this happening on the CCL mailing list (limited to professionals in relevant fields) if you are interested.

• Right. The benchmark (their test set) just compares 3D structures.

Side note: 52% also seems low for Vina, but I haven’t looked into this further. Maybe the benchmark is hard, or maybe the “search space” (user-specified) was too big.

On their other test (in their Extended Data), both Vina and AF3 do much better.

• Determining 3D structures is expensive.

The most realistic thing one could do is repeat this work, with the same settings, but using k-fold cross-validation, where test and training sets are never related (like what I did at Columbia).

This will show how well (or poorly, as the case may be) the method generalizes to unrelated proteins.

I hope someone does it.

• (ligand = drug-like molecule, for anyone else reading)

Right, I didn’t mean exact bitwise memory comparisons.

The dataset is redundant(ish), simply as an artifact of how it’s constructed:

For example, if people know that X binds A, and X ≈ Y, and A ≈ B, they’ll try to add X+B, Y+A and Y+B to the dataset also.

And this makes similarity-based predictions look artificially much more useful than they actually are, because in the “real world”, you will need to make predictions about dissimilar molecules from some collection.

I hope this makes sense.

# I’m a bit skep­ti­cal of AlphaFold 3

25 Jun 2024 0:04 UTC
84 points