Hoagy

Karma: 902

Hoagy 6 Feb 2023 19:29 UTC
49 points
10
in reply to: vitaliya’s comment on: SolidGoldMagikarp (plus, prompt generation)
Good find! Just spelling out the actual source of the dataset contamination for others since the other comments weren’t clear to me:

r/counting is a subreddit in which people ‘count to infinity by 1s’, and the leaderboard for this shows the number of times they’ve ‘counted’ in this subreddit. These users have made 10s to 100s of thousands of reddit comments of just a number. See threads like this:

https://old.reddit.com/r/counting/comments/ghg79v/3723k_counting_thread/

They’d be perfect candidates for exclusion from training data. I wonder how they’d feel to know they posted enough inane comments to cause bugs in LLMs.

Hoagy 11 Jan 2023 13:50 UTC
35 points
11
on: We don’t trade with ants
Putting the entire failure to trade on the ability to communicate seems to understate the issue. Most if not all of the things listed that they ‘could’ do, are things which they could theoretically do with their physical capacities, but not with their cognitive abilities or ability to coordinate within themselves to accomplish a task.

In general, they aren’t able to act with the level of intentionality required to be helpful to us except in cases where those things we want are almost exactly the things they have evolved to do (like bees making honey, as mentioned in another comment).

The ‘failure to communicate’ is therefore in fact a failure to be able to think and act at the required level of flexibility and abstraction, and that seems more likely to carry over to our relations with some theoretical, super advanced AI or civilisation.

Hoagy 19 Apr 2022 22:21 UTC
LW: 34 AF: 13
0
AF
on: “Pivotal Act” Intentions: Negative Consequences and Fallacious Arguments
The synthesis of these options would be an AGI research group whose plan consists of:
- Develop safe AGI.
- Try to convince world governments to perform some such pivotal act (Idea A) - note that per current institutions this needs consensus and strong implementation across all major and medium tech powers.
- Have a back-up plan, if AGI research is proliferating without impending shutdown, to shut down world research unilaterally (Idea B).
What do you think of such a plan?
I think this would be reasonable, but if the plan is taken up then it becomes a cost-benefit analysis of when Idea B should be deployed, which plausibly could be very aggressive, so it could easily boil down to just Idea B.
It’s also worth noting that a research group with an AGI who want world governments to perform a pivotal act would need to be incredibly effective and persuasive. Their options would run a spectrum from normal public-channel and lobbying efforts to AGI-takes-over-the-world-behind-the-scenes (depending on sufficient capability), with a variety of AGI-assisted persuasion techniques in between. At some degree of AI/research group control over government, it’s not clear if this would be an improvement over the original act. Demonstrating the power of AGI in a way that would force governments to listen would need to at least threaten a transformative act (self-driving cars, solving protein folding, passing normal Turing tests clearly aren’t enough) and so the necessary levels of influence and demonstrated capability would be large (and demonstrating capability has obvious potential drawbacks in sparking arms races).

Hoagy 6 Jan 2024 0:07 UTC
LW: 33 AF: 19
2
AF
on: What’s up with LLMs representing XORs of arbitrary features?
My hypothesis about what’s going on here, apologies if it’s already ruled out, is that we should not think of it separately computing the XOR of A and B, but rather that features A and B are computed slightly differently when the other feature is off or on. In a high dimensional space, if the vector $(A | B)$ and the vector $(A | \neg B)$ are slightly different, then as long as this difference is systematic, this should be sufficient to successfully probe for $A \oplus B$ .
For example, if A and B each rely on a sizeable number of different attention heads to pull the information over, they will have some attention heads which participate in both of them, and they would ‘compete’ in the softmax, where if head C is used in both writing features A and B, it will contribute less to writing feature A if it is also being used to pull across feature B, and so the representation of A will be systematically different depending on the presence of B.
It’s harder to draw the exact picture for MLPs but I think similar interdependencies can occur there though I don’t have an exact picture of how, interested to discuss and can try and sketch it out if people are curious. Probably would be like, neurons will participate in both, neurons which participate in A and B will be more saturated if B is active than if B is not active, so the output representation of A will be somewhat dependent on B.
More generally, I expect the computation of features to be ‘good enough’ but still messy and somewhat dependent on which other features are present because this kludginess allows them to pack more computation into the same number of layers than if the features were computed totally independently.
What links here?
- Rohin Shah's comment on What’s up with LLMs representing XORs of arbitrary features? by Sam Marks (10 Jan 2024 9:58 UTC; 14 points)

Hoagy 20 Nov 2018 15:30 UTC
LW: 18 AF: 7
AF
on: Topological Fixed Point Exercises
Cleanest solution I can find for #8:
$f_{t} (x) = \frac{1}{1 + e^{10 (t - x)}}$
Also, if we have a proof for #6 there’s a pleasant method for #7 that should work in any dimension:
We take our closed convex set $S$ that has the bounded function $h : S \to S$ . We take a triangle $T$ that covers $S$ so that any point in $S$ is also in $T$ .
Now we define a new function $h^{'} : T \to T$ such that $h^{'} (x) = h (c_{s} (x))$ where $c_{s} (x)$ is the function that maps $x$ to the nearest point in $S$ .
By #6 we know that $h^{'}$ has a fixed point, since $c_{s}$ is continuous. We know that the fixed point of $h^{'}$ cannot lie outside $S$ because the range of $h^{'}$ is $S$ . This means $h^{'}$ has a fixed point within $S$ and since for $x \in S$ , $h (x) = h^{'} (x)$ , $h$ has a fixed point.
What links here?
- Logan Riggs's comment on Category Qualifications (w/ exercises) by Logan Riggs (15 Sep 2019 20:25 UTC; 1 point)

Hoagy 18 Nov 2018 18:20 UTC
LW: 16 AF: 6
AF
in reply to: Davidmanheim’s comment on: Topological Fixed Point Exercises
I was able to get at least (I think) close to proving 2 using Sperner’s Lemma as follows:
You can map the continuous function f(x) to a path of the kind found in Question 1 of length n+1
by evaluating f(x) at x=0, x=1 and n-1 equally spaced divisions between these two points and setting a node as blue if f(x) < 0 else as green.
By Sperner’s Lemma there is an odd, and therefore non-zero number of b-g vertices. You can then take any b-g pair of nodes as the starting points for a new path and repeat the process. After k iterations you have two values of x—only one where f(x) is below zero—that are 1/(n^k) away from each other. We thus can find arbitrarily close points that straddle zero. By taking the sequence f(x) of initial nodes x we get a sequence that, by B-W, has a sub-sequence which converges to zero. By continuity we have proved the existence of an x such that f(x)=0.
We can be sure that the sub-sequence does in fact converge to zero, rather than any other value because if it converges to any number |a|>0, the gradient of f(x) would have to be arbitrarily high to dip back below/above 0 for a value of x arbitrarily close by and therefore would not be a continuous function.
Comments to tighten up/poke holes in the above appreciated :)

Hoagy 23 Aug 2019 17:43 UTC
LW: 15 AF: 7
AF
on: Torture and Dust Specks and Joy—Oh my! or: Non-Archimedean Utility Functions as Pseudograded Vector Spaces
Apologies if this is not the discussion you wanted, but it’s hard to engage with comparability classes without a framework for how their boundaries are even minimally plausible.
Would you say that all types of discomfort are comparable with higher quantities of themselves? Is there always a marginally worse type of discomfort for any given negative experience? So long as both of these are true (and I struggle to deny them) then transitivity seems to connect the entire spectrum of negative experience. Do you think there is a way to remove the transitivity of comparability and still have a coherent system? This, to me, would be the core requirement for making dust specks and torture incomparable.

Hoagy 27 Jan 2023 16:26 UTC
13 points
6
in reply to: Chris_Leong’s comment on: Alexander and Yudkowsky on AGI goals
Yeah agreed, this doesn’t make sense to me.

There are probably just a few MB (wouldn’t be surprised if it could be compressed into much less) of information which sets up the brain wiring. Somewhere within that information are the structures/biases that, when exposed to the training data of being a human in our world, gives us our altruism (and much else). It’s a hard problem to understand these altruism-forming structures (which are not likely to be distinct things), replicate them in silica and make them robust even to large power differentials.

On the other hand, the human brain presumably has lots of wiring that pushes it towards selfishness and agenthood which we can hopefully just not replicate.

Either way, it seems that they could in theory be instantiated by the right process of trial and error—the question being whether the error (or misuse) gets us first.

Eliezer expects selfishness not to require any wiring once you select for a certain level of capability, meaning there’s no permanent buffer to be gained by not implementing selfishness. The margin for error in this model is thus small, and very hard for us to find without perfect understanding or some huge process.

I agree with this argument for some unknown threshold of capability but it seems strange to phrase it as impossibility unless you’re certain that the threshold is low, and even then it’s a big smuggled assumption.

EDIT: Looking back on this comment, I guess it comes down to the crux that for systems powerful enough to be relevant to alignment, by virtue of their power or research capability, must be doing strong enough optimisation on some function that we should model them as agents acting to further that goal.

Hoagy 30 Mar 2023 13:17 UTC
12 points
4
on: Nobody’s on the ball on AGI alignment

OpenAI would love to hire more alignment researchers, but there just aren’t many great researchers out there focusing on this problem.

This may well be true—but it’s hard to be a researcher focusing on this problem directly unless you have access to the ability to train near-cutting edge models. Otherwise you’re going to have to work on toy models, theory, or a totally different angle.

I’ve personally applied for the DeepMind scalable alignment team—they had a fixed, small available headcount which they filled with other people who I’m sure were better choices—but becoming a better fit for those roles is tricky, unless by just doing mostly unrelated research.

Do you have a list of ideas for research that you think is promising and possible without already being inside an org with big models?

Hoagy 24 May 2021 20:48 UTC
LW: 12 AF: 9
AF
in reply to: johnswentworth’s comment on: Testing The Natural Abstraction Hypothesis: Project Intro
Been a while but I thought the idea was interesting and had a go at implementing it. Houdini was too much for my laptop, let alone my programming skills, but I found a simple particle simulation in pygame which shows the basics, can see below.
exponents of the Jacobian of a 5 particle, 200 step simulation, with groups of 3 and 2 connected by springs
Planned next step is to work on the run-time speed (even this took a couple of minutes run, calculating the frame-to-frame Jacobian is a pain, probably more than necessary) and then add some utilities for creating larger, densely connected objects, will write up as a fuller post once done.
Curious if you’ve got any other uses for a set-up like this.

Hoagy 18 Nov 2018 18:49 UTC
LW: 12 AF: 4
AF
in reply to: Davidmanheim’s comment on: Topological Fixed Point Exercises
Here’s a messy way that at least doesn’t need too much exhaustive search:
First let’s separate all of the red nodes into groups so that within each group you can get to any other node in that group only passing through red nodes, but not to red nodes in any other group.
Now, we trace out the paths that surround these groups—they immediately look like the paths from Question 1 so this feels like a good start. More precisely, we draw out the paths such that each vertex forms one side of a triangle that has a blue node at its opposite corner. Note that you can have multiple paths stemming from the same group if the group touches the side of the larger triangle, or if it has internal holes.
Now we have this set of paths we can split them into three kinds. The first is loops, which arise when you have a group which never touches the edge of the larger triangle, or inside ‘holes’ in large groups. These can be seen as a path starting and finishing at the same node. They therefore have an even number of b-g vertices. The second kind is those that begin at the edge of the large triangle and end at the same edge. These paths begin and end on the same colour and therefore also have an even number of b-g vertices. Finally and most importantly there is a kind of path that goes from one edge to the other -in the case of the reds, the left edge to the right edge. This will happen once with the group that includes the top red node, and if any other group spans the larger triangle then it will generate two more of these paths. Sperner’s lemma tells us that these will have an odd number of b-g vertices and we know that there will be an odd number of such paths, so this final type generates an odd number of total b-g vertices.
By the way that we have defined these paths, the total number of r-g-b triangles is equal to the number of g-b vertices on the paths in the set generated above. This number is the sum of an odd number from the spanning paths and a series of even numbers from the other paths, giving an odd overall number of r-g-b vertices, proving number 4 (as long as I haven’t made an error in categorizing the paths).
I hope this makes sense, let me know if it doesn’t or has errors :)

Hoagy 27 Apr 2023 10:22 UTC
11 points
6
on: Contra Yudkowsky on Doom from Foom #2
The key thing I disagree with is:

In some sense the Foom already occurred—it was us. But it wasn’t the result of any new feature in the brain—our brains are just standard primate brains, scaled up a bit[14] and trained for longer. Human intelligence is the result of a complex one time meta-systems transition: brains networking together and organizing into families, tribes, nations, and civilizations through language. … That transition only happens once—there are not ever more and more levels of universality or linguistic programmability. AGI does not FOOM again in the same way.

Although I think agree the ‘meta-systems transition’ is a super important shift, which can lead us to overestimate the level of difference between us and previous apes, it also doesn’t seem like it was just a one time shift. We had fire, stone tools and probably language for literally millions of years before the Neolithic revolution. For the industrial revolution it seems that a few bits of cognitive technology (not even genes, just memes!) in renaissance Europe sent the world suddenly off on a whole new exponential.

The lesson, for me, is that the capability level of the meta-system/technology frontier is a very sensitive function of the kind of intelligences which are operating, and we therefore shouldn’t feel at all confident generalising out of distribution. Then, once we start to incorporate feedback loops from the technology frontier back into the underlying intelligences which are developing that technology, all modelling goes out the window.

From a technical modelling perspective, I understand that the Roodman model that you reference below (hard singularity at median 2047) has both hyperbolic growth and random shocks, and so even within that model, we shouldn’t be too surprised to see a sudden shift in gears and a much sooner singularity, even without accounting for RSI taking us somehow off-script.

Hoagy 31 Mar 2023 13:23 UTC
11 points
6
in reply to: ESRogs’s comment on: The 0.2 OOMs/year target
Moore’s law is a doubling every 2 years, while this proposes doubling every 18 months, so pretty much what you suggest (not sure if you were disagreeing tbh but seemed like you might be?)

Hoagy 20 Nov 2018 20:45 UTC
LW: 11 AF: 3
AF
in reply to: Chris_Leong’s comment on: Topological Fixed Point Exercises
I’ve realised that you’ve gotta be careful with this method because when you find a trichromatic subtriangle of the original, it won’t necessarily have the property of only having points of two colours along the edges, and so may not in fact contain a point that maps to the centre.
This isn’t a problem if we just increase the number n by which we divide the whole triangle instead of recursively dividing subtriangles. Unfortunately now we’re not reducing the range of co-ords where this fixed point must be, only finding a triad of arbitrarily close points that map to a triangle surrounding the centre. You can, for example, take the centre point of the first of these triangles (with some method of numbering to make the function definite) for each value of $n = 1, 2, 3..$ as a sequence in $R^{2}$ . This must have a convergent sequence which should converge to a point that maps to the centre but I can’t prove that last stage.

Hoagy 20 Nov 2018 15:21 UTC
LW: 11 AF: 3
AF
in reply to: riceissa’s comment on: Topological Fixed Point Exercises
Yeah agreed, in fact I don’t think you even need to continually bisect, you can just increase n indefinitely. Iterating becomes more dangerous as you move to higher dimensions because an n dimensional simplex with n+1 colours that has been coloured according to analogous rules doesn’t necessarily contain the point that maps to zero.
On the second point, yes I’d been assuming that a bounded function had a bounded gradient, which certainly isn’t true for say sin(x^2), the final step needs more work, I like the way you did it in the proof below.

Hoagy 25 Sep 2023 16:47 UTC
10 points
7
on: Amazon to invest up to $4 billion in Anthropic
Would be interested to hear from Anthropic leadership about how this is expected to interact with previous commitments about putting decision making power in the hands of their Long-Term Benefit Trust.

I get that they’re in some sense just another minority investor but a trillion-dollar company having Anthropic be a central plank in their AI strategy with a multi-billion investment and a load of levers to make things difficult for the company (via AWS) is a step up in the level of pressure to aggressively commercialise.

Hoagy 30 May 2021 11:14 UTC
LW: 10 AF: 7
AF
in reply to: johnswentworth’s comment on: Testing The Natural Abstraction Hypothesis: Project Intro
Another little update, speed issue solved for now by adding SymPy’s fortran wrappers to the derivative calculations—calculating the SVD isn’t (yet?) the bottleneck. Can now quickly get results from 1,000+ step simulations of 100s of particles.
Unfortunately, even for the pretty stable configuration below, the values are indeed exploding. I need to go back through the program and double check the logic but I don’t think it should be chaotic, if anything I would expect the values to hit zero.
It might be that there’s some kind of quasi-chaotic behaviour where the residual motion of the particles is impossibly sensitive to the initial conditions, even as the macro state is very stable, with a nicely defined derivative wrt initial conditions. Not yet sure how to deal with this.
wheels are the best object I’ve been able to make so far—they bounce against each other quite nicely. video at imgur.com/QxddkZK

Hoagy 28 Jun 2023 14:32 UTC
9 points
2
on: Aligning AI by optimizing for “wisdom”
On first glance I thought this was too abstract to be a useful plan but coming back to it I think this is promising as a form of automated training for an aligned agent, given that you have an agent that is excellent at evaluating small logic chains, along the lines of Constitutional AI or training for consistency. You have training loops using synthetic data which can train for all of these forms of consistency, probably implementable in an MVP with current systems.

The main unknown would be detecting when you feel confident enough in the alignment of its stated values to human values to start moving down the causal chain towards fitting actions to values, as this is clearly a strongly capabilities-enhancing process.

Perhaps you could at least get a measure by looking at comparisons which require multiple steps, of human value → value → belief etc, and then asking which is the bottleneck to coming to the conclusion that humans would want. Positing that the agent is capable of this might be assuming away a lot of the problem though.

Hoagy 4 Oct 2023 11:12 UTC
8 points
on: Hoagy’s Shortform
There’s an argument that I’ve been thinking about which I’d really like some feedback or pointers to literature on:

the tldr is that overcomplete bases necessitate linear representations
- Neural networks use overcomplete bases to represent concepts. Especially in vector spaces without non-linearity, such as the transformer’s residual stream, there are just many more things that are stored in there than there are dimensions, and as Johnson Lindenstrauss shows, there are exponentially many almost-orthogonal directions to store them in (of course, we can’t assume that they’re stored linearly as directions, but if they were then there’s lots of space). (see also Toy models of transformers, my sparse autoencoder posts)
- Many different concepts may be active at once, and the model’s ability to read a representation needs to be robust to this kind of interference.
- Highly non-linear information storage is going to be very fragile to interference because, by the definition of non-linearity, the model will respond differently to the input depending on the existing level of that feature. For example, if the response is quadratic or higher in the feature direction, then the impact of turning that feature on will be much different depending on whether certain not-quite orthogonal features are also on. If feature spaces are somehow curved then they will be similarly sensitive.
Of course linear representations will still be sensitive to this kind of interferences but I suspect there’s a mathematical proof for why linear features are the most robust to represent information in this kind of situation but I’m not sure where to look for existing work or how to start trying to prove it.

Hoagy 25 Sep 2023 22:29 UTC
8 points
1
in reply to: evhub’s comment on: Amazon to invest up to $4 billion in Anthropic
Cheers, I did see that and wondered whether still to post the comment but I do think that having a gigantic company owning a large chunk and presumably a lot of leverage over the company is a new form of pressure so it’d be reassuring to have some discussion of how to manage that relationship.