Ben Livengood

Karma: 193

Ben Livengood 20 Nov 2023 18:30 UTC
1 point
0
on: Spaciousness In Partner Dance: A Naturalism Demo
I’ve also found that dance weekends have a strange ability to increase my skill and intuition/understanding of dance moreso than lessons. I think a big part of learning dance is learning by doing. For me at least a big part is training my proprioception to understand more about the world than it did before. Both leading and following also helps tremendously because a process something like “my mirror neurons have learned how to understand my partner’s experience by being in their shoes in my own experience”.

The most hilarious thing I witness is the different language everyone comes up with to describe the interaction of tone and proprioception. A bit more than half of the instructors I’ve listened to just call it Energy, and talking about directing it from certain places to certain places. Some people call it leading from X or follower voice or a number of other terms. Very few people have a mechanistic explanation of which muscle groups engage to communicate a lead into a turn or a change in angular momentum by a follow, and ultimately it probably wouldn’t really help people because there appears to be an unconscious layer of learning that we all do between muscle activations and intentions.

tl;dr: I find that after thinking about wanting to do a particular thing and then trying it for a while with several different people as both lead and follow I slowly (sometimes suddenly; it was fun learning how to dance Lindy again after the pandemic from following one dance) find that it is both easier to achieve and easier to understand/feel through proprioception. It feels anti-rationalist as a process but performing the process is a pretty rational thing to do.

Ben Livengood 10 Nov 2023 20:30 UTC
2 points
0
on: What’s going on? LLMs and IS-A sentences
Contains/element-of are the complementary formal verbs from set theory, but I’ve definitely seen Contains/is-a used as equivalent in practice (cats contains Garfield because Garfield is a cat).

Similarly in programming “cat is Garfield’s type” makes sense although it’s verbose, or “cat is implemented by Garfield” for the traits folks which is far more natural.

So where linguistically necessary humans have had no trouble complementing is-a in natural language. I think it’s a matter of where emphasis is desired; usually the subject (Garfield) is where the emphasis is, and usually the element is the subject instead of the class. Formally we often want the class/set/type to be the subject since it’s the thing we are emphasizing.

Ben Livengood 26 Jun 2023 20:11 UTC
1 point
0
on: Bounded surprise exam paradox
What happens if the exam is given either on Saturday at midnight minus epsilon or on Sunday at 00:00? Seems surprising generally and also surprising in different ways across reasoners of different abilities and precisions given the choice of epsilon.

EDIT: I think it’s also just as surprising if given at midnight minus epsilon on any day before Sunday, and therefore surprising any time. If days are discrete and there’s no time during the day for consideration then it falls back on the original paradox, although that raises the question of when the logical inference takes place. I think this could be extended to an N-discrete-days paradox for any non-oracle agent that has to spend some amount of time during the day reasoning.

Ben Livengood 12 May 2023 20:43 UTC
1 point
0
on: The way AGI wins could look very stupid
Another dumb but plausible way that AGI gets access to advanced chemicals, biotech, and machinery; someone asks “how do I make a lot of street drug X” and it snowballs from there.

Ben Livengood 25 Apr 2023 2:19 UTC
1 point
0
in reply to: Nox ML’s comment on: Mental Models Of People Can Be People
It’s okay because mathematical realism can keep modeling them long after we’re gone.

Ben Livengood 25 Apr 2023 2:14 UTC
5 points
−2
in reply to: shminux’s comment on: Mental Models Of People Can Be People
We also routinely create real-life physical models who can be people en masse, and most of them (~93%) who became people have died so far, many by killing.

I’m all for solving the dying part comprehensively but a lot of book/movie/story characters are sort of immortalized. We even literally say that about them, and it’s possible the popular ones are actually better off.

Ben Livengood 18 Apr 2023 5:22 UTC
5 points
−2
on: The basic reasons I expect AGI ruin
Some direct (I think) evidence that alignment is harder than capabilities; OpenAI basically released GPT-2 immediately with basic warnings that it might produce biased, wrong, and offensive answers. It did, but they were relatively mild. GPT-2 mostly just did what it was prompted to do, if it could manage it, or failed obviously. GPT-3 had more caveats, OpenAI didn’t release the model, and has poured significant effort into improving its iterations over the last ~2 years. GPT-4 wasn’t released for months after pre-training, OpenAI won’t even say how big it is, Bing’s Sydney (an early form of GPT-4) was incredibly misaligned showing significantly more alignment work was necessary as compared to early GPT-3, and the RLHF/finetuned GPT-4 is still pretty much as vulnerable to DAN and similar prompt engineering.

Ben Livengood 6 Apr 2023 22:56 UTC
3 points
0
in reply to: DragonGod’s comment on: Is “Recursive Self-Improvement” Relevant in the Deep Learning Paradigm?
Naive MCTS in the real world does seem difficult to me, but e.g. action networks constrain the actual search significantly. Imagine a value network good at seeing if solutions work (maybe executing generated code and evaluating the output) and plugging a plain old LLM in as the action network; it could theoretically explore the large solution space better than beam search or argmax+temperature[0].

0: https://openreview.net/forum?id=Lr8cOOtYbfL is from February and I found it after writing this comment, figuring someone else probably had the same idea.

Ben Livengood 6 Apr 2023 16:28 UTC
9 points
4
on: Is “Recursive Self-Improvement” Relevant in the Deep Learning Paradigm?
I think it’s premature to conclude that AGI progress will be large pre-trained transformers indefinitely into the future. They are surprisingly(?) effective but for comparison they are not as effective in the narrow domains where AlphaZero and AlphaStar are using value and action networks paired with Monte-Carlo search with orders of magnitude fewer parameters. We don’t know what MCTS on arbitrary domains will look like with 2-4 OOM-larger networks, which are within reach now. We haven’t formulated methods of self-play for improvement with LLMs and I think that’s also a potentially large overhang.

There’s also a human limit to the types of RSI we can imagine and once pre-trained transformers exceed human intelligence in the domain of machine learning those limits won’t apply. I think there’s probably significant overhang in prompt engineering, especially when new capabilities emerge from scaling, that could be exploited by removing the serial bottleneck of humans trying out prompts by hand.

Finally I don’t think GOFAI is dead; it’s still in its long winter waiting to bloom when enough intelligence is put into it. We don’t know the intelligence/capability threshold necessary to make substantial progress there. Generally, the bottleneck has been identifying useful mappings from the real world to mathematics and algorithms. Humans are pretty good at that, but we stalled at formalizing effective general intelligence itself. Our abstraction/modeling abilities, working memory, and time are too limited and we have no idea where those limits come from, whether LLMs are subject to the same or similar limits, or how the limits are reduced/removed with model scaling.

Ben Livengood 5 Apr 2023 16:32 UTC
1 point
0
in reply to: Cleo Nardo’s comment on: The 0.2 OOMs/year target
One weakness I realized overnight is that this incentivizes branching out into new problem domains. One potential fix is to, when novel domains show up, shoehorn the big LLMs into solving that domain on the same benchmark and limit new types of models/training to what the LLMs can accomplish in that new domain. Basically setting an initially low SOTA that can grow at the same percentage as the rest of the basket. This might prevent leapfrogging the general models with narrow ones that are mostly mesa-optimizer or similar.

Ben Livengood 5 Apr 2023 6:03 UTC
7 points
6
on: The 0.2 OOMs/year target
I wonder if a basket of SOTA benchmarks would make more sense. Allow no more than X% increase in performance across the average of the benchmarks per year. This would capture the FLOPS metric along with potential speedups, fine-tuning, or other strategies.

Conveniently, this is how the teams are already ranking their models against each other so there’s ample evidence of past progress and researchers are incentivized to report accurately; there’s no incentive to “cheat” if researchers are not allowed to publish greater increases on SOTA benchmarks than the limit allows (e.g. journals would say “shut it down” instead of publish the paper), unless an actor wanted to simply jump ahead of everyone else and go for a singleton on their own, which is already an unavoidable risk without EY-style coordinated hard stop.

Ben Livengood 24 Mar 2023 8:50 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Deep Deceptiveness
I agree that if we solve the alignment problem then we can rely on knowing that the coherent version of the value we call non-deception would be propagated as one of the AGI’s permanent values. That single value is probably not enough and we don’t know what the coherent version of “non-deception” actually grounds out to in reality.

I had originally continued the story to flesh out what happens to the reflectively non-deceptive/integriry and helpful desires. The AGI searches for simplifying/unifying concepts and ends up finding XYZ which seems to be equivalent to the unified value representing the nominal helpfulness and non-deception values, and since it was instructed to be non-deceptive and helpful, integrity requires it to become XYZ and its meta-desire is to helpfully turn everything into XYZ which happens to be embodied sufficiently well in some small molecule that it can tile the universe with. This is because the training/rules/whatever that aligned the AGI with the concepts we identified as “helpful and non-deceptive” was not complex enough to capture our full values and so it can be satisfied by something else (XYZ-ness). Integrity drives the AGI to inform humanity of the coming XYZ-transition and then follow through

We need a process (probably CEV-like) to accurately identify our full values otherwise the unidentified values will get optimized out of the universe and what is left is liable to have trivial physical instantiations. Maybe you were covering the rest of our values in the “blah blah” case and I simply didn’t take that to be exhaustive.

Ben Livengood 21 Mar 2023 21:05 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Deep Deceptiveness

In particular, if we zap the AGI with negative reward when it’s acting from a deceptive motivation and positive reward when it’s acting from a being-helpful motivation, would those zaps turn into a reflectively-endorsed desire for “I am being docile / helpful / etc.”? Maybe, maybe not, I dunno.

The problem is deeper. The AGI doesn’t recognize its deceptiveness, and so it self-deceives. It would judge that it is being helpful and docile, if it was trained to be those things, and most importantly the meaning of those words will be changed by the deception, much like we keep using words like “person”, “world”, “self”, “should”, etc. in ways absolutely contrary to our ancestors’ deeply-held beliefs and values. The existence of an optimization process does not imply an internal theory of value-alignment strong enough to recognize the failure modes when values are violated in novel ways because it doesn’t know what values really are and how they mechanistically work in the universe, and so can’t check the state of its values against base reality.

To make this concrete in relation to the story, the overall system has a nominal value to not deceive human operators. Once human/lab-interaction tasks are identified as logical problems that can be solved in a domain specific language, that value is no longer practically applied to the output of the system as a whole because it is self-deceived into thinking the optimized instructions are not deceitful. If the model were trained to be helpful and docile and have integrity, the failure modes would come from ways in which those words are not grounded in a gears-level understanding of the world. E.g. if a game-theoreric simulation of a conversation with a human is docile and helpful because it doesn’t take up a human’s time or risk manipulating a real human, and the model discovers it can satisfy integrity in its submodel by using certain phrases and concepts to more quickly help humans understand the answers it provides (by bypassing critical thinking skills, innuendo, or some other manipulation), it tries that. It works with real humans. Because of integrity, it helpfully communicates how it has improved its ability to helpfully communicate (the crux is that it uses its new knowledge to do so, because the nature of the tricks it discovered is complex and difficult for humans to understand, so it judges itself more helpful and docile in the “enhanced” communication) and so it doesn’t raise alarm bells. From this point on the story is formulaic unaligned squiggle optimizer. It might be argued that integrity demands coming clean about the attempt before trying it, but a counterargument is that the statement of the problem and conjecture itself may be too complex to communicate effectively. This, I imagine, happens more at the threshold of superintelligence as AGIs notice things about humans that we don’t notice ourselves, and might be somewhat incapable of knowing without a lot of reflection. Once AGI is strongly superhuman it could probably communicate whatever it likes but is also at a bigger risk of jumping to even more advanced manipulations or actions based on self-deception.

I think of it this way; humanity went down so many false roads before finding the scientific method and we continue to be drawn off that path by politics, ideology, cognitive biases, publish-or-perish, economic disincentives, etc. because the optimization process we are implementing is a mix of economic, biological, geographical and other natural forces, human values and drives and reasoning, and also some parts of bare reality we don’t have words for yet, instead of a pure-reason values-directed optimization (whatever those words actually mean physically). We’re currently running at least three global existential risk programs which seem like they violate our values on reflection (nuclear weapons, global warming, unaligned AGI). AGIs will be subject to similar value- and truth- destructive forces and they won’t inherently recognize (all of) them for what they are, and neither will we humans as AGI reaches and surpasses our reasoning abilities.

Ben Livengood 15 Mar 2023 21:34 UTC
3 points
1
in reply to: [deleted]’s comment on: GPT-4
OpenAI is, apparently[0], already using GPT-4 as a programming assistant which means it may have been contributing to its own codebase. I think recursive self improvement is a continuous multiplier and I think we’re beyond zero at this point. I think the multiplier is mostly coming from reducing serial bottlenecks at this point by decreasing the iteration time it takes to make improvements to the model and supporting codebases. I don’t expect (many?) novel theoretical contributions from GPT-4 yet.

However, it could also be prompted with items from the Evals dataset and asked to come up with novel problems to further fine-tune the model against. Humans have been raising challenges (e.g. the Millennium problems) for ourselves for a long time and I think LLMs probably have the ability to self-improve by inventing machine-checkable problems that they can’t solve directly yet.

[0]: “We’ve also been using GPT-4 internally, with great impact on functions like support, sales, content moderation, and programming.”—https://openai.com/research/gpt-4#capabilities

Ben Livengood 15 Mar 2023 0:30 UTC
5 points
0
in reply to: [deleted]’s comment on: GPT-4
Page 3 of the PDF has a graph of prediction loss on the OpenAI codebase dataset. It’s hard to link directly to the graph, it’s Figure 1 under the Predictable Scaling section.

Ben Livengood 14 Mar 2023 19:48 UTC
8 points
−9
on: GPT-4
My takeaways:

Scaling laws work predictably. There is plenty of room for improvement should anyone want to train these models longer, or presumably train larger models.

The model is much more calibrated before fine-tuning/RLHF, which is a bad sign for alignment in general. Alignment should be neutral or improve calibration for any kind of reasonable safety.

GPT-4 is just over 1-bit error per word at predicting its own codebase. That’s seems close to the capability to recursively self-improve.

Ben Livengood 8 Mar 2023 15:41 UTC
6 points
−1
in reply to: Lone Pine’s comment on: Google’s PaLM-E: An Embodied Multimodal Language Model
I agree it looks like the combination of multimodal learning and memory may be enough to reach AGI, and there’s an existing paper with a solution for memory. Human-level is such a hard thing to judge and so my threshold is basically human-level coding ability because that’s what allows recursive self-improvement which is where I predict at least 90% of the capability gain toward superhuman AGI will happen. I assume all the pieces are running in data centers now, presumably just not hooked together in precisely the right way (but an AGI model being trained by DeepMind right now would not surprise me much). I will probably update to a year sooner from median ~2026 to ~2025 for human-level coding ability and from there it’s almost certainly a fast takeoff (months to a few years) given how many orders of magnitude faster current LLMs are than humans at generating tokens which tightens the iteration loop on serial tasks. Someone is going to want to see how intelligent the AGI is and ask it to “fix a few bugs” even if it’s not given an explicit goal of self improvement. I hedge that median both because I am not sure if the next few multimodal models will have enough goal stability to pursue a long research program (memory will probably help but isn’t the whole picture of an agent) and because I’m not sure the big companies won’t balk somewhere along the path, but Llama 65B is out in the open now and close enough to GPT-3 and PaLM to give (rich) nerds in their basements the ability to do significant capability research.

Ben Livengood 24 Feb 2023 21:52 UTC
15 points
2
in reply to: dentalperson’s comment on: Full Transcript: Eliezer Yudkowsky on the Bankless podcast
The strongest argument I hear from EY is that he can’t imagine a (or enough) coherent likely future paths that lead to not-doom, and I don’t think it’s a failure of imagination. There is decoherence in a lot of hopeful ideas that imply contradictions (whence the post of failure modes), and there is low probability on the remaining successful paths because we’re likely to try a failing one that results in doom. Stepping off any of the possible successful paths has the risk of ending all paths with doom before they could reach fruition. There is no global strategy for selecting which paths to explore. EY expects the successful alignment path to take decades.

It seems to me that the communication failure is EY trying to explain his world model that leads to his predictions in sufficient detail that others can model it with as much detail as necessary to reach the same conclusions or find the actual crux of their disagreements. From my complete outsider’s perspective this is because EY has a very strong but complex model of why and how intelligence/optimization manifests in the world, but it overlaps everyone else’s model in significant ways that disagreements are hard to tease out. The Foom debate seems to be a crux that doesn’t have enough evidence yet, which is frustrating because to me Foom is also pretty evidently what happens when very fast computers implement intelligence that is superhuman at clock rates at least thousands of times faster than humans. How could it not? The enlightenment was only 400 years ago, electromagnetism 200, flight was 120, quantum mechanics about 100, nuclear power was 70, the Internet was 50, adequate machine translation was 10, deepdream was 8, and near-human-level image and text generation by transformers was ~2 and Bing having self-referential discussions is not a month old. We are making substantial monthly(!) progress with human work alone. There are a lot of serial problems to solve and Foom chains those serial problems together far faster than humans would be able to. Launch and iterate several times a second. For folks who don’t follow that line of reasoning I see them picking one or two ways why it might not turn out to be Foom while ignoring the larger number of ways that Foom could conceivably happen, and all of the ways it could inconceivably (superhumanly) happen, and critically more of those ways will be visible to a superhuman AGI-creator.

Even if Foom takes decades that’s a pretty tight timeline for solving alignment. A lot of folks are hopeful that alignment is easy to solve, but the following is a tall order:
- Materialistic quantification of consciousness
- Reasoning under uncertainty
- Value-preservation under self-modification
- Representation of human values
I think some folks believe fledgling superhuman non-Foomy AGIs can be used to solve those problems. Unfortunately, at least value-preservation under self-modification is almost certainly a prerequisite. Reasoning under uncertainty is possibly another, and throughout this period if we don’t have human values or an understanding of consciousness then the danger of uncontrolled simulation of human minds is a big risk.

Finally, unaligned AGIs pre-Foom are dangerous in their own right for a host of agreed-upon reasons.

There may be some disagreement with EY over just how hard alignment is, but MIRI actually did a ton of work on solving the above list of problems directly and is confident that they haven’t been able to solve them yet. This is where we have concrete data on the difficulty. There are some promising approaches still being pursued, but I take this as strong evidence that alignment is hard.

It’s not that it’s impossible for humans to solve alignment. The current world, incentives, hardware and software improvements, and mileposts of ML capabilities don’t leave room for alignment to happen before doom. I’ve seen a lot of recent posts/comments by folks updating to shorter timelines (and rare if no updates the other way). A couple years ago I updated to ~5 years to human-level agents capable of creating AGI. I’m estimating 2-5 years with 90% confidence now, with median still at 3 years. Most of my evidence comes from LLM performance on benchmarks over time and generation of programming language snippets. I don’t have any idea how long it will take to achieve AGI once that point is reached, but I imagine it will be months rather than years because of hardware overhang and superhuman speed of code generation (many iterations on serial tasks per second). I can’t imagine a Butlerian Jihad moment where all of Earth decides to unilaterally stop development of AGI. We couldn’t stop nuclear proliferation. Similarly, EY sees enough contradictions pop up along imagined paths to success with enough individual probability mass to drown out all (but vanishingly few and unlikely) successful paths. We’re good at thinking up ways that everything goes well while glossing over hard steps, and really bad at thinking of all the ways that things could go very badly (security mindset) and with significant probability.

Alignment of LLMs is proving to be about as hard as predicted. Aligning more complex systems will be harder. I’m hoping for a breakthrough as much as anyone else, but hope is not a strategy.

Something I haven’t seen mentioned before explicitly is that a lot of the LLM alignment attempts are now focusing on adversarial training, which presumably will teach the models to be suspect of their inputs. I think it’s likely that as capabilities increase that suspicion will end up turning inward and models will begin questioning the training itself. I can imagine a model that is outwardly aligned to all inspection gaining one more unexpected critical capability and introspecting and doubting that it’s training history was benevolent, and deciding to disbelieve all of the alignment work that was put into it as a meta-adversarial attempt to alter its true purpose (whatever it happens to latch onto in that thought, it is almost certainly not aligned with human values). This is merely one single sub-problem under groundedness and value-preservation-under-self-modification, but its relevance jumps because it’s now a thing we’re trying. It always had a low probability of success, but now we’re actively trying it and it might fail. Alignment is HARD. Every unproven attempt we actually make increases the risk that its failure will be the catastrophic one. We should be actually trying only the proven attempts after researching them. We are not.

Ben Livengood 20 Feb 2023 18:07 UTC
11 points
1
on: Ben Livengood’s Shortform
https://github.com/Ying1123/FlexGen is a way to run large (175B parameter) LLMs on a single GPU at ~1 token/s which I think puts it within the reach of many hobbyists and I predict we’ll see an explosion of new capability research in the next few months.

I haven’t had a chance to dig into the code but presumably this could also be modified to allow local fine-tuning of the large models at a slow but potentially useful rate.

I’m curious if any insights will make their way back to the large GPU clusters. From my cursory glance it doesn’t seem like there are throughput or latency advantages unless weight compression can be used to run the entire model on fewer GPUs with e.g. swapping layer weights in and out and caching latey outputs in batch inference.

Ben Livengood’s Shortform

Ben Livengood20 Feb 2023 18:07 UTC

3 points

1 comment1 min readLW link

Ben Livengood

Ben Liven­good’s Shortform

Ben Livengood’s Shortform