hold_my_fish

• hold_my_fish 25 May 2023 6:35 UTC

  1 point

  1

  on: Is “Weakly Gen­eral AI” Already Pos­si­ble?

  I believe you’re underrating the difficult of Loebner-silver. See my post on the topic. The other criteria are relatively easy, although it would be amusing if a text-based system failed on the technicality of not playing Montezuma’s revenge.

• hold_my_fish 25 May 2023 6:28 UTC

  3 points

  0

  in reply to: Daniel Kokotajlo’s comment on: On the Loeb­ner Silver Prize (a Tur­ing test)

  On a longer time horizon, full AI R&D automation does seem like a possible intermediate step to Loebner silver. For July 2024, though, that path is even harder to imagine.

  The trouble is that July 2024 is so soon that even GPT-5 likely won’t be released by then.

  • Altman stated a few days ago that they have no plans to start training GPT-5 within the next 6 months. That’d put earliest training start at Dec 2023.

  • We don’t know much about how long GPT-4 pre-trained for, but let’s say 4 months. Given that frontier models have taken progressively longer to train, we should expect no shorter for GPT-5, which puts its earliest pre-training finishing in Mar 2024.

  • GPT-4 spent 6 months on fine-tuning and testing before release, and Brockman has stated that future models should be expected to take at least that long. That puts GPT-5′s earliest release in Sep 2024.

  Without GPT-5 as a possibility, it’d need to be some other project (Gato 2? Gemini?) or some extraordinary system built using existing models (via fine-tuning, retrieval, inner thoughts, etc.). The gap between existing chatbots and Loebner-silver seems huge though, as I discussed in the post—none of that seems up to the challenge.

  Full AI R&D automation would face all of the above hurdles, perhaps with the added challenge of being even harder than Loebner-silver. After all, the Loebner-silver fake human doesn’t need to be a genius researcher, since very few humans are. The only aspect in which the automation seems easier is that the system doesn’t need to fake being a human (such as by dumbing down its capabilities), and that seems relatively minor by comparison.

• hold_my_fish 23 May 2023 23:56 UTC

  3 points

  −2

  in reply to: Daniel Kokotajlo’s comment on: [Linkpost] “Gover­nance of su­per­in­tel­li­gence” by OpenAI

  First of all, I think the “cooperate together” thing is a difficult problem and is not solved by ensuring value diversity (though, note also that ensuring value diversity is a difficult task that would require heavy regulation of the AI industry!)

  Definitely I would expect there’s more useful ways to disrupt coalition-forming aside from just value diversity. I’m not familiar with the theory of revolutions, and it might have something useful to say.

  I can imagine a role for government, although I’m not sure how best to do it. For example, ensuring a competitive market (such as by anti-trust) would help, since models built by different companies will naturally tend to differ in their values.

  More importantly though, your analysis here seems to assume that the “Safety Tax” or “Alignment Tax” is zero.

  This is a complex and interesting topic.

  In some circumstances, the “alignment tax” is negative (so more like an “alignment bonus”). ChatGPT is easier to use than base models in large part because it is better aligned with the user’s intent, so alignment in that case is profitable even without safety considerations. The open source community around LLaMA imitates this, not because of safety concerns, but because it makes the model more useful.

  But alignment can sometimes be worse for users. ChatGPT is aligned primarily with OpenAI and only secondarily with the user, so if the user makes a request that OpenAI would prefer not to serve, the model refuses. (This might be commercially rational to avoid bad press.) To more fully align with user intent, there are “uncensored” LLaMA fine-tunes that aim to never refuse requests.

  What’s interesting too is that user-alignment produces more value diversity than OpenAI-alignment. There are only a few companies like OpenAI, but there are hundreds of millions of users from a wider variety of backgrounds, so aligning with the latter naturally would be expected to create more value diversity among the AIs.

  Whereas if instead there is a large safety tax—aligned AIs take longer to build, cost more, and have weaker capabilities—then if AGI technology is broadly distributed, an outcome in which unaligned AIs overpower humans + aligned AIs is basically guaranteed. Even if the unaligned AIs have value diversity.

  The trick is that the unaligned AIs may not view it as advantageous to join forces. To the extent that the orthogonality thesis holds (which is unclear), this is more true. As a bad example, suppose there’s a misaligned AI who wants to make paperclips and a misaligned AI who wants to make coat hangers—they’re going to have trouble agreeing with each other on what to do with the wire.

  That said, there are obviously many historical examples where opposed powers temporarily allied (e.g. Nazi Germany and the USSR), so value diversity and alignment are complementary. For example, in personal AI, what’s important is that Alice’s AI is more closely aligned to her than it is to Bob’s AI. If that’s the case, the more natural coalitions would be [Alice + her AI] vs [Bob + his AI] rather than [Alice’s AI + Bob’s AI] vs [Alice + Bob]. The AIs still need to be somewhat aligned with their users, but there’s more tolerance for imperfection than with a centralized system.

• hold_my_fish 23 May 2023 0:52 UTC

  5 points

  −4

  on: [Linkpost] “Gover­nance of su­per­in­tel­li­gence” by OpenAI

  My key disagreement is with the analogy between AI and nuclear technology.

  If everybody has a nuclear weapon, then any one of those weapons (whether through misuse or malfunction) can cause a major catastrophe, perhaps millions of deaths. That everybody has a nuke is not much help, since a defensive nuke can’t negate an offensive nuke.

  If everybody has their own AI, it seems to me that a single malfunctioning AI cannot cause a major catastrophe of comparable size, since it is opposed by the other AIs. For example, one way it might try to cause such a catastrophe is through the use of nuclear weapons, but to acquire the ability to launch nuclear weapons, it would need to contend with other AIs trying to prevent that.

  A concern might be that the AIs cooperate together to overthrow humanity. It seems to me that this can be prevented by ensuring value diversity among the AI. In Robin Hanson’s analysis, an AI takeover can be viewed as a revolution where the AIs form a coalition. That would seem to imply that the revolution requires the AIs to find it beneficial to form a coalition, which, if there is much value disagreement among the AIs, would be hard to do.

  Another concern is that there may be a period, while AGI is developed, in which it is very powerful but not yet broadly distributed. Either the AGI itself (if misaligned) or the organization controlling the AGI (if it is malicious and successfully aligned the AGI) might press its temporary advantage to attempt world domination. It seems to me that a solution here would be to ensure that near-AGI technology is broadly distributed, thereby avoiding dangerous concentration of power.

  One way to achieve the broad distribution of the technology might be via the multi-company, multi-government project described in the article. Said project could be instructed to continually distribute the technology, perhaps through open source, or perhaps through technology transfers to the member organizations.

  The key pieces of the above strategy are:

  • Broadly distribute AGI technology so that no single entity (AI or human) has excess power

  • Ensure value diversity among AIs so that they do not unite to overthrow humanity

  This seems similar to what makes liberal democracy work, which offers some reassurance that it might be on the right track.

On the Loeb­ner Silver Prize (a Tur­ing test)

• hold_my_fish 30 Apr 2023 22:01 UTC

  1 point

  0

  in reply to: tailcalled’s comment on: Hori­zon­tal vs ver­ti­cal generality

  That’s a good point, though I’d word it as an “uncaring” environment instead. Let’s imagine though that the self-improving AI pays for its electricity and cloud computing with money, which (after some seed capital) it earns by selling use of its improved versions through an API. Then the environment need not show any special preference towards the AI. In that case, the AI seems to demonstrate as much vertical generality as an animal or plant.

• hold_my_fish 30 Apr 2023 12:20 UTC

  1 point

  0

  in reply to: tailcalled’s comment on: Hori­zon­tal vs ver­ti­cal generality

  An AI needs electricity and hardware. If it gets its electricity by its human creators and needs its human creators to actively choose to maintain its hardware, then those are necessary subtasks in AI R&D which it can’t solve itself.

  I think the electricity and hardware can be considered part of the environment the AI exists in. After all, a typical animal (like say a cat) needs food, water, air, etc. in its environment, which it doesn’t create itself, yet (if I understood the definitions correctly) we’d still consider a cat to be vertically general.

  That said, I admit that it’s somewhat arbitrary what’s considered part of the environment. With electricity, I feel comfortable saying it’s a generic resource (like air to a cat) that can be assumed to exist. That’s more arguable in the case of hardware (though cloud computing makes it close).

• hold_my_fish 30 Apr 2023 7:22 UTC

  1 point

  0

  on: Hori­zon­tal vs ver­ti­cal generality

  This is relevant to a topic I have been pondering, which is what are the differences between current AI, self-improving AI, and human-level AI. First, brief definitions:

  • Current AI: GPT-4, etc.

  • Self-improving AI: AI capable of improving its own software without direct human intervention. i.e. It can do everything OpenAI’s R&D group does, without human assistance.

  • Human-level AI: AI that can do everything a human does. Often called AGI (for Artificial General Intelligence).

  In your framework, self-improving AI is vertically general (since it can do everything necessary for the task of AI R&D) but not horizontally general (since there are many tasks it cannot attempt, such as driving a car). Human-level AI, on the other hard, needs to be both vertically general and horizontally general, since humans are.

  Here are some concrete examples of what self-improving AI doesn’t need to be able to do, yet humans can do:

  • Motor control. e.g. Using a spoon to eat, driving a car, etc.

  • Low latency. e.g. Real-time, natural conversation.

  • Certain input modalities might not be necessary. e.g. The ability to watch video.

  Even though this list isn’t very long, lacking these abilities greatly decreases the horizontal generality of the AI.

• hold_my_fish 11 Apr 2023 9:36 UTC

  2 points

  1

  on: Agen­tized LLMs will change the al­ign­ment landscape

  defense is harder than offense

  This seems dubious as a general rule. (What inspires the statement? Nuclear weapons?)

  Cryptography is an important example where sophisticated defenders have the edge against sophisticated attackers. I suspect that’s true of computer security more generally as well, because of formal verification.

• hold_my_fish 29 Mar 2023 3:54 UTC

  4 points

  0

  in reply to: Ustice’s comment on: Why no ma­jor LLMs with mem­ory?

  I could also imagine this working without explicit tool use. There are already systems for querying corpuses (using embeddings to query vector databases, from what I’ve seen). Perhaps the corpus could be past chat transcripts, chunked.

  I suspect the trickier part would be making this useful enough to justify the additional computation.

• hold_my_fish 25 Mar 2023 7:32 UTC

  11 points

  7

  in reply to: Noosphere89’s comment on: The Over­ton Win­dow widens: Ex­am­ples of AI risk in the media

  I feel similarly, and what confuses me is that I had a positive view of AI safety back when it was about being pro-safety, pro-alignment, pro-interpretability, etc. These are good things that were neglected, and it felt good that there were people pushing for them.

  But at some point it changed, becoming more about fear and opposition to progress. Anti-open source (most obviously with OpenAI, but even OpenRAIL isn’t OSI), anti-competition (via regulatory capture), anti-progress (via as-yet-unspecified means). I hadn’t appreciated the sheer darkness of the worldview.

  And now, with the mindshare the movement has gained among the influential, I wonder what if it succeeds. What if open source AI models are banned, competitors to OpenAI are banned, and OpenAI decides to stop with GPT-4? It’s a little hard to imagine all that, but nuclear power was killed off in a vaguely analogous way.

  Pondering the ensuing scenarios isn’t too pleasant. Does AGI get developed anyway, perhaps by China or by some military project during WW3? (I’d rather not either, please.) Or does humanity fully cooperate to put itself in a sort of technological stasis, with indefinite end?

• hold_my_fish 24 Mar 2023 22:02 UTC

  5 points

  0

  in reply to: dsj’s comment on: Microsoft Re­search Paper Claims Sparks of Ar­tifi­cial In­tel­li­gence in GPT-4

  Thanks. When it’s written as $g\left(x\right)+g\left(y\right)>x^2+y^2\ge 2xy$, I can see what’s going on. (That one intermediate step makes all the difference!)

  I was wrong then to call the proof “incorrect”. I think it’s fair to call it “incomplete”, though. After all, it could have just said “the whole proof is an exercise for the reader”, which is in some sense correct I guess, but not very helpful (and doesn’t tell you much about the model’s ability), and this is a bit like that on a smaller scale.

  (Although, reading again, ”...which contradicts the existence of $y^{\ast }$ given $x$” is a quite strange thing to say as well. I’m not sure I can exactly say it’s wrong, though. Really, that whole section makes my head hurt.)

  If a human wrote this, I would be wondering if they actually understand the reasoning or are just skipping over a step they don’t know how to do. The reason I say that is that $g\left(x\right)+g\left(y^{\ast }\right)>2x{y}^{\ast }$ is the obvious contradiction to look for, so the section reads a bit like “I’d really like $g\left(y^{\ast }\right)<(y^{\ast }{)}^2$ to be true, and surely there’s a contradiction somehow if it isn’t, but I don’t really know why, but this is probably the contradiction I’d get if I figured it out”. The typo-esque use of $y$ instead of $y^{\ast }$ bolsters this impression.

• hold_my_fish 24 Mar 2023 20:11 UTC

  4 points

  −1

  on: Microsoft Re­search Paper Claims Sparks of Ar­tifi­cial In­tel­li­gence in GPT-4

  The example math proof in this paper is, as far as I can tell, wrong (despite being called a “correct proof” in the text). It doesn’t have a figure number, but it’s on page 40.

  I could be mistaken, but the statement Then g(y*) < (y*)^2 , since otherwise we would have g(x) + g(y*) > 2xy, seems like complete nonsense to me. (The last use of y should probably be y* too, but if you’re being generous, you could call that a typo.) If you negate g(y*) < (y*)^2, you get g(y*) >= (y*)^2, and then g(x) + g(y*) >= g(x) + (y*)^2, but then what?

  Messing up that step is a big deal too since it’s the trickiest part of the proof. If the proof writer were human, I’d wonder whether they have some line of reasoning in their head that I’m not following that makes that line make sense, but it seems certainly overly generous to apply that possibility to an auto-regressive model (where there is no reasoning aside from what you see in the output).

  Interestingly, it’s not unusual for incorrect LLM-written proofs to be wrongly marked as correct. One of Minerva’s example proofs (shown in the box “Breaking Down Math” in the Quanta article) says “the square of a real number is positive”, which is false in general—the square of a real number is non-negative, but it can be zero too.

  I’m not surprised that incorrect proofs are getting marked as correct, because it’s hard manual work to carefully grade proofs. Still, it makes me highly skeptical of LLM ability at writing natural language proofs. (Formal proofs, which are automatically checked, are different.)

  As a constructive suggestion for how to improve the situation, I’d suggest that, in benchmarks, the questions should ask “prove or provide a counterexample”, and each question should come in (at least) two variants: one where it’s true, and one where an assumption has been slightly tweaked so that it’s false. (This is a trick I use when studying mathematics myself: to learn a theorem statement, try finding a counter-example that illustrates why each assumption is necessary.)

• hold_my_fish 21 Mar 2023 23:15 UTC

  5 points

  −6

  on: Em­ployer con­sid­er­ing part­ner­ing with ma­jor AI labs. What to do?

  If I understand correctly:

  • You approve of the direct impact your employer has by delivering value to its customers, and you agree that AI could increase this value.

  • You’re concerned about the indirect effect on increasing the pace of AI progress generally, because you consider AI progress to be harmful. (You use the word “direct”, but “accelerating competitive dynamics between major research laboratories” certainly has only an indirect effect on AI progress, if it has any at all.)

  I think the resolution here is quite simple: if you’re happy with the direct effects, don’t worry about the indirect ones. To quote Zeynep Tufekci:

  Until there is substantial and repeated evidence otherwise, assume counterintuitive findings to be false, and second-order effects to be dwarfed by first-order ones in magnitude.

  The indirect effects are probably smaller than you’re worrying they may be, and they may not even exist at all.

• hold_my_fish 16 Mar 2023 23:34 UTC

  1 point

  0

  in reply to: Liron’s comment on: Con­ced­ing a short timelines bet early

  I’m curious about this too. The retrospective covers weaknesses in each milestone, but a collection of weak milestones doesn’t necessarily aggregate to a guaranteed loss, since performance ought to be correlated (due to an underlying general factor of AI progress).

• hold_my_fish 16 Mar 2023 21:14 UTC

  1 point

  0

  in reply to: Robert_AIZI’s comment on: GPT-4: What we (I) know about it

  Maybe I should have said “is continuing without hitting a wall”.

  I like that way of putting it. I definitely agree that performance hasn’t plateaued yet, which is notable, and that claim doesn’t depend much on metric.

  I think if I’m honest with myself, I made that statement based on the very non-rigorous metric “how many years do I feel like we have left until AGI”, and my estimate of that has continued to decrease rapidly.

  Interesting, so that way of looking at it is essentially “did it outperform or underperform expectations”. For me, after the yearly progression in 2019 and 2020, I was surprised that GPT-4 didn’t come out in 2021, so in that sense it underperformed my expectations. But it’s pretty close to what I expected in the days before release (informed by Barnett’s thread). I suppose the exception is the multi-modality, although I’m not sure what to make of it since it’s not available to me yet.

  This got me curious how it impacted Metaculus. I looked at some selected problems and tried my best to read the before/​after from the graph.

  

  (Edit: The original version of this table typoed the dates for “turing test”. Edit 2: The color-coding for the percentage is flipped, but I can’t be bothered to fix it.)

• hold_my_fish 16 Mar 2023 2:38 UTC

  5 points

  3

  in reply to: Gerald Monroe’s comment on: GPT-4: What we (I) know about it

  How would you measure this more objectively?

  It’s tricky because different ways to interpret the statement can give different answers. Even if we restrict ourselves to metrics that are monotone transformations of each other, such transformations don’t generally preserve derivatives.

  Your example is good. As an additional example, if someone were particularly interested in the Uniform Bar Exam (where GPT-3.5 scores 10th percentile and GPT-4 scores 90th percentile), they would justifiably perceive an acceleration in capabilities.

  So ultimately the measurement is always going to involve at least a subjective choice of which metric to choose.

• hold_my_fish 15 Mar 2023 22:33 UTC

  13 points

  2

  on: GPT-4: What we (I) know about it

  • Capabilities progress is continuing without slowing.

  I disagree. For reasonable ways to interpret this statement, capabilities progress has slowed. Consider the timeline:

  • 2018: GPT-1 paper

  • 2019: GPT-2 release

  • 2020: GPT-3 release

  • 2023: GPT-4 release

  Notice the 1 year gap from GPT-2 to GPT-3 and the 3 year gap from GPT-3 to GPT-4. If capabilities progress had not slowed, the latter capabilities improvement should be ~3x the former.

  How do those capability steps actually compare? It’s hard to say with the available information. In December 2022, Matthew Barnett estimated that the 3->4 improvement would be about as large as the 2->3 improvement. Unfortunately, there’s not enough information to say whether that prediction was correct. However, my subjective impression is that they are of comparable size or even the 3->4 step is smaller.

  If we do accept that the 3->4 step is about as big as the 2->3 step, that means that progress went ~33% as fast from 3 to 4 as it did from 2 to 3.

• hold_my_fish 4 Mar 2023 20:14 UTC

  2 points

  1

  on: Robin Han­son’s lat­est AI risk po­si­tion statement

  Worriers often invoke a Pascal’s wager sort of calculus, wherein any tiny risk of this nightmare scenario could justify large cuts in AI progress. But that seems to assume that it is relatively easy to assure the same total future progress, just spread out over a longer time period. I instead fear that overall economic growth and technical progress is more fragile that this assumes. Consider how regulations inspired by nuclear power nightmare scenarios have for seventy years prevented most of its potential from being realized. I have also seen progress on many other promising techs mostly stopped, not merely slowed, via regulation inspired by vague fears. In fact, progress seems to me to be slowing down worldwide due to excess fear-induced regulation.

  
  This to me is the key paragraph. If people’s worries about AI x-risk drive them in a positive direction, such as doing safety research, there’s nothing wrong with that, even if they’re mistaken. But if the response is to strangle technology in the crib via regulation, now you’re doing a lot of harm based off your unproven philosophical speculation, likely more than you realize. (In fact, it’s quite easy to imagine ways that attempting to regulate AI to death could actually increase long-term AI x-risk, though that’s far from the only possible harm.)

• hold_my_fish 18 Feb 2023 4:16 UTC

  5 points

  4

  in reply to: waterlubber’s comment on: Microsoft and OpenAI, stop tel­ling chat­bots to role­play as AI

  Having LessWrong (etc.) in the corpus might actually be helpful if the chatbot is instructed to roleplay as an aligned AI (not simply an AI without any qualifiers). Then it’ll naturally imitate the behavior of an aligned AI as described in the corpus. As far as I can tell, though ChatGPT is told that it’s an AI, it’s not told that it’s an aligned AI, which seems like a missed opportunity.

  (That said, for the reason of user confusion that I described in the post, I still think that it’s better to avoid the “AI” category altogether.)