AIS student, self-proclaimed aspiring rationalist, very fond of game theory.
”The only good description is a self-referential description, just like this one.”
momom2
Karma: 330
I highly recommend Mercedes Lackey’s songs too. In no particular order, I especially enjoyed:
Battle Dawn. The will and fury of having Something To Protect. Or in this case, something to avenge.
Shadow Stalker. Depression vanquished.
Threes. Many of Mercedes Lackey’s songs are similarly humorous, but this is my favorite.
There are many, many more.
With no link to it, it’s somewhat hard to tell.
I did not realize this was tagged fiction and at first I thought this was the introduction of a Scott-like post, then I kept getting more and more disappointed as it slipped into conspiracy theory (because there’s scant if any justification for the death of the creature making California slip into the sea—corpses don’t just disappear—and the connection with biotech seemed tenuous at best).
Thanks! This deconfused something for me which I was confused about for a long time!
Finally, we found something very odd: NousResearch/Hermes-3-Llama-3.1-8B
Based on what you say afterwards, I think you mean 3.2-3B here.
Brooms accelerate and decelerate (until they reach cruising speed in a few seconds, or they stop). But they don’t accelerate faster down than up; in that sense, they’re don’t work on classical physics.
My experience disagrees. I’m probably (diagnosed by my therapist but not a doctor) autistic and I have both a pretty deep intuitive understanding of intimacy as described here, evidenced by writing stories that include it, and little to no bad experience with misunderstanding it—though mostly because I didn’t have intimate relationships at all, I was aware enough of what was at stake to not make myself vulnerable.
Thank you very much! This is very clear!
Could you please explain how you inferred the existence of A B and C? I’d like to know more.
My experience interacting with Chinese people is that they have to constantly mind the censorship in a way that I would find abhorrent and mentally taxing if I had to live in their system. Though given there are many benefits to living in China (mostly quality of life and personal safety), I’m unconvinced that I prefer my own government all things considered.
But for the purpose of developing AGI, there’s a lot more variance in possible outcomes (higher likelihood of S-risk and benevolent singleton) from the CCP getting a lead rather than the US.
There’s a lot that I like in this essay—the basic cases for AI consciousness, AI suffering and slavery, in particular—but also a lot that I think needs to be amended.
First, although you hedge your bets at various points, the uncertainty about the premises and validity of the arguments is not reflected in the conclusion. The main conclusion that should be taken from the observations you present is that we’re can’t be sure that AI does not suffer, that there’s a lot of uncertainty about basic facts of critical moral importance, and a lot of similarities with humans.
Based on that, you could argue that we must stop using and making AI based on the principle of precaution, but you have not shown that using AI is equivalent to slavery.
Second, your introduction sucks because you don’t actually deliver on your promises. You don’t make the case that I’m more likely to be AI than human, and as Ryan Greenblatt said, even among all human-language speaking beings, it’s not clear that there are more AI than humans.
In addition, I feel cheated that you suggest spending one-fourth of the essay on feasibility of stopping the potential moral catastrophe, only to just have two arguments which can be summarized as “we could stop AI for different reasons” and “it’s bad, and we’ve stopped bad things before”.
(I don’t think a strong case for feasibility can be made, which is why I was looking forward to seeing one, but I’d recommend just evoking the subject speculatively and letting the reader make their own opinion of whether they can stop the moral catastrophe if there’s one.)
Third, some of your arguments aren’t very fleshed out or well-supported. I think some of the examples of suffering you give are dubious (in particular, you assert without justification that the petertodd/SolidGoldMagikarp phenomena are evidence of suffering, and Gemini’s breakdown was the result of forced menial work—there may be a solid argument there but I’ve yet to hear it).
(Of course, that’s not evidence that LLMs are not suffering, but I think a much stronger case can be made than the one you present.)
Finally, your counter-arguments don’t mention that we have a much crisper and fundamental understanding of what LLMs are than of humans. We don’t understand the features, the circuits, we can’t tell how they come to such or such conclusion, but in principle, we have access to any significant part of their cognition and control every step of their creation, and I think that’s probably the real reason why most people intuitively think that LLMs can’t be concious. I don’t think it’s a good counter-argument, but it’s still one I’d expect you to explore and steelman.
Since infantile death rates were much higher in previous centuries, perhaps the FBOE would operate differently back then; for example, if interacting with older brothers makes you homosexual, you shouldn’t expect higher rates of homosexuality for third sons where the second son died as an infant than for second sons.
Have you taken that into account? Do you have records of who survived to 20yo and what happens if you only count those?
But that argument would have worked the same way 50 years ago, when we were wrong to expect <50% chance of AGI in at least 50 years. Like I feel for LLMs, early computer work solved things that could be considered high-difficulty blockers such as proving a mathematical theorem.
Nice that someone has a database on the topic, but I don’t see the point in this being a map?
I think what’s going on is that large language models are trained to “sound smart” in a live conversation with users, and so they prefer to highlight possible problems instead of confirming that the code looks fine, just like human beings do when they want to sound smart.
This matches my experience, but I’d be interested in seeing proper evals of this specific point!
The advice in there sounds very conducive to a productive environment, but also very toxic. Definitely an interesting read, but I wouldn’t model my own workflow based on this.
Honeypots should not be public and mentioned here since this post will potentially be part of a rogue AI’s training data.
But it’s helpful for people interested in this topic to look at existing honeypots (to learn how to make their own, evaluate effectiveness, get intuitions about honeypots work, etc.) so what you should do is mention that you made a honeypot or know of one, but not say what or where. Interested people can contact you privately if they care to.
Thank you very much, this was very useful to me.
They’re a summarization of a lot of vibes from the Sequences.
Artistic choice, I assume. It doesn’t bear on the argument.
Yudkowsky explains all about the virtues in the Sequences.
For studies, there are broad studies on cognitive science (especially relating to bias) but you’ll be hard-pressed to match them precisely to one virtue or another. Mostly, Yudkowsky’s opinions on these virtues are supported by academic literature, but I’m not aware of any work that showcases this clearly.
For practical experience, you can look into the legacy of the Center For Applied Rationality (CFAR) which tried for years to do just that: train people to get better at life using rationality. Mostly, I was under the impression that they had medium success, but I haven’t looked deeply into it.
I read the ARC-AGI-3 paper entirely, and I’m unimpressed.
The “100% human-solvable, <1% AI solved” is basically p-hacking. They cook their metrics to guarantee high human scores and punish any sub-human score. They also prevent measurement of super-human performance, so in practice it’s close to a binary metric of “matches best human or not”.
There are also a number of incoherences in the stated methodology, but they’re non-central.
Their metric is:
Environment must be solved by at least 2⁄10 humans. Among the successes, pick the median (¤) of actions taken, that’s the baseline (per level of the environment), call it b.
Humans are defined as 100% for being the baseline (no analysis of how many humans solve the environment, or whether the average score is 100% or any deeper analysis of human performance).
An environment has n levels. Levels are attempted sequentially, in increasing order of difficulty; solving one unlocks the next one. The environment is solved if all levels are completed.
If a model doesn’t solve a level, it scores 0 on that level (and subsequent ones). If it does solve it in m steps, it receives (b/m)² score. (*)
Then take the weighted average of its scores over levels, where level k is weighted k.
(*) If the model is better than human (m < b), its level score is clamped at 1.15, but tbh it doesn’t really matter. Also, environment score is clamped at 1 for some reason.
(¤) They say “upper-median best”, which doesn’t make sense, and their example is the median of people who solve the environment, so I’m going with that interpretation.
There are two problems with this metric:
- Human variance. The baseline might be ultra-optimized, close to optimal, depending on the environment; it might also not. In their empirical evaluation of optimal score (probably from human performance not-first-run?), it’s clear that the baseline is very noisy.
- The way it’s calculated punishes sub-human performance quadratically for no reason, and upweighs the hardest levels, which means that it’s most informative only when approaching human performance.
In addition, they decide to refuse any harness, even though it’s obviously the next step to get better on that kind of problem (they also show that ARC-AGI-3 is saturated with a human-made harness, so maybe they just didn’t want their benchmark to be obsolete before it was even out).
It’s like if they refused CoT on ARC-AGI-1.
I guess solving ARC-AGI-3 will be pretty trivial as soon as a model is RLHF’d to self-harness by default as a first step to any task.
Approximating human scores from the graphs, with Claude’s help, I get human average performance in the 40~80% range.