David Schneider-Joseph
dsj
Karma: 925
Totally appreciate you guys have a lot on your plate right now! At least two of the examples I mentioned in my top-level comment didn’t seem like misrepresentations to me (though two of them did), which makes me confused as to how much of this post is accurate if it really is saying that all of the quoted examples are misrepresentations (hence the reason for my comment requesting that clarification). I haven’t carefully dug into every claim myself but overall my sense is that this post very heavily blurs the line between vehement disagreement and actual misrepresentation such that I don’t feel there’s a clear takeaway for me about how much of each is going on in Séb’s critique.
Thanks for this reply. I do agree that many people who are skeptical of various AI safety proposals don’t really believe in AGI or ASI, but I know that’s not everyone. For example, I am skeptical of AI 2040’s proposals but I very much believe in AGI and ASI, and that’s at the heart of some of the reasons for my concerns. I don’t know if that’s true in Séb’s case: is there a particular thing he wrote which makes you think he does not, or are you speculating/generalizing from experience with others?
Regarding the misrepresentations vs. disagreements, what I would really appreciate is understanding which are which in this post. I count 16 quotes from Séb listed under “False Representations”, but some of them seem more like disagreements to me. It would be helpful if you could clarify which you believe are actual misrepresentations and which you believe are disagreements.
Some of these do seem like misrepresentations. For example, about your beliefs regarding to whom the economic benefits of AI will accrue, and also about the desirability of having a central planner.
Others seem mainly like disagreements about what the likely effect of proposed policies would be and/or qualitative judgments about those policies.
For example, it’s possible that mandating transparency on AI labs gives more room for policymakers to iterate, but it’s also a very big upfront change. It’s plausible someone could consider that change to bake in too much upfront, without misrepresenting the plan.
It’s also possible that the existing powers the government has are comparable to or greater than the various powers you propose, so that it’s no big deal to give them more powers. Others might disagree. This also seems more like an area of policy disagreement rather than misrepresentation.
I might have misunderstood which are misrepresentations and which are disagreements. Either way, I would be interested in some clarity as to which of these matters you vehemently disagree with Séb on and which you feel are misrepresentations.
This isn’t because the president can’t pass legislation on his own, so without the support of Congress he’s a lame duck even without removal.
I think you mean it is because of that, not that it isn’t? But let me know if I’ve misunderstood you. I agree so far as legislation is concerned, though of course the president has a a huge amount of power beyond the ability to legislate.
There are more differences than you mention. The PM is less hindered by the independent judiciary than the president. The PM in a Westminster system also exerts greater control over the individual legislators via his party than in the American system. The PM can serve for an unlimited time, and call elections at strategic moments, while Trump is limited to two terms. All these things increase the power of the PM and the risk of oppressive rule in Westminster-style parliamentary systems.
I agree that some of these are differences giving a PM more power, in particular the ability to serve indefinitely and call elections strategically (which seems quite bad). The rest do not seem to me to be inherent in parliamentarianism, and indeed it is not clear to me that they are even tendencies.
This is a recent historical trend and not a defining feature of the system itself.
It’s not just a historical trend within the US though, but an observed tendency of other presidential systems, and does follow somewhat from the game-theoretic logic of that system.
The U.S. president holds a weaker office than the heads of government in most other countries. The Canadian and British PMs and the French presidents definitely seem stronger; the German Chancellor seems weaker, and maybe the Israeli and Italian and Japanese PMs? (These aren’t strong views). I most often hear from proponents of the parliamentary system that it is less gridlocked and more powerful/effective rather than less.
It is less gridlocked, but that’s because the PM works for parliament and serves at its pleasure, much as a CEO for a board of directors. The PM normally can be removed by simple majority vote of no confidence at any time. While somewhat infrequent, this occurs often enough — and is a plausible enough threat even when it does not occur — that it cannot really be called exceptional in the way that the successful removal of a president via impeachment would be (which in the US is structurally very burdensome: demanding actual wrongdoing — “high crimes and misdemeanors” — rather than a mere loss of confidence, a majority in the House, an entire trial, and then a two-thirds majority in the Senate, and we have seen how difficult this bar is to meet even for extraordinarily unusual behavior). Furthermore, the PM has no formal say in legislation, which is another reason for less gridlock (though typically, as the head of their party, they do have great influence, but again, only so long as they can maintain a governing coalition within parliament).
It is precisely because of the gridlock created by a presidential system, with its “checks and balances”, that over time more power tends to be arrogated to the president in order to “get things done” that aren’t getting done otherwise, often without the political will to stand in the way of such arrogation when it occurs.
In the US specifically, another way in which the president has recently gained tremendous power stems from these “checks and balances”: the Supreme Court has opined that if presidential acts were subject to regular law, then this would give Congress the power to limit Article II presidential power.[1] This kind of consideration is normally not at issue in a parliamentary system, and thus the PM is normally subject to criminal law.
- ^
This basic logic seems very defensible to me, although they seem to have extended the notion of “official [presidential] acts” substantially beyond anything explicit in the Constitution, and then gone even further, to preclude not only prosecution for such acts, but even judicial consideration of such acts as evidence in a prosecution for non-official acts, under the theory that allowing such evidence would have a chilling effect on the president’s freedom to act within constitutional limits. However, this is very different from how we treat speech: we don’t say that a tweet is inadmissible in court as evidence for a non-speech crime, even though the tweet itself may be constitutionally protected speech which must not be chilled.
- ^
The link to join the discord for the November 1st start date wasn’t working yesterday. Can you update it please?
Evidence on language model consciousness
You might be inferring an implicit “all” before “bad[/nice] people” where an implicit “many” was intended.
But I have not seen any kind of vision painted for how you avoid a bad future, for any length of time, that doesn’t involve some kind of process that is just… pretty godlike?
I’m mostly with you all the way up to and including this line. But I would also add: I have not seen a plausible vision painted for how you avoid a bad future, for any length of time, that does involve some kind of process that is just pretty godlike.
This is why I put myself in the “muddle through” camp. It’s not because I think doing so guarantees a good outcome; indeed I’d be hard-pressed even to say it makes it likely. It’s just that by trying to do more than that — to chart a path through territory that we can’t currently even see — we are likely to make the challenge even harder.
Consider someone in 1712 observing the first industrial steam engines, recognizing the revolutionary potential, and wanting to … make sure it goes well. Perhaps they can anticipate its use outside of coal mines — in mills and ships and trains. But there’s just no way they can envision all of the downstream consequences: electricity, radio and television, aircraft, computing, nuclear weapons, the Internet, Twitter, the effect Twitter will have on American democracy (which by the way doesn’t exist yet…), artificial intelligence, and so on. Any attempt someone would have made, at that time, to design, in detail, a path from the steam engine to a permanently good future would just have been guaranteed at the very least to fail, and probably to make things much worse to the extent they locally succeed in doing anything drastic.
Our position is in many ways more challenging than theirs. We have to be humble about how far into the future we can see. I agree that an open society comes with great danger and it’s hard to see how that goes well in the face of rapid technological change. But so too is it hard to see how centralized power over the future leads to a good outcome, especially if the power centralization begins today, in an era when those who would by default possess that power seem to be … extraordinarily cruel and unenlightened. Just as you, rightly, cannot say if AIs who replace us would have any moral value, I also cannot say that an authoritarian future has any value. Indeed, I cannot even say that its value is not hugely negative.
What I can say, however, is that we have some clear problems directly in front of us, either occurring right now or definitely in sight, one of which is this very possibility of a centralized, authoritarian future, from which we would have no escape. I support muddling through only because I see no alternative.
This seems similar to saying that there are holes in Newton’s theory of gravity, therefore choosing to throw out any particular prediction of the theory.
Newton’s theory of gravity applies to high precision in nearly every everyday context on Earth, and when it doesn’t we can prove it, thus we need not worry that we are misapplying it. By contrast, there are routine and substantial deviations from utility maximizing behavior in the everyday life of the only intelligent agents we know of — all intelligent animals and LLMs — and there are other principles, such as deontological rule following or shard-like contextually-activated action patterns, that are more explanatory for certain very common behaviors. Furthermore, we don’t have simple hard and fast rules that let us say with confidence when we can apply one of these models, unlike the case with gravity.
If someone wanted to model human behavior with VNM axioms, I would say let’s first check the context and whether the many known and substantial deviations from VNM’s predictions apply, and if not then we may use them, but cautiously, recognizing that we should take any extreme prediction about human behavior — such as that they’d violate strongly-held deontological principles for tiny (or even large) gains in nominal utility — with a large serving of salt, rather than confidently declaring that this prediction will be definitely right in such a scenario.
it’s very important to note, if it is indeed the case, that the implications for AI are “human extinction”.
Agreed, and noted. But the question here is the appropriate level of confidence with which those implications apply in these cases.
An update on this.
Delta Replaces Engine Units in Effort to Address Toxic-Fume Surge on Planes (gift link):
Delta Air Lines is replacing power units on more than 300 of its Airbus jets in an effort to stem cases in which toxic fumes have leaked into the air supply and led to health and safety risks for passengers and crew.
… The airline is about 90% of its way through the process of upgrading the engines, a type known as the auxiliary power unit, on each of its Airbus A320 family jets, according to a spokesman for Delta. The airline operates 310 of the narrow-body type, including 76 of the latest generation models as of the end of June.
… Delta hasn’t previously disclosed the APU replacement program, which began in 2022.
Replacing the APU, which can become more prone to fume events with age, mitigates some of the risks from toxic leaks but doesn’t address them entirely. Airbus last year found that most cases on the A320 were linked to leaks entering the APU via an air inlet on the aircraft’s belly.
Another separate cause is leaks in the jet engines themselves, which provide most of the bleed-air supply when active.
Again, I’m not talking about minor differences. Children care an awful lot about whether Santa Claus as usually defined exists. This is not small.
In other words, to control AI we need global government powerful enough to suppress any opposition.
That’s the risk at least, yes. (Not sure I agree with all of the specifics which follow in your comment, but I agree with the gist.)
The degree to which his definition is “very different” is not clear.
I disagree. I think it’s clear that hardly any children use this novel definition of Santa Claus. But if you’re right, then it’s imperative to make it clear before employing your own definition which would serve to mislead.
Definitions vary at least slightly from person to person all the time but we don’t make long semantic declarations in normal conversation unless it serves some specific functional purpose.
But this is not a slight difference, it’s a huge and unusual difference in a commonly used term. The functional purpose here is to avoid lying.
Is the child being clear with his friends that by Santa Claus he means something very different than they do? If not then he is lying to his friends.
What purpose is the new definition of the term “Santa Claus” serving here other than to confuse?
Words take on meaning mainly in two ways: either through natural evolution of usage, or deliberate choice of definition. When we do the latter, we should do it to enable clarity, precision, and distinction of concepts, not to muddy the waters between things which are true and things which are false.
We could call anything anything, but the term “Santa Claus” is not normally used by children to refer to “whatever happens to be causing the phenomenon of presents appearing under the tree”. Rather, it’s used by children (and adults who lie to them) to refer to a particular, specific, and false explanation for that phenomenon.
Therefore, if a child asks me if Santa Claus is real, the meaning of that question is not determined by other possible definitions I could theoretically assign to “Santa Claus”. The meaning of the question is determined by the child’s intended meaning, since the child asked the question. And if I say yes, because in my mind I’m imagining this novel definition for the term, then I have lied to that child. There is an unfortunate tradition, almost as old as this lie itself, for inventing rationalizations for the lie in which we adults convince ourselves that it is not really a lie after all, to ease our conscience about it.
Note that the issue is not one of literalism, as in your Aurora Borealis example. Like you say, everyone today knows today that the term Aurora Borealis is not intended to refer to an actual dawn, so there is no lie in using it. But if someone did think it meant that, and I knew this, and they asked me if it exists, then I would be lying to say yes.
Draconian measures can increase the risk of irrevocable catastrophe
The relevant way in which it’s analogous is that a head of state can’t build [dangerous AI / nuclear weapons] without risking war (or sanctions, etc.).
Fair enough, but China and the US are not going to risk war over that unless they believe doom is anywhere close to as certain as Eliezer believes it to be. And they are not going to believe that, in part because that level of certainty is not justified by any argument anyone including Eliezer has provided. And even if I am wrong on the inside view/object level to say that, there is enough disagreement about that claim among AI existential risk researchers that the outside view of a national government is unlikely to fully adopt Eliezer’s outlier viewpoint as its own.
But in return, we now have the tools of authoritarian control implemented within each participating country. And this is even if they don’t use their control over the computing supply to build powerful AI solely for themselves. Just the regime required to enforce such control would entail draconian invasions into the lives of every person and industry.
Do we know the controls were “bolted on top”? It was a “reduced cyber refusals” model. That description is compatible with some fine-tuning to remove scruples.