shminux
Karma: 28,672
Litany of a Bright Dilettante
Wirehead your Chickens
Wow, I’m blown away by Holden Karnofsky, based on this post alone. His writing is eloquent, non-confrontational and rational. It shows that he spent a lot of time constructing mental models of his audience and anticipated its reaction. Additionally, his intelligence/ego ratio appears to be through the roof. He must have learned a lot since the infamous astroturfing incident. This is the (type of) person SI desperately needs to hire.
Emotions out of the way, it looks like the tool/agent distinction is the main theoretical issue. Fortunately, it is much easier than the general FAI one. Specifically, to test the SI assertion that, paraphrasing Arthur C. Clarke,
Any sufficiently advanced tool is indistinguishable from an agent.
one ought to formulate and prove this as a theorem, and present it for review and improvement to the domain experts (the domain being math and theoretical computer science). If such a proof is constructed, it can then be further examined and potentially tightened, giving new insights to the mission of averting the existential risk from intelligence explosion.
If such a proof cannot be found, this will lend further weight to the HK’s assertion that SI appears to be poorly qualified to address its core mission.
This post had an odd effect on me. I agreed with almost everything in it, as it matches my own logic and intuitions. Then I realized that I strongly disliked the logic in your anti-meat post, because it appeared so severely biased toward a predefined conclusion “eating meat is ethically bad”. So, given the common authorship, I must face the possibility that the quality of the two posts is not significantly different, and it’s my personal biases which make me think that it is. As a result, I am now slightly more inclined to consider the anti-meat arguments seriously and slightly less inclined to agree with the arguments from this post, even though the foggy future and the lack of feedback arguments make a lot of sense.
EDIT: Hmm, whatever shall I do with 1 Eliezer point and 1 Luke point...
What happened to holding off proposing solutions?
Main is useless to me as is. As I mentioned a few times before, it should be replaced/supplemented by a “highly rated”/”greatest hits”/”best of LW” section, whether generated automatically based on post’s karma or updated manually once a day or so.
Also, instead of/in addition to subreddits, you can create a list of approved tags/keywords a poster is forced to select one or more from, which trigger notifications to these busy people, or to anyone else who subscribes to notifications.
So you think you understand Quantum Mechanics
Top lesson from GPT: we will probably destroy humanity “for the lulz” as soon as we are able.
Boeing 737 MAX MCAS as an agent corrigibility failure
I just wanted to mention that you assume consequentialist thinking, specifically of the type “what should we do to change X for the better?” This is not at all how most people think. “Price gouging is unfair” is enough to pass a legislation, without heeding the consequences. “Abortion is against a sacred rule from God” is enough to fight to prohibit it. “But I can change my mind” is argument enough to two-box. And I’m not even touching issues where people don’t reason at all, or, like politicians, optimize something other than the stated goal.
There is no territory, it’s maps all the way down.
Respect Chesterton-Schelling Fences
Just wanted to mention that an amazing amount of arguments in this thread and in the linked piece consists of misidentified non-central fallacies (in Yvain’s labelling). None of the targets of the labels used (“racist”, “eugenics”, “feminist”, what have you), correspond to a typical image evoked by using them.
Given that LW was almost dead a few years back, and is now a vibrant community against the odds, without losing its uniqueness, including Eliezer and sometimes Scott Alexander posting again, kudos go to the LW 2.0 team (and to the GW mirror).
Mathematics as a lossy compression algorithm gone wild
To me this is a metaphor for Alignment research, and LW-style rationality in general, but with an opposite message.
To start, I have this exact AC in my window, and it made a huge difference during last year’s heat dome. (I will use metric units in the following, because eff imperial units.) It was around 39-40C last summer, some 15C above average, for a few days, and the A/C cooled the place down by about 10C, which made a difference between livable and unlivable. It was cooler all through the place, not just in the immediate vicinity of the unit.
How could this happen, in an apparent contradiction to the laws of physics?
Well, three things:
I live in an apartment, so the air coming in is not quite as hot in the hallway as outside, though still pretty warm.
The air coming out of the AC exhaust is pretty hot, hotter than the outside most of the time, so there is a definite cooling that happens despite the air influx from outside.
The differential air pressure in the hallway is positive regardless of the AC (partly because of the exhaust vents that are always on), so adding AC does not significantly change the air flow.
So, physics is safe! What isn’t safe is the theoretical reasoning that “a single-hose AC does not work”.
Why? Because of the assumptions that go into the reasoning, some stated, some unstated. For example, that outside air is necessarily hot, that there is an extra ingress of outside air due to AC, that people use this AC for detached houses, etc.
AI research is much more complicated than HVAC analysis, and so there are many more factors, assumptions and effects that go into it. Confidently proclaiming a certain outcome, like “we are all doomed!” is based on a sound analysis of incomplete data, and better data, given the bounded rationality constraint, can only be obtained iteratively through experiment and incremental improvement.
Note that the analysis can look airtight (no pun intended) from outside: in the AC example this is basic energy conservation and the continuity equation. In the dieting case it’s calories-in-calories out. In AI research it’s the… inability to steer the direction of recursive self-improvement or something. But that model of analysis has been falsified again and again: The Waterfall approach to software development gave way to Agile. Space X’s fast iterations let it run rings around Boeing’s “we carefully design it once and then it will fly”.
The more complicated a problem, the more iterative an acceptable solution will be.
And you cannot pronounce something possible or impossible until you went through a lot of iterations of actually building something and gained the experience that becomes knowledge of what works and what does not. That’s how Junior developers become Senior ones.
Now, note that MIRI has not built a single AI. Unlike those other companies. All their reasoning is a sound theoretical analysis… based on none or very little experimental data.
It may well be that Eliezer and Co are correct about everything. But the outside view puts them in the reference class of those who are more often wrong than right: those who rely exclusively on logic and eschew actually building things and see where they work, where they break and why.
To be fair, there are plenty of examples where theoretical reasoning is enough. For example, all kinds of Perpetual Motion machines are guaranteed to be bunk. Or EM drive-style approaches. Or reducing the entropy of a closed system. If we had ironclad experimentally tested and confirmed laws like that for AI research, we would be able to trust the theoretical conclusions. As it is, we are way too early in the process of experimenting and generalizing our experimental data into laws, as far as AI research and especially Alignment research is concerned. We may be unlucky and accidentally end up all dead. But to figure out the odds of it happening one needs to do realistic experimental alignment research: build small and progressively large AI and actually try to align it. “Debates” are all nice and fun, but no substitute for data.
So the message I get from the OP is “experiment to get the data and adjust your assumptions”, or else you may end up overpaying for a super-duper HVAC system you don’t need, or worse, deciding you cannot afford it and dying from a heat stroke.
Done. I’m glad there was nothing about Schrodinger this time around.
[LINK] Claustrum Stimulation Temporarily Turns Off Consciousness in an otherwise Awake Patient
I had been a small-time LW regular for about 3 years and witnessed its decline until I stopped commenting a couple of months back. It was frustrating to see Eliezer, Yvain, Luke and others leave for other social media platforms, and even more to watch them fragment their writings further between personal blogs, FB, Reddit and tumblr. Not because it’s a wrong thing to do, just because it’s harder to follow and the commenting system is usually even worse than here. Well, except for Reddit. Without a strong leader charismatic emerging and willing to add quality content and drive the changes, I don’t expect any site redesign to revive this rather zombified forum. Or maybe if the forum is redesigned one would emerge, who knows. Chicken and egg.
Or maybe it should be a rationality-related aggregator/hub, where all relevant links get posted and discussed. So that one could see at a glance that Scott A posted something on his blog, Eliezer on tumbler, Brienne on Facebook, gwern on his site and someone else on twitter or reddit. All on one page. There are various sites like that around. With the ability to comment locally, or go to the source and discuss it there. Maybe even add linkbacks to this site.
Just my 2c.
Start your post or comment with a summary when posting anything over 3-5 paragraphs.