Worth noting that the reason SquirrelInHell is dead is that they committed suicide after becoming mentally unstable, likely in part due to experimentation with exotic self-modification techniques. This one in particular seems fine AFAICT, but, ya know, caveat utilitor.
interstice
Karma: 2,076
Noting that the author deleted a critical comment which was somewhat rude but IMO made some reasonable points. That’s fair enough, but in conjunction with the way the site handles deletions this strikes me as bad, since there’s no way of (a) seeing which user posted the deleted comment (this might be a bug?) (b) examining the text of deleted comments. Together this means that you can’t distinguish cases where a post attracts no criticism(an important signal) and cases where there were critical comments that were deleted, and you can’t examine deleted criticisms.
In the same vein, I humbly suggest “The entire bee movie but every time they say bee it gets faster” as a good model for what the singularity will seem like from our perspective.
It’s neither a hoax nor a HLAI, instead a predictable consequence of prompting a LLM with questions about its sentience: it will imitate the answers a human might give when prompted, or the sort of answers an AI in a science fiction story would give.
As John conjectured, alt-complexity is a well-known notion in algorithmic information theory, and differs from K-complexity by at most a constant. See section 4.5 of this book for a proof. So I think the stuff about how physics favors alt-complexity is a bit overstated—or at least, can only contribute a bounded amount of evidence for alt-complexity.
ETA: this result is about the complexity of finite strings, not semimeasures on potentially infinite sequences; for such semimeasures, there actually is a non-constant gap between the log-total-probability and description complexity.
Interpretability seems like our clear best bet for developing a more principled understanding of how deep learning works
If our goal is developing a principled understanding of deep learning, directly trying to do that is likely to be more effective than doing interpretability in the hope that we will develop a principled understanding as a side effect. For this reason I think most alignment researchers have too little awareness of various attempts in academia to develop “grand theories” of deep learning such as the neural tangent kernel. I think the ideal use for interpretability in this quest is as a way of investigating how the existing theories break down—e.g. if we can explain 80% of a given model’s behavior with the NTK, what are the causes of the remaining 20%? I think of interpretability as basically collecting many interesting data points; this type of collection is essential, but it can be much more effective when it’s guided by a provisional theory which tells you what points are expected and what are interesting anomalies which call for a revision of the theory, which in turn guides further exploration, etc.
Interesting that you think the risk of us “going crazy” after getting AI in some way is roughly comparable to overall AI takeover risk. I’d be interested to hear more if you have more detailed thoughts here. On this view it also seems like it could be a great x-risk-reduction opportunity if there are any tractable strategies, given how neglected it is compared to takeover risk.
I think it would also be valuable to have someone translate “in the other direction” and take (for example) Paul Christiano’s writings and produce vivid, concrete parable-like stories based on them. I think such stories can be useful not just as persuasive tools but also epistemically, as a way of grounding the meaning of abstract definitions of the sort Paul likes to argue in terms of.
In rationalist circles, you might find out that you’re being instrumentally or epistemically irrational in the course of a debate—the norms of such a debate encourage you to rebut your opponent’s points if you think they are being unfair. In contrast, the central thesis of this book is that white people disputing their racism is a mechanism for protecting white supremacy and needs to be unlearned, along with other cornerstones of collective epistemology such as the notion of objective knowledge. So under the epistemic conditions promoted by this book, I expect “found about being racist” to roughly translate to “was told you were racist”.
ZMD: Looking at “Silicon Valley’s Safe Space”, I don’t think it was a good article. Specifically, you wrote,
In one post, [Alexander] aligned himself with Charles Murray, who proposed a link between race and I.Q. in “The Bell Curve.” In another, he pointed out that Mr. Murray believes Black people “are genetically less intelligent than white people.”
End quote. So, the problem with this is that the specific post in which Alexander aligned himself with Murray was not talking about race. It was specifically talking about whether specific programs to alleviate poverty will actually work or not.
So on the one hand, this particular paragraph does seem like it’s misleadingly implying Scott was endorsing views on race/iq similar to Murray’s even though, based on the quoted passages alone, there is little reason to think that. On the other hand, it’s totally true that Scott was running a strategy of bringing up or “arguing” with hereditarians with the goal of broadly promoting those views in the rationalist community, without directly being seen to endorse them. So I think it’s actually pretty legitimate for Metz to bring up incidents like this or the Xenosystems link in the blogroll. Scott was basically using a strategy of communicating his views in a plausibly deniable way by saying many little things which are more likely if he was a secret hereditarian, but any individual instance of which is not so damning. So I feel it’s total BS to then complain about how tenuous the individual instances Metz brought up are—he’s using it as an example or a larger trend, which is inevitable given the strategy Scott was using.
(This is not to say that I think Scott should be “canceled” for these views or whatever, not at all, but at this stage the threat of cancelation seems to have passed and we can at least be honest about what actually happened)
Not sure how much I believe this myself, but Jacob cannell has an interesting take that social status isn’t a “base drive” either, but is basically a proxy for “empowerment”, influence over future states of the world. If that’s true it’s perhaps not so surprising that we’re still well-aligned, since “empowerment” is in some sense always being selected for by reality.
I always liked the interpretation of the determinant as measuring the expansion/contraction of n-dimensional volumes induced by a linear map, with the sign being negative if the orientation of space is flipped. This makes various properties intuitively clear such as non-zero determinant being equivalent to invertibility.
In the analogy, it’s only possible to build a calculator that outputs the right answer on non-13 numbers because you already understand the true nature of addition. It might be more difficult if you were confused about addition, and were trying to come up with a general theory by extrapolating from known cases—then, thinking 6 + 7 = 15 could easily send you down the wrong path. In the real world, we’re similarly confused about human preferences, mind architecture, the nature of politics, etc., but some of the information we might want to use to build a general theory is taboo. I think that some of these questions are directly relevant to AI—e.g. the nature of human preferences is relevant to building an AI to satisfy those preferences, the nature of politics could be relevant to reasoning about what the lead-up to AGI will look like, etc.
Seems to me that the ‘helpful’ works you listed contain falsehoods and wrong associations. They also contain useful information and enjoyable aspects, true—but couldn’t the same be said of lots of non-”rational” fiction? As it stands this just looks like a list of fiction that’s popular among our subculture.
An interesting post, but I don’t know if it implies that “strong AI may be near”. Indeed, the author has written another post in which he says that we are “really, really far away” from human-level intelligence: https://karpathy.github.io/2012/10/22/state-of-computer-vision/.
How is Metz’s behavior here worse than Scott’s own behavior defending himself? After all, Metz doesn’t explicitly say that Scott believes in racial iq differences, he just mentions Scott’s endorsement of Murray in one post and his account of Murray’s beliefs in another, in a way that suggests a connection. Similarly, Scott doesn’t explicitly deny believing in racial iq differences in his response post, he just lays out the context of the posts in a way that suggests that the accusation is baseless(perhaps you think Scott’s behavior is locally better? But he’s following a strategy of covertly communicating his true beliefs while making any individual instance look plausibly deniable, so he’s kinda optimizing against “locally good behavior” tracking truth here, so it seems perverse to give him credit for this)
if you act like a ratfic character, you win in ratfic and you win in real life.
Citation needed.
The problem of whether the goals and values of an artificially intelligent agent will align with human goals and values, can be reduced to this problem: Will the goals and values of different human agents ever align with each other?
We can only program AI “in our own image”, so both the features and bugs of humanity will reappear in AI.
Why do you think these statements are true?
There aren’t enough simulators above us that care enough about us-in-particular to pay in paperclips. There are so many things to care about! Why us, rather than giant gold obelisks?
What about neighboring Everett branches where humanity succeeds at alignment? If you think alignment isn’t completely impossible, it seems such branches should have at least roughly comparable weight to branches where we fail, so trade could be possible.
At this point timelines look short enough that you likely increase your personal odds of survival more by increasing the chance that AI goes well than by speeding up timelines. Also I don’t see why you think cryonics doesn’t make sense as alternative option.