Did you consider withdrawal effects at all? A day without caffeine having used it the previous few days is going to be very different from one where you haven’t used caffeine in months.
Tofly
2010s Predictions Review
Memorizing a Deck of Cards
Thoughts on the 5-10 Problem
The brain may also be excessively complicated to defend against parasites.
I believe that fast takeoff is impossible, because of computational complexity.
This post presents a pretty clear summary of my thoughts. Essentially, if the problem of “designing an AI with intelligence level n” scales at any rate greater than linear, this will counteract any benefit an AI received from its increased intelligence, and so its intelligence will converge. I would like to see a more formal model of this.I am aware that Gwern has responded to this argument, but I feel like he missed the main point. He gives many arguments showing ability to solve an NP-complete problem in polynomial time, or still do better than a human, or still gain a benefit from performing mildly better than a human.
But the concern here is really about A.I.s performing better than humans at certain tasks. It’s about them rapidly, and recursively, ascending to godlike levels of intelligence. That’s what is being argued is impossible. And there’s a difference between “superhuman at all tasks” and “godlike intelligence enabling what seems like magic to lesser minds”.
One of Eliezer’s favorite examples of how a powerful AI might take over the world is by solving the protein folding problem, designing some nano-machines, and using an online service to have them constructed. The problem with this scenario is the part where the AI “solves the protein folding problem”. That the problem is NP-hard means that it will be difficult, no matter how intelligent the AI is.
Something I’ve felt confused about, as I’ve been writing this up, is this problem of “what is the computational complexity of designing an AI with intelligence level N?” I have an intuition that there should be some “best architecture”, at least for any given environment, and that this architecture should be relatively “simple”. And once this architecture has been discovered, that’s pretty much it for self-improvement-you can still benefit from acquiring extra resources and improving your hardware, but these have diminishing returns. The alternative to this is that there’s an infinite hierarchy of increasingly good AI designs, which seems implausible to me. (Though there is an infinite hierarchy of increasingly large numbers, so maybe it isn’t.)
Now, it could be that even without fast takeoff, AGI is still a threat. But this is a different argument than “as soon as artificial intelligence surpasses human intelligence, recursive self-improvement will take place, creating an entity we can’t hope to comprehend, let alone oppose.”
Edit: Here is a discussion involving Yudkowsky regarding some of these issues.
Personal data point: I tried this last night, and had 4 vivid and intense nightmares. This is not a usual experience for me.
In most formulations, the five people are on the track ahead, not in the trolley.
I took a look at the course you mentioned:
It looks like I got some of the answers wrong.
Where am I?
In the trolley. You, personally, are not in immediate danger.
Who am I?
A trolley driver.
Who’s in the trolley?
You are. No one in the trolley is in danger.
Who’s on the tracks?
Five workers ahead, one to the right.
Do I work for the trolley company?
Yes.
The problem was not as poorly specified as you implied it to be.
From the old wiki discussion page:
I’m thinking we can leave most of the discussion of probability to Wikipedia. There might be more to say about Bayes as it applies to rationality but that might be best shoved in a separate article, like Bayesian or something. Also, I couldn’t actually find any OB or LW articles directly about Bayes’ theorem, as opposed to Bayesian rationality—if anyone can think of one, please add it. --A soulless automaton 19:31, 10 April 2009 (UTC)
I’d rather go for one article than break out a separate one for Bayesian—we can start splitting things out if the articles start to grow too long. --Paul Crowley (ciphergoth) 22:58, 10 April 2009 (UTC)
I added what I thought was the minimal technical information. I lean towards keeping separate concepts separate, even if the articles are sparse. If someone else feels it would be worthwhile to combine them though, go ahead. --BJR 23:07, 10 April 2009 (UTC)
I really would prefer to keep the maths and statistics separate from the more nebulous day-to-day rationality stuff, especially since Wikipedia already does an excellent job of covering the former, while the latter is much more OB/LW-specific. --A soulless automaton 21:59, 11 April 2009 (UTC)
I’m going to use this space to blurt some thoughts about ai risk.
[Meta] The jump from Distinct Configurations to Collapse Postulates in the Quantum Physics and Many Worlds sequence is a bit much—I don’t think the assertiveness of Collapse Postulates is justified without a full explanation of how many worlds explains things. I’d recommend adding at least On Being Decoherent in between.
Doesn’t that mean the agent never makes a decision?
What exactly is being done—what type of thing is being created—when we run a process like “use gradient descent to minimize a loss function on training data, as long as the loss function is also being minimized on test data”?
Is a language model performing utility maximization during training?
Let’s ignore RLHF for now and just focus on next token prediction. There’s an argument that, of course the LM is maximizing a utility function—namely it’s log score on predicting the next token, over the distribution of all text on the internet (or whatever it was trained on). An immediate reaction I have to this is that this isn’t really what we want, even ignoring that we want the text to be useful (as most internet text isn’t).
This is clearly related to all the problems around overfitting. My understanding is that in practice, this is solved through a combination of regularization, and stopping training once test loss stops decreasing. So even if a language model was a UM during training, we already have some guardrails on it. Are they enough?
Are language models utility maximizes?
I think there’s two different major “phases” of a language model, training and runtime. During training, the model is getting “steered” toward some objective function—first getting the probability of the next token “right”, and then getting positive feedback from humans during rlhf (I think? I should read exactly how rlhf works). Is this utility maximization? It doesn’t feel like it—I think I’ll put my thoughts on this in another comment.
During runtime, at first glance, the model is kind of “deterministic” (wrong word), in that it’s “just multiplying matrices”, but maybe it “learned” some utility maximizers during training and they’re embedded within it. Not sure if this is actually possible, or if it happens in practice, and if the utility maximizers are dominate the agent or can be “overruled” by other parts of it.
There’s sort of a general “digestive issues are the root of all anxiety/evil” thread I’ve seen pop up in a bunch of rationalist-adjacent spaces:
https://www.reddit.com/r/slatestarcodex/comments/yxyrf3/comment/iwr9vf2/
https://twitter.com/Aella_Girl/status/1454952550327848964
I’m curious if there’s any synthesis / study / general theory of this.
As my own datapoint, I’ve had pretty bad digestive issues, trouble eating, and ahedonia for a while. I recently got to do a natural experiment when I accidentally left my milk out. Since I’ve cut milk, I’ve felt much better (though not perfect) on all counts. So I’m probably lactose intolerant (though I never really noticed a correlation between my symptoms and milk consumption).
Probably worth checking if there are any easy fixes to your digestive issues if you have them.
cure Eliezer’s chronic fatigue so he can actually attempt to
grant humanity a couple more bits of information-theoretic dignitysave the worldPossibly relevant: I know someone who had chronic fatigue syndrome which largely disappeared after she had her first child. I could possibly put her in contact with Eliezer or someone working on the problem.
The entrepreneur contacted me again the next day...”I have been cooking all week.”
Hmmmmmm...
Reminds me of arbital.