Inferring properties of the authors of some text isn’t itself something I consider wildly useful for takeover, but I think of it as belonging to this more general cluster of capabilities.
You don’t? Ref the bribery and manipulation in eg. Clippy. Knowing who you are dealing with seems like a very useful capability in a lot of different scenarios. Eg. you mention phishing.
Great post! I’m all for more base model research.
That isn’t my argument, my argument is just that the general tone seems too defeatist.
The question asker was under the impression that the probabilities were %99.X percent against anything okay. My only argument was that this is wrong, and there are good reasons that this is wrong.
Where the p(doom) lies between 99 and 1 percent is left as an exercise for posterity. I’m not totally unhinged in my optimism, I just think the tone of certain doom is poorly founded and there are good reasons to have some measure of hope.
Not just ‘i dunno, maybe it will be fine’ but real reasons why it could conceivably be fine. Again, the probabilities are up for debate, I only wanted to present some concrete reasons.
If there was a system which was really good at harvesting energy and it was maxxed out on intelligence, atoms might be very valuable, especially atoms close to where it is created
The number of atoms on earth is so tiny. Why not just head to the asteroid belt where you can really build?
It would be like a single medieval era human suddenly taking over the world via powerful magic. Would the resulting world after optimization according to that single human’s desires still score reasonably well at IGF?
Interestingly, this thought experiment was run many times at the time, see for example all the wish fulfillment fantasies in the 1001 Nights or things like the Sorcerers Apprentice.
Ah I see.
I certainly concede that the argument about counterfactual populations has a lot more force.
Personally I would solve this with increased support for eg. polygenic screening and other reproductive technologies and less regulation about what they can select for, and hope that people do their weird people thing and choose diversity. I worry that regulation will always result in more standardization.
And I for sure don’t think punishing people for making reproductive choices is a good move, even if those choices result in the extinction of specific populations.
How is this kind of reasoning about counterfactual children never born different from the regular Christian stuff about not masturbating?
A statements like ‘my parents would have used polygenic screening to kill me’ is no more meaningful than ‘you are murdering your counterfactual children when you wear a condom’ or something like that. It seems to have more meaning because you are talking about yourself, but in the universe where ‘you’ were ‘murdered’ by polygenic screening, ‘you’ does not refer to anything.
give THEM plausibility deniability about having to understand or know things based on their own direct assessment
I don’t follow what you are getting at here.
I’m just thinking about historical cases of catastrophic risk, and what was done. One thing that was done, was the the government payed very clever people to put together models of what might happen.
My feeling is that the discussion around AI risk is stuck in an inadequate equilibrium, where everyone on the inside thinks its obvious but people on the outside don’t grok it. I’m trying to think of the minimum possible intervention to bridge that gap, something very very different from your ‘talk … for several days face-to-face about all of this’. As you mentioned, this is not scalable.
In summary: this proposals feels like you’re personally asking to be “convinced in public using means that third parties can watch, so that third parties will grant that it isn’t your personal fault for believing something at variance with the herd’s beliefs” and not like your honest private assessment of the real situation is bleak. These are different things.
Well, thats very unfortunate because that was very much not what I was hoping for.
I’m hoping to convince someone somewhere that proposing a concrete model of foom will be useful to help think about policy proposals and steer public discourse. I don’t think such a model has to be exfohazardous at all (see for example the list of technical approaches to the singularity, in the paper I linked—they are good and quite convincing, and not at all exfohazardous)!
Everyone here acting like this makes him some kind of soothsayer is utterly ridiculous. I don’t know when it became cool and fashionable to toss off your epistemic humility in the face of eternity, I guess it was before my time.
The basilisk is just pascals mugging for edgelords.
One criticism of humanism you don’t seem to touch on is,
isn’t it possible that humanism directly contributes to the ongoing animal welfare catastrophe?
And indeed, it was something very like humanism (let’s call it specific humanism) that laid the ideological foundation for the slave trade and the holocaust.
My view is that humanism can be thought of as a hangover of Christian values, the belief that our minds are the endowments of God.
But if we have been touched by the angels, perhaps the non metaphorical component of that is the development of the infosphere/memetic landscape/culture. Which is close to synonymous with technology. Edit: considering eg. writing a technology that is.
Per the recent Nightshade paper, clown attacks would be a form of semantic poisoning on specific memeplexes, where ‘memeplex’ basically describes the architecture of some neural circuits. Those memeplexes at inference time would produce something designed to propagate themselves (a defence or description of some idea, submeme), and a clown attack would make that propagation less effective at transmitting to eg. specific audiences.
It seems here that you are really worried about ‘foom in danger’ (danger per intelligence, D / I) than regular foom (4+ OOM increase in I), if I am reading you correctly. Like I don’t see a technical argument that eg. the claims in OP about any of
/flop, flop/J, total(J), flop/$, or total($)
are wrong, you are just saying that ‘D / I will foom at some point’ (aka a model becomes much more dangerous quickly, without needing to be vastly more powerful algorithmically or having much more compute).
This doesn’t change things much but I just want to understand better what you mean when you say ‘foom’.
Thats a good paper, but I think it exemplifies the problem outlined by Cowen—it mostly contains references to Bostrom and Yudkowsky, it doesn’t really touch on more technical stuff (Yampolskiy, Schmidhuber) which exists, which makes me think that it isn’t a very thorough review of the field. It seems like more of the same. Maybe the Hubinger paper referenced therein is on the right track?
The question of where to do science is relevent but not important—Cowen even mentions that ‘if it doesn’t get published, just post it online’—he is not against reading forums.
It really looks like there could be enough stuff out there to make a model. Which makes me think the scepticism is even more justified! Because if it looks like a duck and talks like a duck but doesn’t float like a duck, maybe its a lump of stone?
Just to preserve information. It’s not every day that you come across a thermodynamic system that has been evolving so far from equilibrium for so long. There is information here.
In general, I feel like a lot of people in discussion about ASI seem to enjoy fantasizing about science fiction apocalypses of various kinds. Personally I’m not so interested in exercises in fancy, rather looking at ways physical laws might imply that ‘strong orthogonality’ is unlikely to obtain in reality.
“Whatever happened here is a datapoint about matter and energy doing their usual thing over a long period of time.”
Not all thermodynamic systems are created equal. I know enough about information theory to know that making bold claims about what is interesting and meaningful is unwise. But I also know it is not certain that there is no objective difference between a photon wandering through a vacuum and a butterfly.
Here is one framework for understanding complexity that applies equally well for stars, planets, plants, animals, humans and AIs. It is possible I am typical-minding, but it is also possible that the universe cares about complexity in some meaningful way. Maybe it helps increase the rate of entropy relaxation. I don’t know.
spontaneously developing a specific interest in the history of how natural selection developed protein-based organic machines on one particular planet
not ‘one particular planet’ but ‘at all’.
I find it plausible that there is some sense in which the universe is interested in the evolution of complex nanomachines. I find it likely that an evolved being would be interested in the same. I find very likely that an evolved being would be particularly interested in the evolutionary process by which it came into being.
Whether this leads to s-risk or not is another question, but I think your implication that all thermodynamic systems are in some sense equally interesting is just a piece of performative cynicism and not based on anything. Yes this is apparently what matter and energy will do given enough time. Maybe the future evolution of these atoms is all predetermined. But the idea of things being interesting or uninteresting is baked into the idea of having preferences at all, so if you are going to use that vocabulary to talk about an ASI you must already be assuming that it will not see all thermodynamic systems as equal.
I’m not sure what you think I believe, but yeah I think we should be looking at scenarios in between the extremes.
I was giving reasons why I maintain some optimism, and maintaining optimism while reading Yudkowsky leaves me in the middle, where actions can be taken.
You might like this[0]. If I were to make a dashboard per your description my first attempt would be to replicate realtime and per capita that paper.
[0]: Chaisson, E. J. (2010). Energy rate density as a complexity metric and evolutionary driver. Complexity, 16(3), 27–40. doi:10.1002/cplx.20323
I hope the claim was normalized and inflation adjusted, otherwise it’s the same as ‘the latest protest-riot in the world’s richest country’!