This is a prediction I make, with “general-seeming” replaced by “more general”, and I think of this as a prediction inspired much more by CAIS than by EY/Bostrom.
I notice I’m confused. My model of CAIS predicts that there would be poor returns to building general services compared to specialised ones (though this might be more of a claim about economics than a claim about the nature of intelligence).
The following exchange is also relevant:
[-] Raiden 1y link 30
Robin, or anyone who agrees with Robin:
What evidence can you imagine would convince you that AGI would go FOOM?
Reply[-] jprwg 1y link 22
While I find Robin’s model more convincing than Eliezer’s, I’m still pretty uncertain.
That said, two pieces of evidence that would push me somewhat strongly towards the Yudkowskian view:
A fairly confident scientific consensus that the human brain is actually simple and homogeneous after all. This could perhaps be the full blank-slate version of Predictive Processing as Scott Alexander discussedrecently, or something along similar lines.
Long-run data showing AI systems gradually increasing in capability without any increase in complexity. The AGZ example here might be part of an overall trend in that direction, but as a single data point it really doesn’t say much.
Reply[-] RobinHanson 1y link 23
This seems to me a reasonable statement of the kind of evidence that would be most relevant.
EY seems to have interpreted AlphaGo Zero as strong evidence for his view in the AI-foom debate, though Hanson disagrees.
Showing excellent narrow performance *using components that look general* is extremely suggestive [of a future system that can develop lots and lots of different “narrow” expertises, using general components].
It is only broad sets of skills that are suggestive. Being very good at specific tasks is great, but doesn’t suggest much about what it will take to be good at a wide range of tasks. [...] The components look MORE general than the specific problem on which they are applied, but the question is: HOW general overall, relative to the standard of achieving human level abilities across a wide scope of tasks.
It’s somewhat hard to hash this out as an absolute rather than conditional prediction (e.g. conditional on there being breakthroughs involving some domain-specific hacks, and major labs keep working on them, they will somewhat quickly superseded by breakthroughs with general-seming architectures).
Maybe EY would be more bullish on Starcraft without imitation learning, or AlphaFold with only 1 or 2 modules (rather than 4⁄5 or 8⁄9 depending on how you count).
If people provided this as a service, they might be risk-averse (it might make sense for people to be risk-averse with their runway), which means you’d have to pay more than hourly rate/chance of winning.
This might not be a problem, as long as the market does the cool thing markets do: allowing you to find someone with a lower opportunity cost than you for doing something.
I think the question, narrowly interpreted as “what would cause me to spend more time on the object-level answering questions on LW” doesn’t capture most of the exciting things that happen when you build an economy around something. In particular, that suddenly makes various auxiliary work valuable. Examples:
Someone spending a year living off of one’s savings, learning how to summarise comment threads, with the expectation that people will pay well for this ability in the following years
A competent literature-reviewer gathering 5 friends to teach them the skill, in order to scale their reviewing capacity to earn more prize money
A college student building up a strong forecasting track-record and then being paid enough to do forecasting for a few hours each week that they can pursue their own projects instead of having to work full-time over the summer
A college student dropping out to work full-time on answering questions on LessWrong, expecting this to provide a stable funding stream for 2+ years
A professional with a stable job and family and a hard time making changes to their life-situation, taking 2 hours/week off from work to do skilled cost-effectiveness analyses, while being fairly compensated
Some people starting a “Prize VC” or “Prize market maker”, which attempts to find potential prize winners and connect them with prizes (or vice versa), while taking a cut somehow
I have an upcoming post where I describe in more detail what I think is required to make this work.
Thanks for pointing that out, the mention of YouTube might be misleading. Overall this should be read as a first-principles argument, rather than an empirical claim about YouTube in particular.
Why are you measuring it in proportion of time-until-agent-AGI and not years? If it takes 2 years from comprehensive services to agent, and most jobs are automatable within 1.5 years, that seems a lot less striking and important than the claim pre-operationalisation.
A major problem in predicting CAIS safety is to understand the order in which various services are likely to arise, in particular whether risk-reducing services are likely to come before risk-increasing services. This seems to require a lot of work in delineating various kinds of services and how they depend on each other as well as on algorithmic advancements, conceptual insights, computing power, etc. (instead of treating them as largely interchangeable or thinking that safety-relevant services will be there when we need them). Since this analysis seems very hard to do much ahead of time, I think we’ll have to put very wide error bars on any predictions of whether CAIS would be safe or unsafe, until very late in the game.
I’m broadly sympathetic to the empirical claim that we’ll develop AI services which can replace humans at most cognitively difficult jobs significantly before we develop any single superhuman AGI (one unified system that can do nearly all cognitive tasks as well as or better than any human).
I’d be interested in operationalising this further, and hearing takes on how many years “significantly before” entails.
He also adds:
One plausible mechanism is that deep learning continues to succeed on tasks where there’s lots of training data, but doesn’t learn how to reason in general ways—e.g. it could learn from court documents how to imitate lawyers well enough to replace them in most cases, without being able to understand law in the way humans do. Self-driving cars are another pertinent example. If that pattern repeats across most human professions, we might see massive societal shifts well before AI becomes dangerous in the adversarial way that’s usually discussed in the context of AI safety.
Another data-point: I love bullet points and have been sad and confused about how little they’re used in writing generally. In fact, when reading dense text, I often invest a few initial minutes in converting it to bullet points just in order to be able to read and understand it better.
Here’s PG on a related topic, sharing some of his skepticism for when bullet points are not appropriate: http://paulgraham.com/nthings.html
One should be able to think quantitatively about that, eg how many questions do you need to ask until you find out whether your extremization hurt you. I’m surprised by the suggestion that GJP didn’t do enough, unless their extremizations were frequently in the >90% range.
I did, he said a researcher mentioned it in conversation.
Good point, there’s selection pressure for things which happen to try harder to be selected for (“click me! I’m a link!“), regardless of whether they are profitable. But this is not the only pressure, and depending on what happens to a thing when it is “selected” (viewed, interviewed, etc.) this pressure can be amplified (as in OP) or countered (as in Vaniver’s comment).
more recent data suggests that the successes of the extremizing algorithm during the forecasting tournament were a fluke.
Do you have a link to this data?
I haven’t looked through your links in much detail, but wanted to reply to this:
Overall I would suggest to approach this with some intellectual humility and study existing research more, rather then try to reinvent large part of network science on LessWrong. (My guess is something like >2000 research years were spent on the topic often by quite good people.)
I either disagree or am confused. It seems good to use resources to outsource your ability to do literature reviews, distillation or extrapolation, to someone with higher comparative advantage. If the LW question feature can enable that, it will make the market for intellectual progress more efficient; and I wanted to test whether this was so.
I am not trying to reinvent network science, and I’m not that interested in the large amount of theoretical work that has been done. I am trying to 1) apply these insights to very particular problems I face (relating to forecasting and more); and 2) think about this from a cost-effectiveness perspective.
I am very happy to trade money for my time in answering these questions.
(Neither 1) nor 2) seems like something I expect the existing literature to have been very interested in. I believe this for similar reasons to those Holden Karnofsky express here.)
Seems like a sensible worry, and we did consider some version of it. My reasoning was roughly:
1) The questions feature is quite new, and if it will be very valuable, most use-cases and the proper UI haven’t been discovered yet (these can be hard to predict in advance without getting users to play around with different things and then talking to them).
No one has yet attempted to use multiple questions. So it would be valuable for the LW team and the community to experiment with that, despite possible countervailing considerations (any good experiment will have sufficient uncertainty that such considerations will always exist).
2) Questions 1⁄2, 3 and 4 are quite different, and it seems good to be able to do research on one sub-problem without taking mindshare from everyone working on any subproblem.
See this post for a good, simple mathematical description of the discrete version of the phenomenon.
Me and Ben Pace (with some help from Niki Shams) made a Guesstimate model of how much information cascades is costing science in terms of wasted grant money. The model is largely based on the excellent paper “How citation distortions create unfounded authority: analysis of a citation network” (Greenberg, 2009), which traces how an uncertain claim in biomedicine is inflated to established knowledge over a period of 15 years, and used to justify ~$10 million in grant money from the NIH (we calculated the number ourselves here).
There are many open questions about some of the inputs to our model as well as how this generalises outside of academia (or even outside of biomedicine). However, we see this as a “Jellybaby” in Douglas Hubbard’s sense—it’s a first data-point and stab at the problem which brings us from “no idea idea how big or small the costs of info-cascades are”, to at least “it is plausible though very uncertain that the costs can be on the order of magnitude of billions of dollars, yearly, in academic grant money”.
This might be an interesting pointer.
In Note-8 in the supplementary materials, Greenberg begins to quantify the problem. He defines an amplification measure for paper P as the number of citation-paths originating at P and terminating at all other papers, except for paths of length 1 flowing directly to primary data papers. The amplification density of a network is the mean amplification across its papers.
Greenberg then finds that, in the particular network analysed, you can achieve amplification density of about 1000 over a 15 year time-frame. This density grows exponentially with a doubling time of very roughly 2 years.
Here’s a quick bibliography we threw together.
Information Cascades and Rational Herding: An Annotated Bibliography and Resource Reference (Bikchandani et al. 2004). The best resource on the topic, see in particular the initial papers on the subject.
Y2K Bibliography of Experimental Economics and Social Science Information Cascades and Herd Effects (Holt, 1999. Less thorough, but catches some papers the first one misses.
“Information cascade” from Wikipedia. An excellent introduction.
“Understanding Information Cascades” from Investopedia.
Previous LessWrong posts referring to info cascades:
Information cascades, by Johnicholas, 2009
Information cascades in scientific practice, by RichardKennaway, 2009
Information cascades, LW Wiki
And then here are all the LW posts we could find that used the concept (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
. Not sure how relevant they are, but might be useful in orienting around the concept.