I did say given similar levels of cognitive investment.
My guess is that the cognitive work put in GATO’s architectures/algorithms was much better than the specialised arms it dominates.
That or GATO was running on a much larger compute budget.
DragonGod
Karma: 705
I’m under the impression that many of the best chess players are +4 SD and beyond in IQ.
For scientists, I was thinking of that study that claimed an average IQ of around 154 yeah.
Players at a Go school not being very smart has little bearing on my point. If we found out that the average IQ of the best Go players was e.g. < 130, that would be a relevant counterargument, but the anecdote you presented doesn’t sound particularly relevant.
Out of curiosity, what IQ range does a perfect SAT score map to?
Shouldn’t you have waited for April’s Fool for this?
Simulators and Mindcrime
Quoting myself from elsewhere:
Our hard science superstars/chess superstars seem to have a mean and median g factor that’s +3 SD.
This does not seem to be the case for self made billionaires, politicians, bureaucrats or other “powerful people”.
g factor seems to have diminishing marginal returns in how much power it lets you attain?
Wait, how are you estimating Ben Bernanke and Yellen’s g factor. Your reason for guessing it seem much less compelling to me than for Gates and Bezos.
I mean inferring from SAT seems sensible. Valedictorian status is also not as sketchy. I won’t necessarily trust it, but the argument is plausible, and I expect we could later see it validated.
Our hard science superstars/chess superstars seem to have a mean and median g factor that’s +3 SD.
This does not seem to be the case for self made billionaires, politicians, bureaucrats or other “powerful people”.
g factor seems to have diminishing marginal returns in how much power it lets you attain?
No, I wasn’t directly comparing +6 SD to +3 SD.
It’s more that gains from higher g factor beyond +3 SD seem to be minimal/nonexistent in commerce, politics, etc.
Hard science research and cognitive sports are domains in which the most successful seem to be above +3 SD g factor.
I’m not compelled by the small sample size objection because there are actually domains in which the most successful are on average > +3 SD g factor. Those domains just aren’t commerce/politics/other routes of obtaining power.
As best as I can tell, your reply seems like a misunderstanding of my objection?
In general, excluding a few fields, I’m not aware that g-factor beyond +3 SD shows up in an important way in life outcomes.
The richest/most powerful/most successful aren’t generally the smartest (again, excluding a few fields).
It has been pointed out to me that the lack of such evidence of cognitive superiority may simply be because there’s not enough data on people above +3 SD g factor.
But regardless, when I look at our most capable people, they just don’t seem to be all that smart.
This is a position I might change my mind on, if we were able to get good data quantifying the gains to real world capabilities moving further out on the human spectrum.
Why I’m Sceptical of Foom
I agree that I did not specify full, working implementations of “far coordination”. There are details that I did not fill in to avoid prematurely reaching for rigour.
The kind of coordination I imagined is somewhat limited.
I guess this is an idea I may revisit and develop further sometime later. I do think there’s something sensible/useful here, but maybe my exploration of it wasn’t useful.
I had spoken with people that expected our descendants to diverge from us in ways we’d consider especially heinous, that were concerned about astronomical suffering and was persuaded by Hanson’s argument that a desire to maintain civilisational unity may prevent expansion.
So I was in that frame of mind/responding to those arguments when I wrote this.
I’m sympathetic to this reasoning. But I don’t know if it’ll prevail. I’d rather we lock in some meta values and expand to the stars than not expand at all.
“Far Coordination”
X-risk Mitigation Does Actually Require Longtermism
Well, I was arguing that we should discount in proportion to our uncertainty. And it seems you’re pretty confident that it would kill future people (and more people, and we don’t expect people in 10K years to want to die anymore than people in 100 years), so I think I would prefer to plant the 100 years landmine.
That said, expectations of technological progress, that the future would be more resilient, that they can better deal with the landmine, etc. means that in practice (outside the thought experiment), I’ll probably plant the 10K landmine as I expect less people to actually die.
In Defence of Temporal Discounting in Longtermist Ethics
I don’t buy this argument for a few reasons:
SBF met Will MacAskill in 2013 and it was following that discussion that SBF decided to earn to give
EA wasn’t a powerful or influential movement back in 2013, but quite a fringe cause.
SBF was in EA since his college days, long before his career in quantitative finance and later in crypto
SBF didn’t latch onto EA after he acquired some measure of power or when EA was a force to be reckoned with, but pretty early on. He was in a sense “homegrown” within EA.
The “SBF was a sociopath using EA to launder his reputation” is just motivated credulity IMO. There is little evidence in favour of it. It’s just something that sounds good to be true and absolves us of responsibility.
Astrid’s hypothesis is very uncredible when you consider that she doesn’t seem to be aware of SBF’s history within EA. Like what’s the angle here? There’s nothing suggesting SBF planned to enter finance as a college student before MacAskill sold him on earning to give.
Thanks for the answer!
Out of curiosity, what career did you transition to?
I think you should focus on who you want as a supervisor, rather than whether it will be in mathematics or in computer science
...
If you find the right supervisor, it is not important if they are in CS or maths or something else.
I’ll be keeping this in mind.
That is the world I’ve decided to optimise for regardless of what I actually believe timelines are
I don’t really feel like rehashing the arguments for longer timelines here (not all that relevant to my question) but it’s not the case that I have a < 10% probability on pre 2040 timelines, more that I think I can have a much larger impact on post 2040 timelines than on pre 2030, so most of my attention is directed there.
That said computational/biological anchors are a good reason for longer timelines absent foundational breakthroughs in our understanding of intelligence.
Furthermore, I suspect that intelligence is hard, that incremental progress will become harder as systems become more capable, that returns to cumulative investment in cognitive capabilities are sublinear/marginal returns to cognitive investment decay at a superlinear rate, etc.)
I specifically said a human with +12 SD g factor. I didn’t actually consider what a superintelligence that was at that level on all domains would mean, but I don’t think it would matter because of objection 4: by the time superhuman agents arrive, we would already have numerous superhuman non agentic AI, including systems specialised for planning/tactics/strategy.
You’d need to make particular claims about how a superhuman agent performs in a world of humans amplified by superhuman non agents. It’s very not obvious to me that they can win any ensuing cognitive arms race.
I am sceptical that a superhuman agent /agency would easily attain decisive cognitive superiority to the rest of civilisation.