Not sure what’s realistic to do here.
Could write “FLOPS-seconds” maybe, which, although a bit wordy, resembles the common usage of “kilowatt-hours” instead of a more directly joule-based unit.
Here’s something odd that I noticed in one of the examples in the blogpost (https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html).
The question is the one that in part reads “the variance of the first n natural numbers is 10”. The model’s output states, without any reasoning, that this variance is equal to (n^2 − 1)/12, which is correct. Since no reasoning was used, I think it’s safe to assume that the model memorized this formula.
This is not a formula that a random math student would be expected to have memorized. (Anecdotally, I have a mathematics degree and don’t know it.) Because of that, I’d expect that a typical (human) solver would need to derive the formula on the spot. It also strikes me as the sort of knowledge that would be unlikely to matter outside a contest, exam, etc.
That all leads me to think that the model might be over-fitting somewhat to contest/exam/etc.-style questions. By that I mean that it might be memorizing facts that are useful when answering such questions but are not useful when doing math more broadly.
To be clear, there are other aspects of the model output, here and in other questions, that seem genuinely impressive in terms of reasoning ability. But the headline accuracy rate might be inflated by memorization.
Regarding the cost, I’d expect the road to AGI to deliver intermediate technologies that reduce the cost of writing provably secure code. In particular, I’d expect Copilot-like code generation systems to stay close to the leading edge of AI technology, if nothing else then because of their potential to deliver massive economic value.
Imagine some future version of Copilot that, in addition to generating code for you, also proves properties of the generated code. There might be reasons to do that beyond security: the requirement to provide specs and proofs in addition to code might make Copilot-like systems more consistent at generating correct programs.
While I can’t quantify, I think secure computer systems would help a lot by limiting the options of an AI attempting malicious actions.
Imagine a near-AGI system with uneven capabilities compared to humans. Maybe its GPT-like (natural language interaction) and Copilot-like (code understanding and generation) capabilities pass humans but robotics lags behind. More generally, in virtual domains, especially those involving strings of characters, it’s superior, but elsewhere it’s inferior. This is all easy to imagine because it’s just assuming the relative balance of capabilities remains similar to what it is today.
Such a near-AGI system would presumably be superhuman at cyber-attacking. After all, that plays to its strengths. It’d be great at both finding new vulnerabilities and exploiting known ones. Having impenetrable cyber-defenses would neutralize this advantage.
Could the near-AGI system improve its robotics capabilities to gain an advantage in the physical world too? Probably, but that might take a significant amount of time. Doing things in the physical world is hard. No matter how smart you are, your mental model of the world is a simplification of true physical reality, so you will need to run experiments, which takes time and resources. That’s unlike AlphaZero, for example, which can exceed human capabilities quickly because its experiments (self-play games) take place in a perfectly accurate simulation.
One last thing to consider is that provable security has the nice property that you can make progress on it without knowing the nature of the AI you’ll be up against. Having robust cyber-defense will help whether AIs turn out to be deep-learning-based or something else entirely. That makes it in some sense a safe bet, even though it obviously can’t solve AGI risk on its own.
If AGI becomes available then it would replace the labor of AI researchers too. (At least it would if we assume that AGI is cheaper to operate than a human. But that seems almost certain, since humans are very expensive.)
In any case I’m not sure it really makes sense to talk about the productivity of people who aren’t employed. I’m considering the economy-wide stat here.
Division by zero is undefined and it not guaranteed to correspond to +infinity in all context. In this context there might be a difference whether we are talking about the limit when labor apporaches 0 and when labour is completely absent.
True, and in this context the limiting value when approaching from above is certainly the appropriate interpretation. After all, we’re talking about a gradual transition from current use of labor (which is positive) to zero use of labor. If the infinity is still bothersome, imagine somebody is paid to spend 1 second pushing the “start the AGI” button, in which case labor productivity is a gazillion (some enormous finite number) instead of infinity.
If you are benetting from a gift that you don’t have to work for at all it not like you “work at infinite efficiency” for it.
You seem to be arguing against the definition of labor productivity here. I think though that I’m using the most common definition. If you consider for example Our World In Data’s “productivity per hour worked” graph, it uses essentially the same definition that I’m using.
It could also be understood that hitting the AI condition means that human labor productivity becomes 0.
I don’t agree with this. Using the formula “labor productivity = output volume / labor input use” (which I grabbed from Wikipedia, which is maybe not the best source, but it seems right to me), if “labor input use” is zero and “output volume” is positive, then “labor productivity” is +infinity.
[EDIT: This comment is a bit of a mess, but I haven’t figured out yet how to make the reasoning more solid.]
Regarding productivity specifically, it seems relevant that AGI+robotics leads to infinite labor productivity. The reason is that it obsoletes (human) labor, so capital no longer requires any workers to generate output.
Therefore, for there to be any finite limit on labor productivity, it’d need to be the case that AGI+robotics is never developed. That situation, even considered on its own, seems surprising: maybe AGI somehow is physically impossible, or there’s some (non-AGI-related) catastrophe leading to permanent collapse of civilization, or even if AGI is physically possible it’s somehow not possible to invent it, etc. A lot of those reasons would themselves pose problems for technological progress generally: for example, if a catastrophe prevented inventing AGI, it probably prevented inventing a lot of other advanced technology too.
Various discussion in this reddit thread: https://www.reddit.com/r/mlscaling/comments/trwkck/training_computeoptimal_large_language_models/
In particular this comment: https://www.reddit.com/r/mlscaling/comments/trwkck/comment/i2pc6bk/?utm_source=reddit&utm_medium=web2x&context=3
I like the idea of visualizing progress towards a future milestone. However, the presentation as a countdown seems problematic. A countdown implicitly promises that something interesting will happen when the countdown reaches 0, as seen with New Year’s countdowns, launch countdowns, countdowns to scheduled events, etc. But, here, because of the uncertainty, it’s vanishingly unlikely that anything interesting will happen at the moment the countdown completes. (Not only might it not have happened yet, but it might have happened already!)
I can’t think a way to fix this issue. Representing the current uncertainty represented by Metaculus’s probability distribution is already hard enough, and the harder thing is to respond to changes in the Metaculus prediction over time. It might be that countdowns inherently only work with numbers known to high levels of precision.
I’m more interested in schemes to bet reputation/status or labor.
I agree that reputation (I’d say specifically credibility) is the important thing to wager, but I think any public bet implicitly does that.
If, in 2030, there are still humans on Earth’s surface, then the takeaway is “AI x-risk proponent Yudkowsky proved wrong in bet”, and Yudkowsky loses credibility. (See Ehrlich’s famous bet for an example of this pattern.) The upside is raising concern about AI x-risk in the present (2022).
This is a good trade-off if you think increasing concern about AI x-risk in 2022-2029 is worth decreasing concern about AI x-risk in 2030+. Of course, if AGI turns out to be invented before 2030, the trade-off seems good. In the event that it’s not, the trade-off seems bad.
He Jiankui had issues beyond just doing something bioethically controversial. He didn’t make the intended edits cleanly in any embryo (instead there were issues with off-target edits and mosaicism). If I remember correctly, he also misled the parents about the nature of the intervention.
All in all, if you look into the details of what he did, he doesn’t come out looking good from any perspective.
This is an interesting scenario to consider.
I think a physical war is quite disadvantageous for an AGI and thus a smart AGI would not want to fight one.
AGI is more dependent on delicate infrastructure like electric grids and the internet than humans are. This sort of infrastructure tends to get damaged in physical wars.
The AGI’s advantage over humans is in thinking, not in physical combat, so a physical battlefield minimizes its main advantage. As an analogy, if you’re a genius and competing with a dunce, you wouldn’t want to do it in a boxing ring.
What’s worse from the perspective of the AGI is that if humanity unites to force a physical war, you can’t really avoid it. If humans voluntary shut down electric grids and attack your data centers, you might be able to do damage still, but it’s hard to see how you can win.
Thus I think the best bet for an AGI is to avoid creating a situation where humanity wants to unite against you. This seems fairly simple. If you’re powerful and wealthy, people will want to join your team anyway. Thus, to the extent there’s a war at all, it probably looks more like counter-terrorism, a matter of hardening your defenses (in cooperation with your allies) against those weirdos you weren’t able to persuade.
This was clarifying for me. Some other techniques that fall under this umbrella: imagining what some particular person would say about your situation; rubber-duck debugging.
I have the sense that this kind of trick is very common, but I hadn’t noticed before that it’s basically the same thing as prompt engineering.
This got me wondering how to distinguish “maybe amazing” from “probably good”. Mathematically, in both cases higher mean is good, but “maybe amazing” prefers higher variance whereas “probably good” prefers lower variance.
For the purpose of recognizing these, what’s the gut feeling associated with variance? My guesses:
High variance: unfamiliar, novel, unpredictable. Negative emotion: confusion. Positive emotion: excitement.
Low variance: familiar, conventional, predictable. Negative emotion: boredom. Positive emotion: comfort.
I’ve many times made the mistake of rejecting “sounds bad, but is unfamiliar” ideas in favor of “sounds good, but is predictable” ideas in environments where the former is preferable (thick upper tail distribution, as the article describes).
Thanks, this discussion has improved my understanding.
And no one knows how ‘eminence’ works in part because it is intrinsically extremely debatable and hard to measure—but there is clearly a lot of variance unexplained by mere IQ, and other things are necessary, and what everyone tends to conclude (from Galton to Eysenck to Simonton on) is that personality and personality-like traits such as motivation is a big part of the missing puzzle.
I’d be curious to know about the genetics of metrics like “number of patents authored”, where it’s measuring productive activity (instead of test performance and educational attainment).
Selection lets you drastically increase the mean of clones in a way at least twice as difficult as for obtaining equally-elite pairs of parents.
I honestly don’t know what you’re talking about here. What’s “twice as difficult”? Do you mean because you need to find two donors instead of one? I think finding donors is the easiest part, so that doesn’t seem like a problem to me.
(It also doesn’t follow that cloning-like approaches using a single parent are unable to exploit any variance-increasing methods, see the gamete-selection section.)
I was under the impression that we were not considering future technology (such as gamete selection). Cloning primates is current-day technology (though still immature).
Who’s the female von Neumann whose eggs you’re thinking of using...?
The same way you’d find a male donor? Just ask some high achievement women until you find a willing donor (and maybe check for high-achievement relatives and run some polygenic scores, if you want additional confidence). I don’t see why this is a problem.
Note that von Neumann is not a possibility with present-day technology anyway, since he’s dead, and current technology requires a living donor.
Please see the emergenesis link.
Ah, sorry, I missed that when reading your post the first time. It’s true that sufficiently important non-additive effects would overwhelm the disadvantage of lower variance. This becomes a quantitative question: in principle the calculation could come out in favor of either cloning or sexual-reproduction, depending on assumptions.
I had the general impression though that non-additive effects were believed to be of relatively low importance compared to additive effects, but I admit that I don’t know precisely how much lower, and even a small effect could matter when considering extreme outliers.
This is because that remaining 20% variability may be ‘tight’ around the mean, I think.
This is in fact the fatal flaw of cloning (with respect to producing high achievement individuals): it’s much worse than sexual reproduction at doing so, because it’s lower variance, and variance is your friend if you want rare outcomes!
After all, if we’re considering 100 clones of a high-achievement individual being raised by surrogates, the natural comparison is 100 genetic children of two high-achievement individuals being raised by surrogates. (Note: I’m not offering an opinion on the ethics of either, just that this is the most apples-to-apples comparison.) The latter is FAR more likely to produce a super-high-achievement individual, because it starts from the same additive-genetic baseline, but has higher variance due to also including the genetic variance introduced by meiosis.
The only way this analysis could be false is if the advantage of preserving non-additive genetic effects overwhelms the disadvantage of lacking genetic variance, but I would be surprised if that were the case.
Therefore, cloning is just bad (for this goal). Really it seems to have no use whatsoever, and given that it’s illegal and low probability of successful birth, why even bother? On the other hand, if you found consenting participants, distributing embryos from two selected high-achievement individuals to surrogates would be legal and technologically feasible today.
I think this is an interesting discussion to have (even if it’s unlikely to be realized due to negative societal opinions of cloning).
It’s worth being aware that cloning a dead person isn’t currently technologically possible. You need an intact nucleus, I believe (though I’m not an expert). Cloning a living person is probably possible, since other primates have been cloned, though the success rate there was poor.
Regression to the mean is important to consider as well, as other comments have pointed out. If you could clone a great theoretical physicist, what you get is probably not a great theoretical physicist, but rather a pretty good theoretical physicist. There does not seem to be a shortage of pretty good theoretical physicists in the world, and if anything maybe there is a glut. It would seem tragic for the outcome of the project to be a bunch of pretty good theoretical physicists who are forced to settle for data science jobs.
Therefore you may be better off considering professions where there is a shortage of pretty good talent. Software engineering is such a field, which is why pretty good software engineers make a lot of money. So you might consider cloning a great software engineer, such as Jeff Dean or the like (and there are enough of them out there that you could probably find one that would consent to the plan). The Dean-clones would likely not match the original in capability, but they’d be well equipped to make a good living and contribute in a positive way to society.
Another thing to consider more explicitly is the replacement value of a person, from a societal perspective. How much money they make is a relevant but very-imperfect measure. For example, a top pro sports player makes a lot of money but offers barely any value over replacement, because if they didn’t exist, then somebody else would be the top player. For that reason, the safest bet is to consider people who make a lot of money in ways that seem generally pro-social, such as engineers and entrepreneurs.
As far as entrepreneurs go, cloning a founder-billionaire seems potentially interesting. Elon Musk seems to have created billions of dollars value, so maybe Musk-clones, even though (because of regression to the mean) not quite as talented or driven, would create a lot of value too.
I find the PoliMath tweets kind of fascinating (in terms of psychology) because by the last tweet, he’s expressing a reasonable, concrete complaint that I think most people would agree with:
my 8 year old daughter still cries b/c the last time she saw her best friend was when she was 6.
That girl’s parents wouldn’t even let them play together outside and masked, even this summer
The burden placed on the children here is large, and two children playing outside (even unmasked, but especially masked) is extremely safe. I completely sympathize with him here.
But for some reason that led to the hyperbole in the first tweet:
no one is going to admit they were pro-masking for kids in 10 years
Regardless of his feelings about kids and masks, this is not even complaining about the same position as what prompted the thread! His underlying motivating scenario is completely lost.
If I had only read the first tweet, I would have wrote him off as completely insane and blocked him (and in fact I think I do have him blocked on twitter). But thanks to your summary here, I can see this is more a case of being bad at communication.
A takeaway lesson as a listener: absurd hyperbole can conceal legitimate grounds for frustration.
A takeaway lesson as a communicator: to be more persuasive, identify the specific, concrete scenario that you believe is being handled incorrectly, and present that instead of vague generalities.