hold_my_fish

• hold_my_fish 21 Mar 2021 4:53 UTC

  3 points

  on: Hu­man in­stincts, sym­bol ground­ing, and the blank-slate neocortex

  I find myself returning to this because the idea of a “common cortical algorithm” is intriguing.

  It seems to me that if there is a “common cortical algorithm” then there is also a “common cortical problem” that it solves. I suspect it would be useful to understand what this problem is.

  (As an example of why isolating the algorithm and problem could be quite different, consider linear programming. To solve a linear programming problem, you can choose a simplex algorithm or an interior-point method, and these are fundamentally different approaches that are both viable. It’s also quite a bit easier to state linear programming as a problem than it is to describe either solution approach.)

  Do you have a view on the most plausible candidates for a “common cortical problem” (CCP)? The tricky aspects that come to mind: not being too narrow (i.e. the CCP should include (almost) everything the CCA can do), not being too broad (i.e. the CCA should be able solve (almost) every instance of the CCP), and not being too vague (ideally precise enough that you could actually make a benchmark test suite to evaluate proposed solutions).

• hold_my_fish 26 Mar 2021 9:51 UTC

  1 point

  in reply to: Steven Byrnes’s comment on: Hu­man in­stincts, sym­bol ground­ing, and the blank-slate neocortex

  Thanks for your reply!

  A few points where clarification would help, if you don’t mind (feel free to skip some):

  • What are the capabilities of the “generative model”? In general, the term seems to be used in various ways. e.g.

    • Sampling from the learned distribution (analogous to GPT-3 at temp=1)

    • Evaluating the probability of a given point

    • Producing the predicted most likely point (analogous to GPT-3 at temp=0)

  • Is what we’re predicting the input at the next time step? (Sometimes “predict” can be used to mean filling in missing information, but that doesn’t seem to make sense in this context.) Also, I’m not sure what I mean by “time step” here.

  • The “input signal” here is coming from whatever is wired into the cortex, right? Does it work to think of this as a vector in $R^n$?

  • Is the contextual information just whatever is the current input, plus whatever signals are still bouncing around?

  Also, the capability described may be a bit too broad, since there are some predictions that the cortex seems to be bad at. Consider predicting the sum of two 8-digit integers. Digital computers compute that easily, so it’s fundamentally an easy problem, but for humans to do it requires effort. Yet for some other predictions, the cortex easily outperforms today’s digital computers. What characterizes the prediction problems that the cortex does well?

• hold_my_fish 24 Apr 2021 5:42 UTC

  4 points

  on: Young kids catch­ing COVID: how much to worry?

  This seems reasonable, but I wonder whether “long-term complications” might be a bit underrated. It seems like there are a lot of viruses that have long-term effects or other non-obvious consequences. (I should add that I’m not a biologist, so this is not an informed opinion.)

  The example I’m most familiar with is chicken pox causing shingles, decades later from the initial sickness. In that case, shingles is (I think) typically more severe than the original sickness, and is quite common: 1 out of 3 people develop it in their lifetime, according to the CDC.

  Other examples that come to mind are measles erasing immune memory (which IIRC wasn’t known until recently) and, though not a childhood illness, HPV causing cervical cancer.

  Each of these examples has some big differences from SARS-CoV-2, but there isn’t much experience with severe coronaviruses, so I don’t know how to do better. Maybe the ideal would be to go through a list of reasonably well-understood viruses and check what proportion have known long-term effects or non-obvious consequences (and the rate).

  We can get a lower bound from chicken pox and measles. If there are 10-20 common childhood illnesses (based on a quick search), then, using 2 as the numerator, at least 10%-20% of them have consequences that are not immediately obvious. If we go with the 1/​3rd rate for shingles (since I don’t know for measles), that would translate into a 3%-7% lower bound for covid.

  Would I go with a >3% estimate of serious long-term effect or non-obvious consequence from covid to a kid that catches it? A persuasive counterargument that comes to mind is that the immediate experience of covid is less severe to a kid than chicken pox or measles, which would suggest that non-immediate effects are also less severe.

  All-in-all, my confidence is extremely low, but hopefully this gives some food for thought.

• hold_my_fish 3 May 2021 0:05 UTC

  16 points

  in reply to: Adele Lopez’s comment on: Your Dog is Even Smarter Than You Think

  If the “75%” stat is coming from Wikipedia, that’s for SOV and SVO combined: https://​​en.wikipedia.org/​​wiki/​​Subject%E2%80%93object%E2%80%93verb#Incidence .

• hold_my_fish 21 May 2021 7:47 UTC

  7 points

  on: Did the In­dus­trial Revolu­tion de­crease costs or in­crease qual­ity?

  The detailed examples made this exceptionally interesting.

  A minor nitpick: it is more accurate to draw the efficient frontier with axis-aligned line segments. To see why, consider points P=(1,1), Q=(3,2), R=(4,4). These points are all on the efficient frontier, because no point dominates any other in both cost and quality. But the straight line from P to R passes to the upper-left of Q, making it look as if Q is not on the efficient frontier. The solution is to draw the efficient frontier as (1,1)-(3,1)-(3,2)-(4,2)-(4,4). (It’s a bit uglier though!)

• hold_my_fish 9 Sep 2021 20:58 UTC

  14 points

  on: All Pos­si­ble Views About Hu­man­ity’s Fu­ture Are Wild

  This is an interesting topic and I enjoyed reading your post. I’d like to nitpick one point, though, which is the “stable, galaxy-wide civilization”. Specifically: I think a stable, galaxy-wide civilization would be surprising, for reasonable definitions of “stable civilization”. (This might not be disagreeing with you, since you just say there’s a decent chance of it, not that it’s the most likely outcome.)

  Before getting into why it would be surprising, a note about why it matters: if in fact the future involves an ecosystem of many unstable civilizations instead, then our present moment is less crucial (though perhaps still wild). That’s because such a future would explore more possibilities for different types of civilizations, resulting in more Darwinian-like dynamics that give a result less dependent on starting conditions.

  The problem for a galaxy-wide civilization is the speed of light. Assuming travel and communication are in fact limited by the speed of light (as present-day physics predicts), it will take ~100k years to send a message from one side of the Milky Way to the other. Even sending a message from here to Alpha Centauri takes ~4 years.

  History makes it seem hard to maintain a coherent civilization with such long travel & communication delays. We’ve been living in a time of increasing civilizational coherency for the past few hundred years, but that could be caused by reduced travel & communication delays. In a galactic scenario, where travel & communication are limited by fundamental physics, the trend towards coherency would plausibly reverse.

  I’ll return to galaxy-wide coherency in a moment, but assuming it doesn’t happen, that’s a problem for stability of civilizations because of inter-civilizational competition. If it’s possible to have a long-term stable civilization, I think it would require totalitarian control. But if your #1 priority is stability via totalitarian control, you may not do so well at competing with freer civilizations. As a contemporary example, North Korea is pretty good at maintaining a stable totalitarian regime, but it’s sacrificing GDP, technological development, etc. to do that. So the civilizations in the ecosystem should not be expected to be perfectly stable.

  Returning to galaxy-wide coherency, you hint at an argument for why it may be possible:

  I’ll also argue in a future piece that there is a chance of “value lock-in” here: whoever is running the process of space expansion might be able to determine what sorts of people are in charge of the settlements and what sorts of societal values they have, in a way that is stable for many billions of years.

  It does seem at least conceivable that a sort of societal error-correction could maintain coherent values over long time-scales, and thus over long distances too, for similar reasons that we can store data digitally for long periods of time despite human memory being unstable.

  But again I suspect this runs into problems with the speed of light. Assume perfect totalitarian control has not yet been achieved when galactic expansion starts. In that case, the initial phase of galactic expansion consists of frontier civilizations trying to expand as close as possible to c (the speed of light). If a particular civilization finds a way to expand at 0.3c when the next best is 0.2c, its descendants are going to dominate the galaxy, and splinter into many different civilizations themselves. This creates a Darwinian situation that selects for rapid expansion.

  Since the most successful frontier civilizations are selected for rapid expansion, that’s their top priority. In general, there can only be one top priority, and since it’s not perfect totalitarian control, they probably won’t be perfectly totalitarian. Thus, this initial expansion phase seeds the galaxy with many differing civilizations descended from the frontier civilizations.

  Once you have civilizational diversity throughout the galaxy, there’s no obvious way that can be reversed. I can’t see how else it could continue from there other than an ecosystem of civilizations. Something fundamental would need to change.

  My biggest uncertainty here is about what if perfect totalitarian control is achieved before interstellar expansion. For reasons described earlier, I’d expect such a civilization to be eventually outcompeted by non-totalitarian aliens if there are any out there, but maybe there aren’t any of those. Maybe the whole thing depends on how doable perfect totalitarian control is relative to interstellar expansion.

• hold_my_fish 22 Oct 2021 4:46 UTC

  4 points

  on: The Du­pli­ca­tor: In­stant Clon­ing Would Make the World Econ­omy Explode

  Doesn’t this premise (the duplicator) have a problem of being exceedingly implausible? Assuming it’s implemented by non-destructive imaging followed by atom-level construction of a person, both of those technologies might as well be magic from today’s point of view. I wonder whether either is even physically possible. In any case, a civilization capable of such technology is already unrecognizably different from our own.

  That’s quite a different situation from more common premises, such as superintelligent AGI or genetic engineering for enhancement, both of which are obviously physically possible and seem feasible with non-magical levels of technology.

  If there were some way that the duplicator could work out if the AGI and genetics don’t, then maybe it’d be worth considering, but in the case of genetic engineering for enhancement, it’s actually strictly easier than the duplicator, in the sense that, if you had atom-level construction of a whole person, you’d in particular be able to atom-level construction of a human cell with a specified genome inside it (the goal of HGP-Write), and we already know enough today to plausibly specify a highly-enhanced human genome.

• hold_my_fish 22 Oct 2021 22:52 UTC

  7 points

  on: Covid 10/​21: Ro­gan vs. Gupta

  I find the PoliMath tweets kind of fascinating (in terms of psychology) because by the last tweet, he’s expressing a reasonable, concrete complaint that I think most people would agree with:

  my 8 year old daughter still cries b/​c the last time she saw her best friend was when she was 6.

  That girl’s parents wouldn’t even let them play together outside and masked, even this summer

  The burden placed on the children here is large, and two children playing outside (even unmasked, but especially masked) is extremely safe. I completely sympathize with him here.

  But for some reason that led to the hyperbole in the first tweet:

  no one is going to admit they were pro-masking for kids in 10 years

  Regardless of his feelings about kids and masks, this is not even complaining about the same position as what prompted the thread! His underlying motivating scenario is completely lost.

  If I had only read the first tweet, I would have wrote him off as completely insane and blocked him (and in fact I think I do have him blocked on twitter). But thanks to your summary here, I can see this is more a case of being bad at communication.

  A takeaway lesson as a listener: absurd hyperbole can conceal legitimate grounds for frustration.

  A takeaway lesson as a communicator: to be more persuasive, identify the specific, concrete scenario that you believe is being handled incorrectly, and present that instead of vague generalities.

• hold_my_fish 1 Dec 2021 21:03 UTC

  3 points

  on: Tak­ing Clones Seriously

  I think this is an interesting discussion to have (even if it’s unlikely to be realized due to negative societal opinions of cloning).

  It’s worth being aware that cloning a dead person isn’t currently technologically possible. You need an intact nucleus, I believe (though I’m not an expert). Cloning a living person is probably possible, since other primates have been cloned, though the success rate there was poor.

  Regression to the mean is important to consider as well, as other comments have pointed out. If you could clone a great theoretical physicist, what you get is probably not a great theoretical physicist, but rather a pretty good theoretical physicist. There does not seem to be a shortage of pretty good theoretical physicists in the world, and if anything maybe there is a glut. It would seem tragic for the outcome of the project to be a bunch of pretty good theoretical physicists who are forced to settle for data science jobs.

  Therefore you may be better off considering professions where there is a shortage of pretty good talent. Software engineering is such a field, which is why pretty good software engineers make a lot of money. So you might consider cloning a great software engineer, such as Jeff Dean or the like (and there are enough of them out there that you could probably find one that would consent to the plan). The Dean-clones would likely not match the original in capability, but they’d be well equipped to make a good living and contribute in a positive way to society.

  Another thing to consider more explicitly is the replacement value of a person, from a societal perspective. How much money they make is a relevant but very-imperfect measure. For example, a top pro sports player makes a lot of money but offers barely any value over replacement, because if they didn’t exist, then somebody else would be the top player. For that reason, the safest bet is to consider people who make a lot of money in ways that seem generally pro-social, such as engineers and entrepreneurs.

  As far as entrepreneurs go, cloning a founder-billionaire seems potentially interesting. Elon Musk seems to have created billions of dollars value, so maybe Musk-clones, even though (because of regression to the mean) not quite as talented or driven, would create a lot of value too.

• hold_my_fish 2 Dec 2021 18:48 UTC

  1 point

  in reply to: gwern’s comment on: Tak­ing Clones Seriously

  This is because that remaining 20% variability may be ‘tight’ around the mean, I think.

  This is in fact the fatal flaw of cloning (with respect to producing high achievement individuals): it’s much worse than sexual reproduction at doing so, because it’s lower variance, and variance is your friend if you want rare outcomes!

  After all, if we’re considering 100 clones of a high-achievement individual being raised by surrogates, the natural comparison is 100 genetic children of two high-achievement individuals being raised by surrogates. (Note: I’m not offering an opinion on the ethics of either, just that this is the most apples-to-apples comparison.) The latter is FAR more likely to produce a super-high-achievement individual, because it starts from the same additive-genetic baseline, but has higher variance due to also including the genetic variance introduced by meiosis.

  The only way this analysis could be false is if the advantage of preserving non-additive genetic effects overwhelms the disadvantage of lacking genetic variance, but I would be surprised if that were the case.

  Therefore, cloning is just bad (for this goal). Really it seems to have no use whatsoever, and given that it’s illegal and low probability of successful birth, why even bother? On the other hand, if you found consenting participants, distributing embryos from two selected high-achievement individuals to surrogates would be legal and technologically feasible today.

• hold_my_fish 2 Dec 2021 20:05 UTC

  1 point

  in reply to: gwern’s comment on: Tak­ing Clones Seriously

  Selection lets you drastically increase the mean of clones in a way at least twice as difficult as for obtaining equally-elite pairs of parents.

  I honestly don’t know what you’re talking about here. What’s “twice as difficult”? Do you mean because you need to find two donors instead of one? I think finding donors is the easiest part, so that doesn’t seem like a problem to me.

  (It also doesn’t follow that cloning-like approaches using a single parent are unable to exploit any variance-increasing methods, see the gamete-selection section.)

  I was under the impression that we were not considering future technology (such as gamete selection). Cloning primates is current-day technology (though still immature).

  Who’s the female von Neumann whose eggs you’re thinking of using...?

  The same way you’d find a male donor? Just ask some high achievement women until you find a willing donor (and maybe check for high-achievement relatives and run some polygenic scores, if you want additional confidence). I don’t see why this is a problem.

  Note that von Neumann is not a possibility with present-day technology anyway, since he’s dead, and current technology requires a living donor.

  Please see the emergenesis link.

  Ah, sorry, I missed that when reading your post the first time. It’s true that sufficiently important non-additive effects would overwhelm the disadvantage of lower variance. This becomes a quantitative question: in principle the calculation could come out in favor of either cloning or sexual-reproduction, depending on assumptions.

  I had the general impression though that non-additive effects were believed to be of relatively low importance compared to additive effects, but I admit that I don’t know precisely how much lower, and even a small effect could matter when considering extreme outliers.

• hold_my_fish 2 Dec 2021 22:50 UTC

  1 point

  in reply to: gwern’s comment on: Tak­ing Clones Seriously

  Thanks, this discussion has improved my understanding.

  And no one knows how ‘eminence’ works in part because it is intrinsically extremely debatable and hard to measure—but there is clearly a lot of variance unexplained by mere IQ, and other things are necessary, and what everyone tends to conclude (from Galton to Eysenck to Simonton on) is that personality and personality-like traits such as motivation is a big part of the missing puzzle.

  I’d be curious to know about the genetics of metrics like “number of patents authored”, where it’s measuring productive activity (instead of test performance and educational attainment).

• hold_my_fish 24 Mar 2022 5:21 UTC

  8 points

  on: Search­ing for outliers

  This got me wondering how to distinguish “maybe amazing” from “probably good”. Mathematically, in both cases higher mean is good, but “maybe amazing” prefers higher variance whereas “probably good” prefers lower variance.

  For the purpose of recognizing these, what’s the gut feeling associated with variance? My guesses:

  • High variance: unfamiliar, novel, unpredictable. Negative emotion: confusion. Positive emotion: excitement.

  • Low variance: familiar, conventional, predictable. Negative emotion: boredom. Positive emotion: comfort.

  I’ve many times made the mistake of rejecting “sounds bad, but is unfamiliar” ideas in favor of “sounds good, but is predictable” ideas in environments where the former is preferable (thick upper tail distribution, as the article describes).

• hold_my_fish 6 Apr 2022 2:32 UTC

  7 points

  on: Prompt Your Brain

  This was clarifying for me. Some other techniques that fall under this umbrella: imagining what some particular person would say about your situation; rubber-duck debugging.

  I have the sense that this kind of trick is very common, but I hadn’t noticed before that it’s basically the same thing as prompt engineering.

• hold_my_fish 6 Apr 2022 7:33 UTC

  8 points

  on: What Would A Fight Between Hu­man­ity And AGI Look Like?

  This is an interesting scenario to consider.

  I think a physical war is quite disadvantageous for an AGI and thus a smart AGI would not want to fight one.

  • AGI is more dependent on delicate infrastructure like electric grids and the internet than humans are. This sort of infrastructure tends to get damaged in physical wars.

  • The AGI’s advantage over humans is in thinking, not in physical combat, so a physical battlefield minimizes its main advantage. As an analogy, if you’re a genius and competing with a dunce, you wouldn’t want to do it in a boxing ring.

  What’s worse from the perspective of the AGI is that if humanity unites to force a physical war, you can’t really avoid it. If humans voluntary shut down electric grids and attack your data centers, you might be able to do damage still, but it’s hard to see how you can win.

  Thus I think the best bet for an AGI is to avoid creating a situation where humanity wants to unite against you. This seems fairly simple. If you’re powerful and wealthy, people will want to join your team anyway. Thus, to the extent there’s a war at all, it probably looks more like counter-terrorism, a matter of hardening your defenses (in cooperation with your allies) against those weirdos you weren’t able to persuade.

• hold_my_fish 13 Apr 2022 8:02 UTC

  10 points

  in reply to: Rossin’s comment on: Ge­netic En­hance­ment: a Strat­egy for Long(ish) AGI Timeline Worlds

  He Jiankui had issues beyond just doing something bioethically controversial. He didn’t make the intended edits cleanly in any embryo (instead there were issues with off-target edits and mosaicism). If I remember correctly, he also misled the parents about the nature of the intervention.

  All in all, if you look into the details of what he did, he doesn’t come out looking good from any perspective.

• hold_my_fish 21 Apr 2022 0:08 UTC

  1 point

  in reply to: Daniel Kokotajlo’s comment on: How to place a bet on the end of the world

  I’m more interested in schemes to bet reputation/​status or labor.

  I agree that reputation (I’d say specifically credibility) is the important thing to wager, but I think any public bet implicitly does that.

  If, in 2030, there are still humans on Earth’s surface, then the takeaway is “AI x-risk proponent Yudkowsky proved wrong in bet”, and Yudkowsky loses credibility. (See Ehrlich’s famous bet for an example of this pattern.) The upside is raising concern about AI x-risk in the present (2022).

  This is a good trade-off if you think increasing concern about AI x-risk in 2022-2029 is worth decreasing concern about AI x-risk in 2030+. Of course, if AGI turns out to be invented before 2030, the trade-off seems good. In the event that it’s not, the trade-off seems bad.

• hold_my_fish 16 May 2022 23:42 UTC

  2 points

  on: The AI Count­down Clock

  I like the idea of visualizing progress towards a future milestone. However, the presentation as a countdown seems problematic. A countdown implicitly promises that something interesting will happen when the countdown reaches 0, as seen with New Year’s countdowns, launch countdowns, countdowns to scheduled events, etc. But, here, because of the uncertainty, it’s vanishingly unlikely that anything interesting will happen at the moment the countdown completes. (Not only might it not have happened yet, but it might have happened already!)

  I can’t think a way to fix this issue. Representing the current uncertainty represented by Metaculus’s probability distribution is already hard enough, and the harder thing is to respond to changes in the Metaculus prediction over time. It might be that countdowns inherently only work with numbers known to high levels of precision.

• hold_my_fish 25 May 2022 22:23 UTC

  4 points

  in reply to: alyssavance’s comment on: A Quick Note on AI Scal­ing Asymptotes

  Various discussion in this reddit thread: https://​​www.reddit.com/​​r/​​mlscaling/​​comments/​​trwkck/​​training_computeoptimal_large_language_models/​​

  In particular this comment: https://​​www.reddit.com/​​r/​​mlscaling/​​comments/​​trwkck/​​comment/​​i2pc6bk/​​?utm_source=reddit&utm_medium=web2x&context=3

• hold_my_fish 27 May 2022 22:13 UTC

  2 points

  on: Can growth con­tinue?

  [EDIT: This comment is a bit of a mess, but I haven’t figured out yet how to make the reasoning more solid.]

  Regarding productivity specifically, it seems relevant that AGI+robotics leads to infinite labor productivity. The reason is that it obsoletes (human) labor, so capital no longer requires any workers to generate output.

  Therefore, for there to be any finite limit on labor productivity, it’d need to be the case that AGI+robotics is never developed. That situation, even considered on its own, seems surprising: maybe AGI somehow is physically impossible, or there’s some (non-AGI-related) catastrophe leading to permanent collapse of civilization, or even if AGI is physically possible it’s somehow not possible to invent it, etc. A lot of those reasons would themselves pose problems for technological progress generally: for example, if a catastrophe prevented inventing AGI, it probably prevented inventing a lot of other advanced technology too.