tailcalled

Karma: 7,802

tailcalled Apr 3, 2025, 5:56 AM
2 points
0
in reply to: Davidmanheim’s comment on: Is instrumental convergence a thing for virtue-driven agents?
The methods for converting policies to utility functions assume no systematic errors, which doesn’t seem compatible with varying the intelligence levels.

tailcalled Apr 3, 2025, 5:18 AM
2 points
0
in reply to: mattmacdermott’s comment on: Is instrumental convergence a thing for virtue-driven agents?
This.
In particular imagine if the state space of the MDP factors into three variables x, y and z, and the agent has a bunch of actions with complicated influence on x, y and z but also just some actions that override y directly with a given value.
In some such MDPs, you might want a policy that does nothing other than copy a specific function of x to y. This policy could easily be seen as a virtue, e.g. if x is some type of event and y is some logging or broadcasting input, then it would be a sort of information-sharing virtue.
While there are certain circumstances where consequentialism can specify this virtue, it’s quite difficult to do in general. (E.g. you can’t just minimize the difference between f(x) and y because then it might manipulate x instead of y.)

tailcalled Apr 2, 2025, 12:34 PM
6 points
2
in reply to: Davidmanheim’s comment on: Is instrumental convergence a thing for virtue-driven agents?
I didn’t claim virtue ethics says not to predict consequences of actions. I said that a virtue is more like a procedure than it is like a utility function. A procedure can include a subroutine predicting the consequences of actions and it doesn’t become any more of a utility function by that.
The notion that “intelligence is channeled differently” under virtue ethics requires some sort of rule, like the consequentialist argmax or Bayes, for converting intelligence into ways of choosing.

tailcalled Apr 2, 2025, 7:27 AM
18 points
−3
on: Is instrumental convergence a thing for virtue-driven agents?
Consequentialism is an approach for converting intelligence (the ability to make use of symmetries to e.g. generalize information from one context into predictions in another context or to e.g. search through highly structured search spaces) into agency, as one can use the intelligence to predict the consequences of actions and find a policy which achieves some criterion unusually well.
While it seems intuitively appealing that non-consequentialist approaches could be used to convert intelligence into agency, I have tried a lot and not been able to come up with anything convincing. For virtues in particular, I would intuitively think that a virtue is not a motivator per se, but rather the policy generated by the motivator. So I think virtue-driven AI agency just reduces to ordinary programming/GOFAI, and that there’s no general virtue-ethical algorithm to convert intelligence into agency.
The most straightforward approach to programming a loyal friend would be to let the structure of the program mirror the structure^[1] of the loyal friendship. That is, you would think of some situation that a loyal friend might encounter, and write some code that detects and handles this situation. Having a program whose internal structure mirrors its external behavior avoids instrumental convergence (or any kind of convergence) because each behavior is specified separately and one can make arbitrary exceptions as one sees fit. However, it also means that the development and maintenance burden scales directly with how many situations the program generalizes to.
1. ^
  This is the “standard” way to write programs—e.g. if you make a SaaS app, you often have template files with a fairly 1:1 correspondence to the user interface, database columns with a 1:many correspondence to the user interface fields, etc.. By contrast, a chess bot that does a tree search does not have a 1:1 correspondence between the code and the plays; for instance the piece value table does not clearly affect it’s behavior in any one situation, but obviously kinda affects its behavior in almost all situations. (I don’t think consequentialism is the only way for the structure of a program to not mirror the structure of its behavior, but it’s the most obvious way.)

tailcalled Mar 25, 2025, 3:04 PM
2 points
0
in reply to: Abhimanyu Pallavi Sudhir’s comment on: Latent variables for prediction markets: motivation, technical guide, and design considerations
Not sure what you mean. Are you doing a definitional dispute about what counts as the “standard” definition of Bayesian networks?

tailcalled Mar 25, 2025, 9:36 AM
2 points
0
in reply to: Abhimanyu Pallavi Sudhir’s comment on: Latent variables for prediction markets: motivation, technical guide, and design considerations
Your linked paper is kind of long—is there a single part of it that summarizes the scoring so I don’t have to read all of it?
Either way, yes, it does seem plausible that one could create a market structure that supports latent variables without rewarding people in the way I described it.

tailcalled Mar 21, 2025, 12:20 PM
7 points
1
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
I’m not convinced Scott Alexander’s mistakes page accurately tracks his mistakes. E.g. the mistake on it I know the most about is this one:
56: (5/27/23) In Raise Your Threshold For Accusing People Of Faking Bisexuality, I cited a study finding that most men’s genital arousal tracked their stated sexual orientation (ie straight men were aroused by women, gay men were aroused by men, bi men were aroused by either), but women’s genital arousal seemed to follow a bisexual pattern regardless of what orientation they thought they were—and concluded that although men’s orientation seemed hard-coded, women’s orientation must be more psychological. But Ozy cites a followup study showing that women (though not men) also show genital arousal in response to chimps having sex, suggesting women’s genital arousal doesn’t track actual attraction and is just some sort of mechanical process triggered by sexual stimuli. I should not have interpreted the results of genital arousal studies as necessarily implying attraction.
But that’s basically wrong. The study found women’s arousal to chimps having sex to be very close to their arousal to nonsexual stimuli, and far below their arousal to sexual stimuli.

tailcalled Mar 20, 2025, 5:47 PM
4 points
2
in reply to: Christopher King’s comment on: How far along Metr’s law can AI start automating or helping with alignment research?
I mean I don’t really believe the premises of the question. But I took “Even if you’re not a fan of automating alignment, if we do make it to that point we might as well give it a shot!” to imply that even in such a circumstance, you still want me to come up with some sort of answer.

tailcalled Mar 20, 2025, 5:37 PM
0 points
−7
on: How far along Metr’s law can AI start automating or helping with alignment research?
Life on earth started 3.5 billion years ago. Log_2(3.5 billion years/1 hour) = 45 doublings. With one doubling every 7 months, that makes 26 years, or in 2051.
(Obviously this model underestimates the difficulty of getting superalignment to work. But also extrapolating the METR trend is questionable for 45 doublings is dubious in an unknown direction. So whatever.)

tailcalled Mar 15, 2025, 1:15 PM
6 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
I talk to geneticists (mostly on Twitter, or rather now BlueSky) and they don’t really know about this stuff.

tailcalled Mar 15, 2025, 1:03 PM
4 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
(Presumably there exists some standard text about this that one can just link to lol.)
I don’t think so.
I’m still curious whether this actually happens.… I guess you can have the “propensity” be near its ceiling.… (I thought that didn’t make sense, but I guess you sometimes have the probability of disease for a near-ceiling propensity be some number like 20% rather than 100%?) I guess intuitively it seems a bit weird for a disease to have disjunctive causes like this, but then be able to max out at the risk at 20% with just one of the disjunctive causes? IDK. Likewise personality...
For something like divorce, you could imagine the following causes:
- Most common cause is you married someone who just sucks
- … but maybe you married a closeted gay person
- … or maybe your partner was good but then got cancer and you decided to abandon them rather than support them through the treatment
The genetic propensities for these three things are probably pretty different: If you’ve married someone who just sucks, then a counterfactually higher genetic propensity to marry people who suck might counterfactually lead to having married someone who sucks more, but a counterfactually higher genetic propensity to marry a closeted gay person probably wouldn’t lead to counterfactually having married someone who sucks more, nor have much counterfactual effect on them being gay (because it’s probably a nonlinear thing), so only the genetic propensity to marry someone who sucks matters.
In fact, probably the genetic propensity to marry someone who sucks is inversely related to the genetic propensity to divorce someone who encounters hardship, so the final cause of divorce is probably even more distinct from the first one.

tailcalled Mar 15, 2025, 12:47 PM
10 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
Ok, more specifically, the decrease in the narrowsense heritability gets “double-counted” (after you’ve computed the reduced coefficients, those coefficients also get applied to those who are low in the first chunk and not just those who are high, when you start making predictions), whereas the decrease in the broadsense heritability is only single-counted. Since the single-counting represents a genuine reduction while the double-counting represents a bias, it only really makes sense to think of the double-counting as pathological.

tailcalled Mar 15, 2025, 12:32 PM
2 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
It would decrease the narrowsense (or additive) heritability, which you can basically think of as the squared length of your coefficient vector, but it wouldn’t decrease the broadsense heritability, which is basically the phenotypic variance in expected trait levels you’d get by shuffling around the genotypes. The missing heritability problem is that when we measure these two heritabilities, the former heritability is lower than the latter.

tailcalled Mar 15, 2025, 12:17 PM
2 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
If some amount of heritability is from the second chunk, then to that extent, there’s a bunch of pairs of people whose trait differences are explained by second chunk differences. If you made a PGS, you’d see these pairs of people and then you’d find out how specifically the second chunk affects the trait.
This only applies if the people are low in the first chunk and differ in the second chunk. Among the people who are high in the first chunk but differ in the second chunk, the logarithm of their trait level will be basically the same regardless of the second chunk (because the logarithm suppresses things by the total), so these people will reduce the PGS coefficients rather than increasing the PGS coefficients. When you create the PGS, you include both groups, so the PGS coefficients will be downwards biased relative to $γ$ .

tailcalled Mar 15, 2025, 12:00 PM
2 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
Why?

tailcalled Mar 15, 2025, 11:44 AM
2 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
Some of the heritability would be from the second chunk of genes.

tailcalled Mar 15, 2025, 11:18 AM
2 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
The original discussion was about how personality traits and social outcomes could behave fundamentally differently from biological traits when it comes to genetics. So this isn’t necessarily meant to apply to disease risks.

tailcalled Mar 15, 2025, 10:19 AM
2 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
Let’s start with the basics: If the outcome $f$ is a linear function of the genes $x$ , that is $f (x) = β x$ , then the effect of each gene is given by the gradient of $f$ , i.e. $\nabla_{x} f (x) = β$ . (This is technically a bit sketchy since a genetic variant is discrete while gradients require continuity, but it works well enough as a conceptual approximation for our purposes.) Under this circumstance, we can think of genomic studies as finding $β$ . (This is also technically a bit sketchy because of linkage disequillibrium and such, but it works well enough as a conceptual approximation for our purposes.)
If $f$ isn’t a linear function, then there is no constant $β$ to find. However, the argument for genomic studies still mostly goes through that they can find $E [\nabla_{x} f (x)]$ , it’s just that this expression now denotes a weird mismash effect size that’s not very interpretable.
As you observed, if $f$ is almost-linear, for example if $f (x) = e^{β x}$ , then genomic studies still have good options. The best is probably to measure the genetic influence on $log f$ , as then we get a pretty meaningful coefficient out of it. (If we measured the genetic influence of $f$ without the logarithm, I think under commonly viable assumptions we would get $β_{i}^{'} \propto e^{β_{i}} - 1$ , but don’t cite me on that.)
The trouble arises when you have deeply nonlinear forms such as $f (x) = e^{β x} + e^{γ x}$ . If we take the gradient of this, then the chain rule gives us $\nabla log f (x) = \frac{e^{β x} β + e^{γ x} γ}{e^{β x} + e^{γ x}}$ . That is, the two different mechanisms “suppress” each other, so if $e^{β x}$ is usually high, then the $γ$ term would usually be (implicitly!) excluded from the analysis.

tailcalled Mar 14, 2025, 12:49 PM
2 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
It kind-of applies to the Bernoulli-sigmoid-linear case that would usually be applied to binary diagnoses (but only because of sample size issues and because they usually perform the regression one variable at a time to reduce computational difficulty), but it doesn’t apply as strongly as it does to the polynomial case, and it doesn’t apply to the purely linear (or exponential-linear) case at all.
If you have a purely linear case, then the expected slope of a genetic variant onto an outcome of interest is proportional to the effect of the genetic variant.
The issue is in the polynomial case, the effect size of one genetic variant depends on the status of other genetic variants within the same term in the sum. Statistics gives you a sort of average effect size, but that average effect size is only going to be accurate for the people with the most common kind of depression.

tailcalled Mar 14, 2025, 12:30 PM
2 points
0
in reply to: TsviBT’s comment on: How to Make Superbabies
It doesn’t matter if depression-common is genetic or environmental. Depression-common leads to the genetic difference between your cases and controls to be small along the latent trait axis that causes depression-rare. So the effect gets estimated to be not-that-high. The exact details of how it fails depends on the mathematical method used to estimate the effect.