Arms races are bad things. First best by far is if nobody has the doomsday devices, but second best is if we attempt nonproliferation of doomsday devices.
As a parallel, we would have been still at risk in a world where DeepMind was working on building ASI but where Elon didn’t freak out and start a competitor (followed by another competitor), but not as much risk. That’s not because DeepMind are “the good guys”, it’s because of race dynamics.
Capabilities being more jagged reduces p(doom), less jagged increases it.
Ceteris paribus, perhaps, but I think the more important factor is that more jagged capabilities imply a faster timeline. In order to be an existential threat, an AI only needs to have one superhuman ability that suffices for victory (be that superpersuasion or hacking or certain kinds of engineering etc), rather than needing to exceed human capabilities across the board.
rot13: Gur snvyfnsr jnf gur bayl cneg gung fhecevfrq zr, naq gur qvrtrgvp rkcynangvba sbe ubj guvf frghc pbhyq unir unccrarq ng nyy jnf phgr.
I may never stop finding it funny the extent to which Trump will seek out the one thing we know definitively is going badly, then and choose that to lie and brag about.
We’re not the target audience for those posts. He’s telling everyone who’s kissing his ass that this is a topic he wants them to lie about as well.
If we were doing a better job catching cheaters presumably people would be doing it less?
Seems trivially easy for an idiot to start an account, lose some games, get frustrated, and start asking a chess bot what to do so that they can rescue a bad position / show that bastard what’s what / vicariously enjoy winning. I expect that crowd to be the majority of cheaters, and for them to be essentially inelastic to the probability with which they get caught (the first time).
I believe that Cade knows perfectly well what everyone has been saying for years; he’s being disingenuous because the object level doesn’t matter to him, and the only important thing is ensuring that these weirdos don’t get status. He’s never once engaged on simulacrum level 1 with this community.
At long last, Elon Musk has created the Yandere Simulator from the classic short story Don’t Create The Yandere Simulator, Among Other Things
On several of these I think you might be confounding the positively correlated variables “has heard of this question” and “is on the autism spectrum”, the latter of which is anticorrelated with the kind of behavior you want (intuitive, empathetic listening without needing to ask a bunch of explicit questions). I find it unlikely that asking the question makes people worse at the behavior you want.
Zvi is arguing “X implies Y” here. Zvi happens to believe Y but disbelieve X; however, he is writing to people who think “X and not-Y”, in order to nudge them to support Y.
Here X = it is good for the US to build superintelligence fast, before China does, and Y = we should have some diffusion rules making it harder for China to catch up to the USA.
Zvi believes Z = nobody should be building superintelligence soon, and believes Z implies Y, but it is useful to show that X implies Y as well.
Question for @Scott Garrabrant, @TsviBT, @Andrew_Critch, @So8res, @jessicata, and anyone else who knows the answer: the logical inductor constructed in the paper is not merely computable but also primitive recursive, right?
Seems obvious to me (because the fixed price point is approximated, etc), but I want to be sure I’m not missing something.
To be fair, I also think it was a correct decision to focus on the Democratic Party when advocating for sane AI notkilleveryoneism policy before the 2024 election, because that was playing to our outs. Trump will do on AI whatever he is personally paid to do, and the accelerationists have more money; anyone sane but unable/unwilling to bribe Trump has no influence whatsoever with him in this administration.
(I went around saying the above before the election, by the way.)
(Sorry for the mindkilling, it’s relevant on the object level.)
[EDIT: Never mind, this is just Kleene’s second recursion theorem!]
Quick question about Kleene’s recursion theorem:
Let’s say F is a computable function from ℕ^N to ℕ. Is there a single computable function X from ℕ^N to ℕ such that
X = F(X, y_2,..., y_N) for all y_2,...,y_N in ℕ
(taking the X within F as the binary code of X in a fixed encoding) or do there need to be additional conditions?
My current candidate definitions, with some significant issues in the footnotes:
A fair environment is a probabilistic function from an array of actions to an array of payoffs.
An agent is a random variable
which takes in a fair environment [1] and a list of agents (including itself), and outputs a mixed strategy over its available actions in . [2]
A fair agent is one whose mixed strategy is a function of subjective probabilities[3] that it assigns to [the actions of some finite collection of agents in fair environments, where any agents not appearing in the original problem must themselves be fair].
Formally, if is a fair agent in with a subjective probability estimator , ’s mixed strategy in a fair environment ,
should depend only on a finite collection of ’s subjective probabilities about outcomes
for a set of fair environments and an additional set of fair[4] agents[5] if needed (note that not all agents need to appear in all environments).
A fair problem is a fair environment with one designated player, where all other agents are fair agents.
I might need to require every to have a default action , so that I don’t need to worry about axiom-of-choice issues when defining an agent over the space of all fair environments.
I specified a probabilistic environment and mixed strategies because I think there should be a unique fixed point for agents, such that this is well-defined for any fair environment . (By analogy to reflective oracles.) But I might be wrong, or I might need further restrictions on .
Grossly underspecified. What kinds of properties are required for subjective probabilities here? You can obviously cheat by writing BlueEyedBot into your probability estimator.
This is an infinite recursion, of course. It works if we require each to have a strictly lower complexity in some sense than (e.g. the rank of an agent is the largest number of environments it can reason about when making any decision, and each needs to be lower-rank than ), but I worry that’s too strong of a restriction and would exclude some well-definable and interesting agents.
Does the fairness requirement on the suffice to avert the MetaBlueEyedBot problem in general? I’m unsure.
How do you formalize the definition of a decision-theoretically fair problem, even when abstracting away the definition of an agent as well as embedded agency?
I’ve failed to find anything in our literature.
It’s simple to define a fair environment, given those abstractions: a function E from an array of actions to an array of payoffs, with no reference to any other details of the non-embedded agents that took those actions and received those payoffs.
However, fair problems are more than just fair environments: we want a definition of a fair problem (and fair agents) under which, among other things:
The classic Newcomb’s Problem against Omega, with certainty or with 1% random noise: fair
Omega puts $1M in the box iff it predicts that the player consciously endorses one-boxing, regardless of what it predicts the player will actually do (e.g. misunderstand the instructions and take a different action than they endorsed): unfair
Prisoner’s Dilemma between two agents who base their actions on not only each others’ predicted actions in the current environment, but also their predicted actions in other defined-as-fair dilemmas: fair
For example, PrudentBot will cooperate with you if it deduces that you will cooperate with it and also that you would defect against DefectBot, because it wants to exploit CooperateBots).
Prisoner’s Dilemma between two agents who base their actions on each others’ predicted actions in defined-as-unfair dilemmas: unfair
It would let us smuggle in unfairness from other dilemmas; e.g. if BlueEyedBot only tries Löbian cooperation against agents with blue eyes, and MetaBlueEyedBot only tries Löbian cooperation against agents that predictably cooperate with BlueEyedBot, then the Prisoner’s Dilemma against MetaBlueEyedBot should count as unfair.
Modal combat doesn’t need to worry about this, because all the agents in it are fair-by-construction.
Yeah, I know, it’s about a decade late to be asking this question.
Over the past three years, as my timelines have shortened and my hopes for alignment or coordination have dwindled, I’ve switched over to consumption. I just make sure to keep a long runway, so that I could pivot if AGI progress is somehow halted or sputters out on its own or something.
The fault does not lie with Jacob, but wow, this post aged like an open bag of bread.
I suggest a fourth default question for these reading groups:
How did this post age?
Soon the two are lost in a maze of words defined in other words, the problem that Steven Harnad once described as trying to learn Chinese from a Chinese/Chinese dictionary.
Of course, it turned out that LLMs do this just fine, thank you.
When it specifically comes to loss-of-control risks killing or sidelining all of humanity, I don’t believe Sam or Dario or Demis or Elon want that to happen, because it would happen to them too. (Larry Page is different on that count, of course.) You do have conflict theory over the fact that some of them would like ASI to make them god-emperor of the universe, but all of them would definitely take a solution to “loss of control” if it were handed to them on a silver platter.