Where can I find those events if I want to be a non-speaker participant?
Vanessa Kosoy(Vanessa Kosoy)
Karma: 2,539
I’m glad it worked :) It’s not that surprising given that pain is known to be susceptible to the placebo effect. I would link the SSC post, but, alas...
Well, HRAD certainly has relations to my own research programme. Embedded agency seems important since human values are probably “embedded” to some extent, counterfactuals are important for translating knowledge from the user’s subjective vantage point to the AI’s subjective vantage point, reflection is important if it’s required for high capability (as Turning RL suggests). I do agree that having a high level plan for solving the problem is important to focus the research in the right directions.
There are “shared” phobias, and common types of paranoia. There are also beliefs many people share that have little to do with reality, such as conspiracy theories or UFOs. Of course in the latter case they share those beliefs because they transmitted them to each other, but the mystics are also influenced by each other.
Trial and error assumes an objective, measurable loss function. What is the loss function here, and why is it relevant to ethics? Also, can you give some examples how this method allows solving questions debated in Western philosophy, such as population ethics, the moral status of animals, time discount or nosy preferences?
Mysticism is a suite of techniques designed to observe the constructs of your mind in absolute terms.
I am skeptical. Designed by who, and how? Why do you think this design is any good? What makes it so much better than Western philosophy at understanding ethics?
I don’t understand how do you get from “mystical experiences follow structured patterns” to “mystical experiences have implications on ethics”. Btw I think that mental illness also follows structured patterns.
I was with you until “But there is a way to observe ethics in absolute terms. It is called “mysticism”″ I have no idea what you mean there.
I think theoretical work on AI safety has multiple different benefits, but I prefer a slightly different categorization. I like categorizing in terms of the sort of safety guarantees we can get, on a spectrum from “stronger but harder to get” to “weaker but easier to get”. Specifically, the reasonable goals for such research IMO are as follows.
Plan A is having (i) a mathematical formalization of alignment (ii) a specific practical algorithm (iii) a proof that this algorithm is aligned, or at least a solid base of theoretical and empirical evidence, similarly to the situation in cryptography. This more or less correspond to World 2.
Plan B is having (i) a mathematical formalization of alignment (ii) a specific practical algorithm (iii) a specific impractical but provably aligned algorithm (iv) informal and empirical arguments suggesting that the former algorithm is as aligned as the latter. As an analogy consider Q-learning (an impractical algorithm with provable convergence guarantees) and deep Q-learning (a practical algorithm with no currently known convergence guarantees, designed by analogy to the former). This sort of still corresponds to World 2 but not quite.
Plan C is having enough theory to at least have rigorous models of all possible failure modes, and theory-inspired informal and empirical arguments why a certain algorithm avoids them. As an analogy, concepts such as VC dimension and Rademacher complexity allow us being more precise in our reasoning about underfitting and overfitting, even if we don’t know how to compute them in practical scenarios. This corresponds to World 1, I guess?
In a sane civilization the solution would be not building AGI until we can implement Plan A. In the real civilization, we should go with the best plan that will be ready by the time competing projects become too dangerous to ignore.
World 3 seems too ambitious to me, since analyzing arbitrary code is almost always an intractable problem (e.g. Rice’s theorem). You would need at least some constraints on how your agent is designed.
As an avid reader of romance novels (which is a genre written by women for women), my observation is that some male protagonists are kinda assholes, but that’s probably a minority. It is true that almost all male protagonists are stereotypical “alpha males”: strong, courageous, confident, assertive, high status, possessive. But many of them are also honorable and kind, which is the opposite of asshole.
Personally I prefer nice nerds, but I’m probably atypical.
That video is very long, can you explain what is meant by “scramble an arbitrary word”?
The fact that one copy gets shot doesn’t mean that “there’s only one thing that matches”. In spacetime the copy that got shot still exists. You do have hypotheses of the form “locate a person that still lives after time and track their history to the beginning and forward to the future”, but those hypotheses are suppressed by .
I didn’t follow all of that, but your probability count after shooting seems wrong. IIUC, you claim that the probabilities go from {1/2 uncopied, 1⁄4 copy #1, 1⁄4 copy #2} to {1/2 uncopied, 1⁄2 surviving copy}. This is not right. The hypotheses considered in Solomonoff induction are supposed to describe your entire history of subjective experiences. Some hypotheses are going to produce histories in which you get shot. Updating on not being shot is just a Bayesian update, it doesn’t change the complexity count. Therefore, the resulting probabilities are {2/3 uncopied, 1⁄3 surviving copy}.
This glosses over what Solomonoff induction thinks you will experience if you do get shot, which requires a formalism for embedded agency to treat properly (in which case the answer becomes, ofc, that you don’t experience anything), but the counting principle remains the same.
By the same token, you can pray to Elua, the deity of human flourishing. Ey exist in about the same senses your ideal future-self exists: (i) ey exist somewhere in the multiverse (because the multiverse contains anything that can be imagined) and ey exist with some probability in the future (because maybe we will create a superintelligent AI that will make it real) (ii) ey exist inside you to the extent you feel motivated by making the world better (iii) we can decide to believe in em together.
It probably sounds like I’m being ironic, but actually I’m serious. I’m just worried that talking about this too much will make us sound crazy, and also about some people taking it too seriously and actually going crazy.
Like I wrote before, slow takeoff might be actually worse than fast takeoff. This is because even if the first powerful AIs are aligned, their head start on unaligned AIs will not count as much, and alignment might (and probably will) require overhead that will give the unaligned AIs an advantage. Therefore, success would require that the institutions will either prevent or quickly shut down unaligned AIs for enough time that aligned AIs gain the necessary edge.
How about the following model for what is going on.
A human is an RL agent with a complicated reward function, and one of the terms in this reward function requires (internally) generating an explanation for the agent’s behavior. You can think of it as a transparency technique. The evolutionary reason for this is that other people can ask you to explain your behavior (for example, to make sure they can trust you) and generating the explanation on demand would be too slow (or outright impossible if you don’t remember the relevant facts), so you need to do it continuously. Like often with evolution, an instrumental (from the perspective of reproductive fitness) goal has been transformed into a terminal (from the perspective of the evolved mind) goal.
The criteria for what comprises a valid explanation are at least to some extent learned (we can speculate that the learning begins when parents ask children to explain their behavior, or suggest their own explanations). We can think of the explanation as a “proof” and the culture as determining the “system of axioms”. Moreover, the resulting relationship between the actual behavior and explanation is two way: on the one hand, if you behaved in a certain way, then the explanation needs to reflect it, on the other hand, if you already generated some explanation, then your future behavior should be consistent with it. This might be the primary mechanism by which the culture’s notions about morality are “installed” into the individual’s preferences: you need to (internally) explain your behavior in a way consistent with the prevailing norms, therefore you need to actually not stray too far from those norms.
The thing we call “self” or “consciousness” is not the agent, is not even a subroutine inside the agent, it is the explanation. This is because any time someone describes eir internal experiences, ey are actually describing this “innate narrative”: after all, this is exactly its original function.
I spent a lot of the last two years getting really into categorical logic (as in, using category theory to study logic), because I’m really into logic, and category theory seemed to be able to provide cool alternate foundations of mathematics.
Turns out it doesn’t really.
Category thing doesn’t. But, (the closely related) homotopy type theory does.
I accepted 3 candidates. Unfortunately, all of them dropped out some time into the programme (each of them lasted a few months give or take). I’m not sure whether it’s because (i) I’m a poor mentor (ii) I chose the wrong candidates (iii) there were no suitable candidates or (iv) just bad luck. Currently I am working with a collaborator, but ey arrived in a different way. Maybe I will write a detailed post-mortem some time, but I’m not sure.
Actually, as opposed to what I claimed before, we don’t need computational complexity bounds for this definition to make sense. This is because the Solomonoff prior is made of computable hypotheses but is uncomputable itself.
Given , we define that ” has (unbounded) goal-directed intelligence (at least) ” when there is a prior and utility function s.t. for any policy , if then . Here, is the Solomonoff prior and is Kolmogorov complexity. When (i.e. no computable policy can match the expected utility of ; in particular, this implies is optimal since any policy can be approximated by a computable policy), we say that is “perfectly (unbounded) goal-directed”.
Compare this notion to the Legg-Hutter intelligence measure. The LH measure depends on the choice of UTM in radical ways. In fact, for some UTMs, AIXI (which is the maximum of the LH measure) becomes computable or even really stupid. For example, it can always keep taking the same action because of the fear that taking any other action leads to an inescapable “hell” state. On the other hand, goal-directed intelligence differs only by between UTMs, just like Kolmogorov complexity. A perfectly unbounded goal-directed policy has to be uncomputable, and the notion of which policies are such doesn’t depend on the UTM at all.
I think that it’s also possible to prove that intelligence is rare, in the sense that, for any computable stochastic policy, if we regard it as a probability measure over deterministic policies, then for any there is s.t. the probability to get intelligence at least is smaller than .
Also interesting is that, for bounded goal-directed intelligence, increasing the prices can only decrease intelligence by , and a policy that is perfectly goal-directed w.r.t. lower prices is also such w.r.t. higher prices (I think). In particular, a perfectly unbounded goal-directed policy is perfectly goal-directed for any price vector. Informally speaking, an agent that is very smart relatively to a context with cheap computational resources is still very smart relatively to a context where they are expensive, which makes intuitive sense.
If we choose just one computational resource, we can speak of the minimal price for which a given policy is perfectly goal-directed, which is another way to measure intelligence with a more restricted domain. Curiously, our bounded Solomonoff-like prior has the shape of a Maxwell-Boltzmann distribution in which the prices are thermodynamic parameters. Perhaps we can regard the minimal price as the point of a phase transition.
This idea is certainly not new, for example in an essay about TDT from 2009, Yudkowsky wrote:
(emphasis mine)
The relevance of TDT/UDT/FDT to voting surfaced in discussions many times, but possibly nobody wrote a detailed essay on the subject.