Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
It’s an expectation that has to do with a function of the thing, an expectation that the thing will function for some purpose. I suppose you could decompose that kind of claim to a more complex claim that doesn’t involve “function”, but in practice this is difficult.
I guess my main point is that sometimes fulfilling one’s functions is necessary for knowledge, e.g. you need to check proofs correctly to have the knowledge that the proofs you have checked are correct, the expectation that you check proofs correctly is connected with the behavior of checking them correctly.
I paid attention to this mainly because other people wanted me to, but the high IQ thing also draws some attention. I’ve seen ideas like “theory of cognitive processes should be integrated into philosophy of science” elsewhere (and have advocated such ideas myself), “syndiffeonesis” seems like an original term (although some versions of it appear in type theory), “conspansion” seems pretty Deleuzian, UBT is Spinozan, “telic recursion” is maybe original but highly underspecified… I think what I found useful about it is that it had a lot of these ideas, at least some of which are good, and different takes on/explanations of them than I’ve found elsewhere even when the ideas themselves aren’t original.
I don’t see any. He even says his approach “leaves the current picture of reality virtually intact”. In Popper’s terms this would be metaphysics, not science, which is part of why I’m skeptical of the claimed applications to quantum mechanics and so on. Note that, while there’s a common interpretation of Popper saying metaphysics is meaningless, he contradicts this.
Quoting Popper:
Language analysts believe that there are no genuine philosophical problems, or that the problems of philosophy, if any, are problems of linguistic usage, or of the meaning of words. I, however, believe that there is at least one philosophical problem in which all thinking men are interested. It is the problem of cosmology: the problem of understanding the world—including ourselves, and our knowledge, as part of the world. All science is cosmology, I believe, and for me the interest of philosophy, no less than of science, lies solely in the contributions which it has made to it.
...
I have tried to show that the most important of the traditional problems of epistemology—those connected with the growth of knowledge—transcend the two standard methods of linguistic analysis and require the analysis of scientific knowledge. But the last thing I wish to do, however, is to advocate another dogma. Even the analysis of science—the ‘philosophy of science’—is threatening to become a fashion, a specialism. yet philosophers should not be specialists. For myself, I am interested in science and in philosophy only because I want to learn something about the riddle of the world in which we live, and the riddle of man’s knowledge of that world. And I believe that only a revival of interest in these riddles can save the sciences and philosophy from narrow specialization and from an obscurantist faith in the expert’s special skill, and in his personal knowledge and authority; a faith that so well fits our ‘post-rationalist’ and ‘post-critical’ age, proudly dedicated to the destruction of the tradition of rational philosophy, and of rational thought itself.
...
Positivists usually interpret the problem of demarcation in a naturalistic way; they interpret it as if it were a problem of natural science. Instead of taking it as their task to propose a suitable convention, they believe they have to discover a difference, existing in the nature of things, as it were, between empirical science on the one hand and metaphysics on the other. They are constantly trying to prove that metaphysics by its very nature is nothing but nonsensical twaddle—‘sophistry and illusion’, as Hume says, which we should ‘commit to the flames’. If by the words ‘nonsensical’ or ‘meaningless’ we wish to express no more, by definition, than ‘not belonging to empirical science’, then the characterization of metaphysics as meaningless nonsense would be trivial; for metaphysics has usually been defined as non-empirical. But of course, the positivists believe they can say much more about metaphysics than that some of its statements are non-empirical. The words ‘meaningless’ or ‘nonsensical’ convey, and are meant to convey, a derogatory evaluation; and there is no doubt that what the positivists really want to achieve is not so much a successful demarcation as the final overthrow and the annihilation of metaphysics. However this may be, we find that each time the positivists tried to say more clearly what ‘meaningful’ meant, the attempt led to the same result—to a definition of ‘meaningful sentence’ (in contradistinction to ‘meaningless pseudo-sentence’) which simply reiterated the criterion of demarcation of their inductive logic.
...
In contrast to these anti-metaphysical stratagems—anti-metaphysical in intention, that is—my business, as I see it, is not to bring about the overthrow of metaphysics. It is, rather, to formulate a suitable characterization of empirical science, or to define the concepts ‘empirical science’ and ‘metaphysics’ in such a way that we shall be able to say of a given system of statements whether or not its closer study is the concern of empirical science.
Ok, I misunderstood. (See also my post on the relation between local and global optimality, and another post on coordinating local decisions using MCMC)
UDT1.0, since it’s just considering modifying its own move, corresponds to a player that’s acting as if it’s independent of what everyone else is deciding, instead of teaming up with its alternate selves to play the globally optimal policy.
I thought UDT by definition pre-computes the globally optimal policy? At least, that’s the impression I get from reading Wei Dai’s original posts.
Some possible AI architectures are structured as goal function optimization and by assumption that the human brain contains one or more expected utility maximizers, there is a human utility function that could be a possible AI goal. I’m not saying it’s likely.
With just that you could get upper bounds for the real. You could get some lower bounds by showing all rationals in the enumeration are greater than some rational, but this isn’t always possible to do, so maybe your type includes things that aren’t real numbers with provable lower bounds.
If you require both then we’re back at the situation where, if there’s a constructive proof that the enumerations min/max to the same value, you can get a Cauchy real out of this, and perhaps these are equivalent.
It seems that a real number defined this way will have some perhaps-infinite list of rationals it’s less than and one it’s greater than. You might want to add a constraint that the maximum of the list of numbers it’s above gets arbitrarily close to the minimum of the list of numbers it’s below (as Tailcalled suggested).
With respect to Cauchy sequences, the issue is how to specify convergence; the epsilon/N definition is one way to do this and, constructively, gives a way of computing epsilon-good approximations.
The power of this seems similar to the power of constructive Cauchy sequences because you can use the (x < y) → A u B function to approximate the value to any positive precision error.
By truth values do you mean Prop or something else?
Here’s how one might specify Dedekind cuts in type theory. Provide two types A,B with mappings , . To show these cover all the rationals, provide such that the value returned by c maps back to its argument, through functions or . But this lets us re-construct a function by seeing whether provides an A or a B. There are other ways of doing this but I’m not sure what else is worth analyzing.
Well, the one thing making that difficult is that I did not know the Lagrange multiplier theorem until reading this comment.
I agree this is in practice not directly applicable because buying contracts with all your money is silly.
All you need is to construct an appropriate probability space and use basic probability theory instead of inventing clever reasons why it doesn’t apply in this particular case.
I don’t see how to do that but maybe your plan is to get to that at some point
Am I missing something? How is it at all controversial?
it’s not, it’s just a modification on the usual halfer argument that “you don’t learn anything upon waking up”
halfers have to condition on there being at least one observer in the possible world. if the coin can come up 0,1,2 at 1⁄3 each, and Sleeping Beauty wakes up that number of times, halfers still think the 0 outcome is 0% likely upon waking up.
halfers also have to construct the reference class carefully. if there are many events of people with amnesia waking up once or twice, and SSA’s reference class consists of the set of awakenings from these, then SSA and SIA will agree on a 1⁄3 probability. this is because in a large population, about 1⁄3 of awakenings are in worlds where the coin came up such that there would be one awakening.
I don’t have a better solution right now, but one problem to note is that this agent will strongly bet that the button will be independent of the human pressing the button. So it could lose money to a different agent that thinks these are correlated, as they are.
Nice job with the bound! I’ve heard a number of people in my social sphere say very positive things about DACs so this is mainly my response to them.
You mentioned wanting to get the game theory of love correct. Understanding a game involves understanding the situations and motives of the involved agents. So getting the game theory of love correct with respect to some agent implies understanding that agent’s situation.
This seems more like “imagining being nice to Hitler, as one could be nice to anyone” than “imagining what Hitler was in fact like and why his decisions seemed to him like the thing to do”. Computing the game theoretically right strategy involves understanding different agents’ situations, the kind of empathy that couldn’t be confused with being a doormat, sometimes called “cognitive empathy”.
I respect Sarah Constantin’s attempt to understand Hitler’s psychological situation.
If you define “human values” as “what humans would say about their values across situations”, then yes, predicting “human values” is a reasonable training objective. Those just aren’t really what we “want” as agents, and agentic humans would have motives not to let the future be controlled by an AI optimizing for human approval.
That’s also not how I defined human values, which is based on the assumption that the human brain contains one or more expected utility maximizers. It’s possible that the objectives of these maximizers are affected by socialization, but they’ll be less affected by socialization than verbal statements about values, because they’re harder to fake so less affected by preference falsification.
Children learn some sense of what they’re supposed to say about values, but have some pre-built sense of “what to do / aim for” that’s affected by evopsych and so on. It seems like there’s a huge semantic problem with talking about “values” in a way that’s ambiguous between “in-built evopsych-ish motives” and “things learned from culture about what to endorse”, but Yudkowsky writing on complexity of value is clearly talking about stuff affected by evopsych. I think it was a semantic error for the discourse to use the term “values” rather than “preferences”.
In the section on subversion I made the case that terminal values make much more difference in subversive behavior than compliant behavior.
It seems like to get at the values of approximate utility maximizers located in the brain you would need something like Goal Inference as Inverse Planning rather than just predicting behavior.
Regarding quantum, I’d missed the bottom text. It seems if I only read the main text, the obvious interpretation is that points are events and the circles restrict which other events they can interact with. He says “At the same time, conspansion gives the quantum wave function of objects a new home: inside the conspanding objects themselves” which implies the wave function is somehow located in the objects.
From the diagram text, it seems he is instead saying that each circle represents entangled wavefunctions of some subset of objects that generated the circle. I still don’t see how to get quantum non-locality from this. The wave function can be represented as a complex valued function on configuration space; how could it be factored into a number of entanglements that only involve a small number of objects? In probability theory you can represent a probability measure as a factor graph, where each factor only involves a limited subset of variables, but (a) not all distributions can be efficiently factored this way, (b) generalizing this to quantum wave functions is additionally complicated due to how wave functions differ from probability distributions.