# Vanessa Kowalski

Karma: 785

NewTop

I strongly agree with these comments regarding is-ought. To add a little, talking about winning/losing, effective strategies or game theory assumes a specific

*utility function*. To say Maria Teresa “lost” we need to first agree that death and pain are bad. And even the concept of “survival” is not really well-defined. What does it mean to survive? If humanity is replaced by “descendants” which are completely alien or even monstrous from our point of view, did humanity “survive”? Surviving means little without thriving and*both*concepts are subjective and require already having some kind of value system to specify.

It also seems worth pointing out that the referent of the metaphor indeed has more than two levels. For example, we can try to break it down as genetic evolution → memetic evolution → unconscious mind → conscious mind. Each level is a “character” to the “player” of the previous level. Or, in computer science terms, we have a program writing a program writing a program writing a program.

Hmm, interesting. But

*why*was the cost of capital relative to labor so high?

This analysis seems correct but somewhat misleading. Specifically, I think that when a technology is enabled by a change in economic conditions, it is often the case that the change in economic conditions was caused by a different technology. So, the ultimate limiting factor is still insight.

In particular, Gutenberg’s printing press doesn’t seem like a great example for the “insight is not the limiting factor” thesis. First, the Chinese had movable type earlier but it was not as efficient with the Chinese language because of the enormous number of characters, which is why it didn’t become more popular in China. Second, you say yourself that “printing presses with movable type followed a century later”. A century is still a lot of time! Third, coming back to what I said before, why did paper production only took off in Europe in the 1300s? As far as I understand, it was invented in China, from there it propagated to the Muslim word, and from there it reached Europe through Spain. So, for many centuries, the reason the Europeans didn’t use paper was lack of insight. Only when the knowledge that originated it China reached them did they catch on.

You can get divorced and still have both parents in your kids’ lives. Conversely, you can remain married and make your kids miserable. There is no system that can force someone to be a good parent.

I mostly agree with all of that, but also, the case against “everyone managed it back in the good old days” seems understated IMO. If in the “good old days” everyone stayed in miserable relationships because of social and legal barriers to leaving, that’s not a point in favor of the good old days. I don’t see the advantage of modern society in this respect as a trade-off, it seems more like a win-win.

Nearly everything you said here was already addressed in my previous comment. Perhaps I didn’t explain myself clearly?

It would be trickier for the device I described to pull off such a deception, because it would have to actually halt and show us its output in such cases.

I wrote before that “I wonder how would you tell whether it is the hypercomputer you imagine it to be, versus the realization of the same hypercomputer in some non-standard model of ZFC?”

So, the realization of a particular hypercomputer in a non-standard model of ZFC would pass all of your tests. You could examine its internal state or its output any way you like (i.e. ask any question that can be formulated in the language of ZFC) and everything you see would be consistent with ZFC. The number of steps for a machine that shouldn’t halt would be a non-standard number, so it would not fit on any finite storage. You could examine some finite subset of its digits (either from the end or from the beginning), for example, but that would not tell you the number is non-standard. For any question of the form “is larger than some known number ?” the answer would always be “yes”.

But finite resource bounds already prevent us from completely ruling out far-fetched hypotheses about even normal computers. We’ll never be able to test, e.g., an arbitrary-precision integer comparison function on all inputs that could feasibly be written down. Can we be sure it always returns a Boolean value, and never returns the Warner Brothers dancing frog?

Once again, there is a difference of principle. I wrote before that: “...given an uncomputable function and a system under test , there is no sequence of computable tests that will allow you to form some credence about the hypothesis s.t. this credence will converge to when the hypothesis is true and when the hypothesis is false. (This can be made an actual theorem.) This is different from the situation with normal computers (i.e. computable ) when you

*can*devise such a sequence of tests.”So, with normal computers you can become increasingly certain your hypothesis regarding the computer is true (even if you never become literally 100% certain, except in the limit), whereas with a hypercomputer you cannot.

Actually, hypothesizing that my device “computed” a nonstandard version of the halting function would already be sort of self-defeating from a standpoint of skepticism about hypercomputation, because all nonstandard models of Peano arithmetic are known to be uncomputable.

Yes, I already wrote that: “Although you can in principle have a

*class*of uncomputable hypotheses s.t. you can asymptotically verify is in the class, for example the class of all functions s.t. it is consistent with ZFC that is the halting function. But the verification would be extremely slow and relatively parsimonious competing hypotheses would remain plausible for an extremely (uncomputably) long time. In any case, notice that the class itself has, in some strong sense, a computable description: specifically, the computable verification procedure itself.”So, yes, you could theoretically become certain the device is a hypercomputer (although reaching high certainly would take very long time), without knowing precisely

*which*hypercomputer it is, but that doesn’t mean you need to add non-computable hypotheses to your “prior”, since that knowledge would still be expressible as a computable property of the world.I don’t know enough about Solomonoff induction to say whether it would unduly privilege such hypotheses over the hypothesis that the device was a true hypercomputer (if it could even entertain such a hypothesis).

Literal Solomonoff induction (or even bounded versions of Solomonoff induction) is probably

*not*the ultimate “true” model of induction, I was just using it as a simple example before. The true model will allow expressing hypotheses such as “all the even-numbered bits in the sequence are “, which involve computable properties of the environment that do not specify it completely. Making this idea precise is somewhat technical.

I agree with Ben, and also, humanity successfully sent a spaceship to the moon surface on the second attempt and successfully sent

*people*(higher stakes) to the moon surface on the first attempt. This shows that difficult technological problems*can*be solved without extensive trial and error. (Obviously some trial and error on easier problems was done to get to the point of landing on the moon, and no doubt the same will be true of AGI. But, there is hope that the*actual*AGI can be constructed without trial and error, or at least without the sort of trial and error where error is potentially catastrophic.)

In some sense, yes, although for conventional computers you might settle on very slow verification. Unless you mean that, your mind has only finite memory/lifespan and therefore you cannot verify an arbitrary conventional computer within any given credence, which is also true. Under favorable conditions, you can quickly verify something in PSPACE (using interactive proof protocols), and given extra assumptions you might be able to do better (if you have two provers that cannot communicate you can do NEXP, or if you have a computer whose memory you can reliably delete you can do an EXP-complete language), however it is not clear whether you can be justifiably highly certain of such extra assumptions.

See also my reply to lbThingrb.

It is true that a human brain is more precisely described as a finite automaton than a Turing machine. And if we take finite lifespan into account, then it’s not even a finite automaton. However, these abstractions are useful models since they become accurate in certain asymptotic limits that are sufficiently useful to describe reality. On the other hand, I doubt that there is a useful approximation in which the brain is a hypercomputer (except

*maybe*some weak forms of hypercomputation like non-uniform computation / circuit complexity).Moreover, one should distinguish between different senses in which we can be “modeling” something. The first sense is the core, unconscious ability of the brain to generate models, and in particular that which we experience as intuition. This ability can (IMO) be thought of as some kind of machine learning algorithm, and, I doubt that hypercomputation is relevant there in any way. The second sense is the “modeling” we do by manipulating linguistic (symbolic) constructs in our conscious mind. These constructs might be formulas in some mathematical theory, including formulas that represent claims about uncomputable objects. However, these symbolic manipulations are just another

*computable*process, and it is only the results of these manipulations that we use to generate predictions and/or test models, since this is the only access we have to those uncomputable objects.Regarding your hypothetical device, I wonder how would you tell whether it is the hypercomputer you imagine it to be, versus the realization of the same hypercomputer in some non-standard model of ZFC? (In particular, the latter could tell you that some Turing machine halts when it “really” doesn’t, because in the model it halts after some non-standard number of computing steps.) More generally, given an uncomputable function and a system under test , there is no sequence of computable tests that will allow you to form some credence about the hypothesis s.t. this credence will converge to when the hypothesis is true and when the hypothesis is false. (This can be made an actual theorem.) This is different from the situation with normal computers (i.e. computable ) when you

*can*devise such a sequence of tests. (Although you can in principle have a*class*of uncomputable hypotheses s.t. you can asymptotically verify is in the class, for example the class of all functions s.t. it is consistent with ZFC that is the halting function. But the verification would be extremely slow and relatively parsimonious competing hypotheses would remain plausible for an extremely (uncomputably) long time. In any case, notice that the class itself has, in some strong sense, a computable description: specifically, the computable verification procedure itself.)My point is, the Church-Turing thesis implies (IMO) that the mathematical model of rationality/intelligence should be based on Turing machines

*at most*, and this observation does*not*strongly depend on assumptions about physics. (Well, if hypercomputation*is*physically possible,*and*realized in the brain, and there is some intuitive part of our mind that uses hypercomputation in a crucial way, then this assertion would be wrong. That would contradict my own intuition about what reasoning*is*(including intuitive reasoning),*besides*everything we know about physics, but obviously this hypothesis has*some*positive probability.)

What does it mean to have a box for solving the halting problem? How do you know it really solves the halting problem? There are some computable tests we can think of, but they would be incomplete, and you would only verify that the box satisfies those

*computable*tests, not that is “really” a hypercomputer. There would be a lot of possible boxes that*don’t*solve the halting problem that pass the same computable tests.If there is some powerful computational hardware available, I would want the AI the use that hardware. If you imagine the hardware as being hypercomputers, then you can think of such an AI as having a “prior over hypercomputable worlds”. But you can alternatively think of it as reasoning using computable hypotheses about the correspondence between the output of this hardware and the output of its sensors. The latter point of view is better, I think, because you can never know the hardware is really a hypercomputer.

Physics is not the territory, physics is (quite explicitly) the models we have of the territory. Rationality consists of the rules for formulating these models, and in this sense it is prior to physics and more fundumental. (This might be a disagreement over use of words. If by “physics” you, by definition, refer to the territory, then it seems to miss my point about Occam’s razor. Occam’s razor says that the

*map*should be parsimonious, not the territory: the latter would be a type error.) In fact, we can adopt the view that Solomonoff induction (which is a model of rationality) is the ultimate physical law: it is a mathematical rule of making predictions that generates all the other rules we can come up with. Such a point of view, although in some sense justified,*at present*would be impractical: this is because we know how to compute using actual physical models (including running computer simulations), but not so much using models of rationality. But this is just another way of saying we haven’t constructed AGI yet.I don’t think it’s meaningful to say that “weird physics may enable super Turing computation.” Hypercomputation is just a mathematical abstraction. We can imagine, to a point, that we live in a universe which contains hypercomputers, but since our own brain is

*not*a hypercomputer, we can never fully*test*such a theory. This IMO is the most fundumental significance of the Church-Turing thesis: since we only perceive the world through the lens of our own mind,

One way to deal with this is, have an entire set of utility functions (the different “lines”), normalize them so that they approximately agree inside “Mediocristan”, and choose the “cautious” strategy, i.e. the strategy the maximizes the

*minimum*of expected utility over this set. This way you are at least guaranteed not the end up in a place that is*worse*than “Mediocristan”.

From a multiverse perspective, it might be alright even if you resurrect a version significantly different from the original, see the essay by avturchin. Yudkowsky also discussed it somewhere on Facebook, but I don’t know how to find it.

I have not been in your situation, but I can relate in some ways, being ethnically Jewish and an atheist, and also having some profound world view differences with my parents. In your place, I would try to gain financial independence as fast as possible. This would both make your safe from any threats on part of your father, and (probably) make your father more reluctant to escalate, having no power to enforce control and (probably) having no desire to lose the connection with eir child, even if the child is an unbeliever.

In any case, you have my sincere sympathy and I hope everything will turn out for the best.

Hugs if wanted!

It seems like we might actually agree on this point: an abstract theory of evolution is not very useful for either building organisms or analysing how they work, and so too may an abstract theory of intelligence not be very useful for building intelligent agents or analysing how they work. But what we want is to build better birds! The abstract theory of evolution can tell us things like “species will evolve faster when there are predators in their environment” and “species which use sexual reproduction will be able to adapt faster to novel environments”. The analogous abstract theory of intelligence can tell us things like “agents will be less able to achieve their goals when they are opposed by other agents” and “agents with more compute will perform better in novel environments”. These sorts of conclusions are not very useful for safety.

As a matter of fact, I emphatically do

*not*agree. “Birds” are a confusing example, because it speaks of modifying an existing (messy, complicated, poorly designed) system rather than making something from scratch. If we wanted to make something vaguely bird-like from scratch, we might have needed something like a “theory of self-sustaining, self-replicating machines”.Let’s consider a clearer example: cars. In order to build a car, it is

*very*useful to have a theory of mechanics, chemistry, thermodynamic etc. Just doings things by trial and error would be much less effective,*especially*if you don’t want the car to occasionally explode (given that the frequency of explosions might be too low to affordably detect during testing). This is not because a car is “simple”: a spaceship or, let’s say, a gravity wave detector is much more complex than a car, and yet you hardly need*less*theory to make one.And another example: cryptography. In fact, cryptography is not so far from AI safety: in the former case, you defend against an external adversary whereas in the latter you defend against perverse incentives and subagents inside the AI. If we had this conversation in the 1960s (say), you might have said that cryptography is obviously a complex, messy domain, and theorizing about it is next to useless, or at least not helpful for designing actual encryption systems (there was Shannon’s work, but since it ignored computational complexity you can maybe compare it to algorithmic information theory and statistical learning theory for AI today; if we had this conversation in the 1930s, then there would next to no theory at all, even though encryption was practiced since ancient times). And yet, today theory plays an essential role in this field. The domain actually

*is*very hard: most of the theory relies on complexity theoretic conjectures that we are still far from being able to prove (although I expect that most theoretical computer scientists would agree that eventually we*will*solve them).*However*, even without being able to formally prove everything, the ability to reduce the safety of many different protocols to a limited number of interconnected conjectures (some of which have an abundance of both theoretical and empirical evidence) allows us to immensely increase our confidence in those protocols.Similarly, I expect an abstract theory of intelligence to be immensely useful for AI safety. Even just having precise language to define what “AI safety” means would be very helpful, especially to avoid counter-intuitive failure modes like the malign prior.

*At the very least*, we could have provably safe but impractical machine learning protocols that would be an*inspiration*to more complex algorithms about which we cannot prove things directly (like in deep learning today). More optimistically (but still realistically IMO) we could have practical algorithms satisfying theoretical guarantees modulo a small number of well-studied conjectures, like in cryptography today. This way, theoretical and empirical research could feed into each other, the whole significantly surpassing the sum of its parts.

...what I was trying to get at with “define its fitness in terms of more basic traits” is being able to build a model of how it can or should actually work, not just specify measurement criteria.

Once again, it seems perfectly possible to build an

*abstract*theory of evolution (for example, evolutionary game theory would be one component of that theory). Of course, the specific organisms we have on Earth with their specific quirks is not something we can describe by simple equations: unsurprisingly, since they are a rather arbitrary point in the space of all possible organisms!I do consider computational learning theory to be evidence for rationality realism. However, I think it’s an open question whether CLT will turn out to be particularly useful as we build smarter and smarter agents—to my knowledge it hasn’t played an important role in the success of deep learning, for instance.

It plays a minor role in deep learning, in the sense that some “deep” algorithms are adaptations of algorithms that have theoretical guarantees. For example, deep Q-learning is an adaptation of ordinary Q-learning. Obviously I cannot

*prove*that it is possible to create an abstract theory of intelligence without actually creating the theory. However, the same could be said about any endeavor in history.It may be analogous to mathematical models of evolution, which are certainly true but don’t help you build better birds.

Mathematical models of evolution might help you to build better

*evolutions*. In order to build better birds, you would need mathematical models of birds, which are going to be much more messy.This feels more like a restatement of our disagreement than an argument. I do feel some of the force of this intuition, but I can also picture a world in which it’s not the case.

I don’t think it’s a mere restatement? I am trying to show that “rationality realism” is what you should expect based on Occam’s razor, which is a fundamental principle of reason. Possibly I just don’t understand your position. In particular, I don’t know what epistemology is like in the world you imagine. Maybe it’s a subject for your next essay.

Note that most of the reasoning humans do is not math-like, but rather a sort of intuitive inference where we draw links between different vague concepts and recognise useful patterns

This seems to be confusing between objects and representations of objects. The assumption there is some mathematical theory at the core of human reasoning does

*not*mean that a description of this mathematically theory should automatically exist in the conscious, symbol-manipulating part of the mind. You can have a reinforcement learning algorithm that is perfectly well-understood mathematically, and yet nowhere inside the state of the algorithm is a description of the algorithm itself or the mathematics behind it.There may be questions which we all agree are very morally important, but where most of us have ill-defined preferences such that our responses depend on the framing of the problem (e.g. the repugnant conclusion).

The response might depend on the framing if you’re asked a question and given 10 seconds to answer it. If you’re allowed to deliberate on the question, and in particular consider

*alternative*framings, the answer becomes more well-defined. However, even if it is ill-defined, it doesn’t really change anything. We can still ask the question “given the ability to optimize any utility function over the world*now*, what utility function should we choose?” Perhaps it means that we need consider our answers to ethical questions provided a randomly generated framing. Or maybe it means something else. But in any case, it is a question that can and should be answered.

Machine learning uses data samples about an unknown phenomenon to extrapolate and predict the phenomenon in new instances. Such algorithms can have provable guarantees regarding the quality of the generalization: this is exactly what computational learning theory is about.

*Deep*learning is currently poorly understood, but this seems more like a result of how young the field is, rather than some inherent mysteriousness of neural networks. And even so, there is already some progress. People have been making buildings and cannons before Newtonian mechanics, engines before thermodynamics and ways of using chemical reactions before quantum mechanics or modern atomic theory. The fact you can do something using trial and error doesn’t mean trial and error is the only way to do it.

I don’t know a lot about the study of matrix multiplication complexity, but I think that one of the following two possibilities is likely to be true:

There is some and an algorithm for matrix multiplication of complexity for any s.t. no algorithm of complexity exists (AFAIK, the prevailing conjecture is ). This algorithm is simple enough for human mathematicians to find it, understand it and analyze its computational complexity. Moreover, there is a mathematical proof of its optimality that is simple enough for human mathematicians to find and understand.

There is a progression of algorithms for lower and lower exponents that increases in description complexity without bound as the exponent approaches from above, and the problem of computing a program with given exponent is computationally intractable or even uncomputable. This fact has a mathematical proof that is simple enough for human mathematicians to find and understand.

Moreover, if we only care about having a polynomial time algorithm with

*some*exponent then the solution is simple (and doesn’t require any astronomical coefficients like Levin search; incidentally, the algorithm is also good enough for most real world applications). In either case, the computational complexity of matrix multiplication is*understandable*in the sense I expect intelligence to be understandable.So, it is possible that there is a relatively simple and effective algorithm for intelligence (although I still expect a lot of “messy” tweaking to get a good algorithm for any specific hardware architecture; indeed, computational complexity is only defined up to a polynomial if you don’t specify a model of computation), or it is possible that there is a progression of increasingly complex and powerful algorithms that are very expensive to find. In the latter case, long AGI timelines become much more probable since biological evolution invested an enormous amount of resources in the search which we cannot easily match. In either case, there should be a theory that (i) defines what intelligence is (ii) predicts how intelligence depends on parameters such as description complexity and computational complexity.

Crossposted from SSC comments sectionI haven’t read Kuhn and I don’t know whether I’m interpreting em correctly, but to me it seems not that simple at all.

Saying there is an objective reality doesn’t explain why this reality is

comprehensible. In statistical learning theory there are various analyses of what mathematical conditions must hold for it to be possible to learn a model from observations (i.e. so that you can avoid the no-free-lunch theorems) and how difficult it is to learn it, and when you add computational complexity considerations into the mix it becomes even more complicated. Our understanding of these questions is far from complete.In particular, our ability to understand physics seems to rely on the

hierarchicalnature of physical phenomena. You can discover classical mechanics without knowing anything about molecules or quantum physics, you can discover atomic and molecular physics while knowing little about nuclear physics, and you can discover nuclear and particle physics without understanding quantum gravity (i.e. what happens to spacetime on the Planck scale). If the universe was s.t. it is impossible to compute the trajectory of a tennis ball without string theory, we might have never discovered any physics.