royf

• royfJun 6, 2012, 9:26 PM

  0 points

  in reply to: Ronny Fernandez’s comment on: Fake Causality

  if it [...] has a false conclusion, you should forget about the reasoning altogether

  That’s both obvious and irrelevant.

  [...] either you are too confident in your premises [...]

  the premise itself could be false, and I suspect that it is

  Are you even trying to have a discussion here? Or are you just stating obvious and irrelevant facts about rationality?

• royfJun 6, 2012, 4:07 AM

  0 points

  in reply to: Ronny Fernandez’s comment on: Fake Causality

  If you’re going to use it as evidence for your conclusion, or part of your worldview, you should really be sure that it’s true

  (I never saw this claim discussed seriously, so I don’t know how factual it is; but the logic of it is what I’m getting at.)

  Was my disclaimer insufficient? I was using the unchecked claim to convey a piece of reasoning. The claim itself is unimportant in this context, only its reasoning that its conclusion should follow from its premise. Checking the truth of the conclusion may not be difficult, but the premise itself could be false, and I suspect that it is, and that it’s much harder to verify.

  And even the reasoning, which is essentially mathematically provable, I have repeatedly urged the skeptic reader to doubt until they see a proof.

  using “logic” that leads to empirically falsifiable claims—is essentially never fruitful.

  Did you mean false claims? I sure do hope that my logic (without quotes) implies empirically flasifiable (but unfalsified) claims.

• royfJun 5, 2012, 9:50 PM

  0 points

  in reply to: CuSithBell’s comment on: Fake Causality

  Suppose an AI were to design and implement more efficient algorithms for processing sensory stimuli? Or add a “face recognition” module when it determines that this would be useful for interacting with humans?

  The ancient Greeks have developed methods of improved memorization. It has been shown that human-trained dogs and chimps are more capable of human-face recognition than others of their kind. None of them were artificial (discounting selective breeding in dogs and Greeks).

  It seems that you should be able to write a simple program that overwrites its own code with an arbitrary value. Wouldn’t that be a counterexample?

  Would you consider such a machine an artificial intelligent agent? Isn’t it just a glorified printing press?

  I’m not saying that some configurations of memory are physically impossible. I’m saying that intelligent agency entails typicality, and therefore, for any intelligent agent, there are some things it is extremely unlikely to do, to the point of practical impossibility.

  Do we agree, then, that humans and artificial agents are both subject to laws forbidding logical contradictions and the like, but that artificial agents are not in principle necessarily bound by the same additional restrictions as humans?

  I would actually argue the opposite.

  Are you familiar with the claim that people are getting less intelligent since modern technology allows less intelligent people and their children to survive? (I never saw this claim discussed seriously, so I don’t know how factual it is; but the logic of it is what I’m getting at.) The idea is that people today are less constrained in their required intelligence, and therefore the typical human is becoming less intelligent.

  Other claims are that activities such as browsing the internet and video gaming are changing the set of mental skills which humans are good at. We improve in tasks which we need to be good at, and give up skills which are less useful. You gave yet another example in your comment regarding face recognition.

  The elasticity of biological agents is (quantitatively) limited, and improvement by evolution takes time. This is where artificial agents step in. They can be better than humans, but the typical agent will only actually be better if it has to. Generally, more intelligent agents are those which are forced to comply to tighter constraints, not looser ones.

• royfJun 5, 2012, 3:49 AM

  0 points

  in reply to: wedrifid’s comment on: Fake Causality

  That’s a big “ouch” on my part. Sorry. Lesson learned.

• royfJun 5, 2012, 3:09 AM

  0 points

  in reply to: wedrifid’s comment on: Fake Causality

  I wasn’t basing it on popular culture, but that doesn’t mean I’m not wrong.

  Do you have a better suggestion?

  If not, I’d ask CuSithBell to please clarify her (or his) ideas without using controversially defined terminology (which was also my sentiment before).

• royfJun 5, 2012, 2:56 AM

  0 points

  in reply to: Kindly’s comment on: Fake Causality

  I don’t think “surviving” is a well-defined term here. Every time you self-modify, you replace yourself with a different agent, so in that sense any agent that keeps surviving is one that does not self-modify.

  I placed “survive” in quotation marks to signal that I was aware of that, and that I meant “the other thing”. I didn’t realize that this was far from clear enough, sorry.

  For lack of better shared terminology, what I meant by “surviving” is continuing to be executable. Self modification is not suicide, you and I are doing it all the time.

  Can I write a program that begins by computing the cluster of all agents similar to it, and switches to the next one (lexicographically) every 24 hours?

  No, you cannot. This function is non-computable in the Turing sense.

  A computable limited version of it (whatever it is) could be possible. But this particular agent cannot modify itself “in any way it wants”, so it’s consistent with my proposition.

  The natural objection is that there is one part of the agent’s state that is inviolate in this example: the 24-hour rotation period

  This is a very weak limitation of the space of possible modifications. I meant a much stronger one.

  But I’m skeptical that this limitation can be encoded mathematically.

  This weak limitation is easy to formalize.

  The stronger limitation I’m thinking of is challenging to formalize, but I’m pretty confident that it can be done.

• royfJun 5, 2012, 2:19 AM

  1 point

  in reply to: Kindly’s comment on: Fake Causality

  Rewrite its own source code with a random binary string

  This is in a sense the electronic equivalent of setting oneself on fire—replacing oneself with maximum entropy. An artificial agent is extremely unlikely to “survive” this operation.

  any possible program can be obtained as a result

  Any possible program could be obtained, and the huge number of possible programs should hint that most are extremely unlikely to be obtained.

  I assumed we were talking about an agent that is active and kicking, and with some non-negligible chance to keep surviving. Such an agent must have a strongly non-uniform distribution over its next internal state (code included). This means that only a tiny fraction of possible programs will have any significant probability of being obtained. I believe one can give a formula for (at least an upper bound on) the expected size of this fraction (actually, the expected log size), but I also believe nobody has ever done that, so you may doubt this particular point until I prove it.

• royfJun 5, 2012, 1:47 AM

  0 points

  in reply to: TheOtherDave’s comment on: Fake Causality

  As I said, when it secretes serotonin into its bloodstream and synapses.

• royfJun 4, 2012, 11:27 PM

  0 points

  in reply to: CuSithBell’s comment on: Fake Causality

  Thanks for challenging my position. This discussion is very stimulating for me!

  Sure, but we could imagine an AI deciding something like “I do not want to enjoy frozen yogurt”, and then altering its code in such a way that it is no longer appropriate to describe it as enjoying frozen yogurt, yeah?

  I’m actually having trouble imagining this without anthropomorphizing (or at least zoomorphizing) the agent. When is it appropriate to describe an artificial agent as enjoying something? Surely not when it secretes serotonin into its bloodstream and synapses?

  This seems trivially false—if an AI is instantiated as a bunch of zeros and ones in some substrate, how could Godel or similar concerns stop it from altering any subset of those bits?

  It’s not a question of stopping it. Gödel is not giving it a stern look, saying: “you can’t alter your own code until you’ve done your homework”. It’s more that these considerations prevent the agent from being in a state where it will, in fact, alter its own code in certain ways. This claim can and should be proved mathematically, but I don’t have the resources to do that at the moment. In the meanwhile, I’d agree if you wanted to disagree.

  You see reasons to believe that any artificial intelligence is limited to altering its motivations and desires in a way that is qualitatively similar to humans? This seems like a pretty extreme claim—what are the salient features of human self-rewriting that you think must be preserved?

  I believe that this is likely, yes. The “salient feature” is being subject to the laws of nature, which in turn seem to be consistent with particular theories of logic and probability. The problem with such a claim is that these theories are still not fully understood.

• royfJun 4, 2012, 6:13 AM

  3 points

  in reply to: CuSithBell’s comment on: Fake Causality

  I am saying pretty much exactly that. To clarify further, the words “deliberate”, “conscious” and “wants” again belong to the level of emergent behavior: they can be used to describe the agent, not to explain it (what could not be explained by “the agent did X because it wanted to”?).

  Let’s instead make an attempt to explain. A complete control of an agent’s own code, in the strict sense, is in contradiction of Gödel’s incompleteness theorem. Furthermore, information-theoretic considerations significantly limit the degree to which an agent can control its own code (I’m wondering if anyone has ever done the math. I expect not. I intend to look further into this). In information-theoretic terminology, the agent will be limited to typical manipulations of its own code, which will be a strict (and presumably very small) subset of all possible manipulations.

  Can an agent be made more effective than humans in manipulating its own code? I have very little doubt that it can. Can it lead to agents qualitatively more intelligent than humans? Again, I believe so. But I don’t see a reason to believe that the code-rewriting ability itself can be qualitatively different than a human’s, only quantitatively so (although of course the engineering details can be much different; I’m referring to the algorithmic level here).

  Generally GAIs are ascribed extreme powers around here

  As you’ve probably figured out, I’m new here. I encountered this post while reading the sequences. Although I’m somewhat learned on the subject, I haven’t yet reached the part (which I trust exists) where GAI is discussed here.

  On my path there, I’m actively trying to avoid a certain degree of group thinking which I detect in some of the comments here. Please take no offense, but it’s phrases like the above quote which worry me: is there really a consensus around here about such profound questions? Hopefully it’s only the terminology which is agreed upon, in which case I will learn it in time. But please, let’s make our terminology “pay rent”.

• royfJun 4, 2012, 5:08 AM

  0 points

  in reply to: CuSithBell’s comment on: Fake Causality

  I believe that is a misconception. Perhaps I’m not being reasonable, but I would expect the level at which you could describe such a creature in terms of “desires” to be conceptually distinct from the level at which it can operate on its own code.

  This is the same old question of “free will” again. Desires don’t exist as a mechanism. They exist as an approximate model of describing the emergent behavior of intelligent agents.

• royfJun 4, 2012, 4:51 AM

  0 points

  in reply to: wedrifid’s comment on: Fake Causality

  Having asserted that your claim is, in fact, new information: can you please clarify and explain why you believe that?

• royfJun 4, 2012, 3:53 AM

  −2 points

  on: Fake Causality

  Sadly, we humans can’t rewrite our own code, the way a properly designed AI could.

  Sure we can!
  In fact, we can’t stop rewriting our own code.

  When you use the word “code” to describe humans, you take a certain degree of semantic liberty. So we first need to understand what is meant by “code” in this context.

  In artificial computing machines, code is nothing more than a state of a chunk of memory hardware that causes the computation hardware to operate in a certain way (for a certain input). Only a tiny subset of the possible states of any chunk of memory hardware are “executable”, i.e. don’t cause the computation hardware to reach a certain “failure” state. This gives us an almost clear-cut distinction between (executable) code and (non-executable) data, under the assumption that data is very unlikely to be executable by chance. Given the correct design, a machine can write code to its memory and then execute it.

  In humans, the distinction between memory hardware and computation hardware is unclear, if it exists at all. Moreover, it’s unclear how to apply the above distinction between code and data: what is a human’s “failure” state? I guess a state of the brain (containing both memory and computation hardware, until and unless we can ever separate the two) can be said to be “executable” if, placed in a certain environment, the person doesn’t go and die.

  It follows that any change that the brain does to its own state, which then affects its computation, to the result of not dying, is, in fact, “rewriting its own code”. This, of course, happens all the time and (perhaps ironically) cannot be stopped without killing the brain.

  In a wider loop, we also have drugs, medications and, eventually, gene therapy. But that’s more similar to a robot reaching for the keyboard (or the screwdriver).

• royfJun 4, 2012, 2:45 AM

  0 points

  on: Fake Causality

  This is a great layperson explanation of the belief propagation algorithm.

  However, the phlogiston example doesn’t show how this algorithm is improperly implemented in humans. To show this, you need an example of incorrect beliefs drawn from a correct model, i.e. good input to the algorithm resulting in bad output. The phlogiston model was clearly incorrect. As other commenters have pointed out, contemporary scientists were painfully aware of this, and have eventually abandoned the model. Bad output from bad input doesn’t demonstrate a bug in implementation, certainly not the specific bug you mentioned:

  we don’t keep rigorously separate books for the backward-message and forward-message

  Such a defect would probably not even allow a mouse to be as intelligent as it is.

• royfJun 2, 2012, 7:35 AM

  3 points

  in reply to: Dmytry’s comment on: Fake Explanations

  I agree that the word “why” is perhaps not the best choice, but it really stands for “what is the physical mechanism for”.

  Anyway, what you describe is engineering, not science. A scientist explains the mechanism by which a stone ax can kill a mammoth. An engineer uses this understanding to design a better ax. A craftsman builds the ax. A warrior takes the ax, kills the mammoth, and wins the girl.

• royfJun 2, 2012, 7:13 AM

  10 points

  1

  in reply to: Eliezer Yudkowsky’s comment on: Fake Explanations

  You are overstating the case by a large margin.

  [Saying “I don’t know”] is still the rational thing to say when, in fact, you don’t know.

  Saying “I don’t know” may be, to a large degree, the true state of your belief when you use probability theory. But in this case it’s not the rational thing to say when you use decision theory. “I don’t know” is true, but it is a non-answer to the question, and doesn’t get you points. It’s a different matter whether this point system is effective or moral, but as long as it’s there, that’s what you play by.

  You can easily do worse than maximum entropy if you guess at random.

  If you base your guess correctly on an incomplete model of reality, which you’ve constructed correctly from past observations, you can never do worse, on average, than maximum entropy. More evidence can never lead to less information (as per the Data Processing Inequality).

  Furthermore, “getting it right” [...] does not necessarily mean that you possess any anticipation-controllers.

  On the contrary, it mean exactly that. Being rewarded for predictive powers improves your model of the world, whereas “I don’t know” is an excuse for not knowing.

  In fact, the mechanism employed by the teacher, for all its flaws, achieves 3 important goals:

  • It motivates students to pay attention, raises their level of alertness, activates their brains.

  • It rewards students to engage their past observations to generate the most accurate belief they can about the right answer. In the process, they build a better model of the world, and they make their unknown unknowns a little more known.

  • By forcing students to generate a belief and commit to it before the correct answer is revealed, their hindsight bias is reduced.

• royfJun 1, 2012, 4:39 AM

  −1 points

  0

  in reply to: gwern’s comment on: Con­ser­va­tion of Ex­pected Evidence

  (I’ll adopt the seemingly rationalist trait of never taking questions as rhetorical, though both your questions strongly have that flavor).

  A central part of the modern scientific method is due to Popper, who gave an essentially Bayesian answer to your first question. However, Science wouldn’t fall apart if it turned out that priors aren’t a physical reality. Occam’s razor is non-Bayesian, and it alone accounts for a large portion of our scientific intuitions. At the bottom line, the scientific method doesn’t have to be itself true in order to be effective in discovering truths and discarding falsehoods.

  The concept of “proving a paradox” is unclear to me (almost a paradox in itself...). Paradoxes are mirages. Also, it seems that you have some specific piece of scientific history in mind, but I’m uncertain which.

  Luckily, we did have Jaynes and others to promote what I believe to be both a compelling mathematical framework and a physical reality. Before them, well, it would be wishful to think I could hold on to Bayesian ideas in the face of apparent paradoxes. The shoulders of giants etc.

• royfJun 1, 2012, 2:56 AM

  1 point

  0

  in reply to: Tom_McCabe’s comment on: Con­ser­va­tion of Ex­pected Evidence

  [Bayesianism i]s mathematically proven.

  More importantly, it’s physically proven. The fact that the math is consistent (and elegant!) would not have been so powerful if it wasn’t also true, particularly since Bayesianism implies some very surprising predictions.

  Fortunately, it is the happy case that, to the best of my knowledge, no experiments thus far contradict Bayesianism, and not for the lack of trying, which is as much proof as physically possible.

• royfMay 31, 2012, 5:12 PM

  10 points

  0

  in reply to: thomblake’s comment on: Fo­cus Your Uncertainty

  This solution is the Kelly strategy. It isn’t generally optimal, but this makes it optimal in this case:

  if some particular outcome occurs, then your utility function is logarithmic in time spent preparing the excuse

• royfMay 28, 2012, 5:16 AM

  4 points

  on: How Much Ev­i­dence Does It Take?

  131,115,985 to 1 [...] this amount of evidence would only be enough to give you an even chance of winning the lottery.

  The number of false bleeps is distributed almost exactly Poisson with $\lambda =1$. The important figure is not the expected number of bleeps ($Ex1$, which is indeed 2). It’s the expected probability that a random bleep is the true one, $E\frac{1}{x1}$. At the moment I can’t find an analytic solution (and a short search suggests none is known), but a computation shows the result is around 63.2%, much better than 50%. Similarly, with 14 boxes (arguably “28 bits of evidence”), the chance of winning is about 79.1% on average, much better than $\frac{2}{3}$.