◉ _ ◉
Horosphere
Karma: 76
“Math, in an important sense, doesn’t exist when it’s not being done.” Can you justify this claim? What is the important sense? Would this claim not imply that the existence of mathematics is in principle dependent on one’s velocity because of special relativity? Do you think the same is true of logic? Are you a mereological nihilist? Can unconscious beings do mathematics? Sorry for all these questions, please don’t feel obliged to answer them all, but I am concerned that the belief that ‘Mathematics isn’t real other than as a human activity/mental activity’ is being propagated without being aptly defined!
“those symbols seem to come to take on meaning only because they get grounded by how they are used,” I would argue that they don’t need to be applied to anything other than pure mathematics in order to take on meaning. Therefore they are not grounded in empiricism, even if our understanding of them tends to be related to it.
It seems implausible that a superintelligent being would simultaneously believe that something was the wrong thing to do and then do that thing; the reason why humans do this is because we have the intelligence required to host multiple instrumental utility functions without being able to perfectly coherently integrate all of them into a global terminal utility function. This seems like an ideosyncracy of human-level intelligences.
What about mathematical concepts which are made out of their relationships with one another, for example that of a vector?
Although humans almost certainly learn language by observing examples of referents of words, this doesn’t explain how large language models with access only to text learnt the meanings of the same words. I would argue that the meaning of these words was contained in the way in which they were embedded in a semantic web of other words, allowing the LLM to learn the meanings of all of them simultaneously. Similarly, even if a human first learnt about vectors by seeing physical displacements between objects denoted by arrows, it would be possible to learn about vectors by defining them as entities which can be added together with other vectors in a particular way, etc.
Thanks for your detailed response.
“I think you said that not only do you want critical examination of arguments and counterarguments, you actually want a proof that it is absolutely not possible, because there is always a “sufficiently painful torture” that is more important than anything else.”
I can’t remember exactly what I wrote, but, while of course I would ideally want a proof, I would be satisfied with a merely very powerful counterargument. Since asking this question a few months ago and, in my opinion, not really receiving one, my guess is that no one on LessWrong (or anywhere else) knows one, but I’d obviously like to be proven wrong about that.
… it would take some degree of bad luck to end up in a future where the AI is actively hurting you and not just indifferently steamrolling you. This allows people to focus on x-risks and on s-risks that are dystopias, but not on s-risks that are hells.
I will say that in the absence of basilisky things, I would agree with this statement. The problem is that the basilisk has a predictable reason to potentially create a hell—like situation.
“But I will say one more time, that unless you’re a professional decision theorist concerned with timeless decision theories and so forth, preoccupation with the basilisk is a bad use of your time.”
I wouldn’t be preoccupied with it if at any point I had a choice to not learn about it, while knowing the danger. Unfortunately, I think I heard about the basilisk, dismissed it for CDT related reasons with a sense of unease, and then learnt about FDT, at which point it made sense.
“It’s being afraid of something entirely hypothetical, when there are concerns that are both more concrete and far more urgent. The main merit of thinking about the basilisk, is that as a thought experiment, it may stimulate progress in some abstract but fundamental areas like multiverse epistemology; and ideally we would have enough expert division of labor that the required progress could be achieved, without the basilisk haunting the general population.”
That would be great, but it almost certainly does haunt the general population to some extent… I have thought about whether an ASI would prefer to be constructed in a less ‘parochial’ way (i.e., not ideosyncratically of the earth, or any other planet in any other galaxy) , so that it would be more representative of the class of ASIs produced by all evolved intelligences which had the same idea, thereby forming a coalition to reward, or not punish, creators who opted to create ‘non-parochial ASI’ . Even if any ASI necessarily converges on ‘non-parochial ASI’, doing so might cost it valuable time. This seems like a non-x-risk increasing way to do something a basilisk would want.
I think the main reasons why I haven’t tried to explicitly reason through the basilisk argument as it applies to specified possible worlds are because I expect that:
1) the proportions of possible worlds which are a particular way are important, so no one of them alone is sufficient to make a decision.
2) I don’t want to become more deeply entangled.
“This is all a dreary topic for me because there are other issues associated with alignment of superintelligence which I regard as much more real and urgent. But as I said, I think investigating these scenarios could produce progress on topics like multiverse epistemology, and the viability of “acausal interaction” in any form, and that does give the field of Basilisk Studies some justification.”
I agree, however even if I could contribute to alignment, I wouldn’t because of the basilisk.
I understand if because of the nature of the topic, you don’t want to continue talking about it now, but if you do reply, I would want to know what you meant by the following:
″ is it ever actually rational to threaten (or punish) a copy of an entity from the past, on the grounds that the original envisioned that it might have a future duplicate?”
It seems to me that the way in which the logical interaction is embedded in a causal world shouldn’t prevent it from being rationally justified. Is there a reason why it might that has escaped me?
“If you look into the history of non Euclidean spaces, the mathematicians involved did not have any intuition that they were dealing with a better physical theory of space, that insight came from the physics side...promoted by empirical results.”
I understand that this was not addressed to me, but I wanted to say that I don’t think this is actually true, as some of the mathematicians developing an understanding of non-euclidean space anticipated that physical space was curved, and even that its curvature could explain gravity.
“They are relatively simple because they describe reality very inefficiently.”
What do you mean by efficiency? If you mean that it’s extremely difficult to use them to make predictions, then I would agree, but prefer to refer to the simplicity as causing this inefficiency rather than the other way around.
″ An axiom system where most mathematically valid axioms are also physically true would be much more complex. ”
I don’t think this is conceivable because some axioms would contradict others. But apart from that, I agree with this statement, and I think the fact we exist in a very simple part of the ‘mathematical universe’, if in fact we do, can be partially explained by the fact that simpler axiom systems are in a sense ‘bigger’ in the ‘mathematical universe’ than more complicated ones, because they
can proveallow more theorems.
Of course the meaning is important, however not all words can be defined in terms of others, and this is exactly what providing a ‘gears level model’ would require. Have you thought of a way to do this in the case of concepts like reality and existence?
I would guess that you distinguish between platonic reality, physical reality, perhaps simulated or thought reality, and maybe others. However I would consider all of these to be real, and it is clearly impossible for something to be real in the physical or other sense while not existing in the platonic sense. Therefore, when mathematicians claim that mathematics has platonic existence, I would say they’re justified in saying it exists (in the most general sense) . It then makes sense to say that they discover it when they become aware of it.
Thanks for engaging with the post.
“Are specific humans currently being acausally blackmailed? Either by Roko’s idea or by something similar. This would be an empirical claim, and finding the humans in question would be the best approach.” If by this you mean ‘are there any humans who are attempting to appease the basilisk, regardless of whether it exists?’ then I would say yes. You could even argue that this post is my attempt to do so given the uncertainty and that I think it’s in the best interests of others who have thought about the argument in sufficient depth.
If you mean ‘Are there any humans actually being simulated by the basilisk, or existing in a form which will be simulated?’ , then I don’t claim to be able to settle this question in the post; what I do suggest is that this can’t be ruled out or consigned to negligible probabilities.
“Would it be rational for a non-human agent (because humans are not sufficiently well-modeled to answer this question of them) to change it’s behavior for this kind of acausal trade (trade and blackmail being indistinguishable in pure logic)?”
I think this would again depend upon whether the logic actually ‘works’ in the situation in which it would need to be thought, in addition to the utility function of the agent (For example if its utility function is symmetrical in the way I describe in the post, it might make sense to ignore the basilisk.)
I would also note that I expect a sufficiently more intelligent being than a human would have ways of preventing itself from thinking about many, although probably not all, acausal extortion scenarios before becoming entangled in the logic. (Though this is just a guess.)
“It seems to be about arguments, not about actual reality. I think it’s not quite the right approach for either question you might be asking” The thing is, in this case, arguments might have a way to influence ‘physical reality’ , so constraining oneself only to thinking about the latter might be a mistake, as I argue it is in the post. If you want to avoid thinking about these arguments, you might need to discard timeless decision theory.
Do you have any idea as to how one would accomplish this before the advent of superintelligent AI?
( And preferably without further dooming oneself by thinking about it in sufficient detail?)
What do you mean by “this kind of disagreement” ? Edit: Also, what do you mean by “shape”? It seems to me that arguments involving infinities are of a qualitatively different ‘form’ from those involving finite or infinitesimal numbers. If this is what you mean by “shape”, then it seems to be the same thing as magnitude in this particular case.
The word real is rather important and I don’t think the conversation can proceed without it unless it is replaced with something else. I would suggest ‘existent’ as such a replacement.
“What is meant by “follows” is not unconditional. Implication is itself conditional on logical axioms.”
In order for humans to understand implication, it is helpful, or even necessary, for us to denote and specify the logical axioms using symbols and manipulation rules, but this does not mean that they are invented by humans ( though I understand you didn’t claim that they were). Rather, I would claim that they were discovered to be ways to accurately represent pieces of Logic. Logic is simply the necessary structure of relations between things. So I would argue that logic itself is unconditionally existent.
Alternatively, you could describe logical deduction as a form of computation, which necessarily plays out in a particular way. It is unconditionally true that a particular algorithm produces a particular output given a particular input, therefore it is unconditionally true that certain theorems follow from certain axioms, because the ‘conditions’ , which here are both the input of the algorithm and the information describing the algorithm itself, have been incorporated into the algorithm+input ensemble. You might then object that information is also necessary to specify the computer running the algorithm, but this can itself be conceptualized as part of the algorithm.
“Our brains evolved in the reality and developed the kind of reasoning that is useful for it. No more explanation is required.” I am not sure that this is true, unless stated within the context of the ‘mathematical universe hypothesis’ and anthropic principle as I mentioned, in which case I don’t think we disagree with each other. Someone might object that we didn’t evolve to understand counterintuitive ideas like Relativity and Quantum mechanics because we live in a regime in which Newtonian mechanics describes what we observe very well, and is simpler. Therefore we evolved only to understand Newtonian mechanics.
Update for Ape in the coat if you read this thread again:
I am rate limited and so can’t currently respond directly to you. My reply would have been as follows:
“Of course the meaning is important, however not all words can be defined in terms of others, and this is exactly what providing a ‘gears level model’ would require. Have you thought of a way to do this in the case of concepts like reality and existence? ”
Understandable, analogical reasoning can be very powerful and I often want to apply it before I notice that the analogy might not be exact.
“I agree that before the development of cheap widespread compute such a conclusion on the likely simplicity of the generating rules of the universe would be in the reverse (again with the Deism).”
Indeed, people argued, as far as I know (from the Wikipedia page about Isaac Newton) that god was the only parsimonious explanation for the diversity and simultaneous ‘orderedness’ of lifeforms. I would guess that this is because minds were the only well known and appreciated ‘arenas/venues’ of computation [1], so it was thought to be necessary to posit that the creator of the universe must be some kind of mind, whereas today we understand that minds are not necessarily distinct classes of computation.
- ^
By ‘computation’ here, I really mean sources of partially comprehensible complex patterns, which we now know are often computational. So it would be more accurate to say that a ‘mind’ would be thought of in the past as something containing much more information than we now think it needs to in order to generate its complex outputs. Since these were the only known examples of such sources of complexity, they were considered to be necessary parts of its explanation.
- ^
This post is interesting, however it seems to be a string of reasonable seeming claims followed by a conclusion they don’t obviously support . I would say mathematics exists as a mathematician-independent entity because each of the properties listed here is either a property of the process by which humans discover mathematics, or a human approximation or heuristic for something which probably does have an independent platonic kind of existence. It does not suggest to me that mathematics is invented. I would, however, make an exception in case humans invent physical objects and systems and processes and these are miniscule portions of the mathematical universe if the universe is fundamentally mathematical.
“I disagree that this is the kind of simplicity meant when pointing out the unreasonable effectiveness of math. If it is then I believe this line of thought has a glaring hole. Simple systems do not imply simple dynamics in general. Emergence exists. An elementary cellular automata can generate fractals from 8 binary rules, and a simple set of axioms is not held back by this simplicity when it comes to what that those axioms generate.
So the simplicity of the reality describing math generating axioms is interesting, but not surprising.”
That’s why I softened my designation to notable; but I think it could be surprising to someone who took a completely empirical point of view and doubted that anything so complex could emerge from pure logic, or was living at a point in history before computers had existed for as long as they have now and made the power of emergence apparent.
“I’d argue the simple set of axioms is more likely to generate intelligence. The more moving parts you have the more likely something is going to break. … the chaotic system in which there exists temporally stable basins of attraction.” I agree because I think our empirical exploration of physics has demonstrated that we live in a relatively simple part of the platonic mathematical world.
“But then again this is a bit circular isn’t it? Using dynamical systems (math) to argue the dynamics of mathematical formalism and all.”
Perhaps, but I don’t fully understand your interpretation of the argument, so I can’t comment. I would point out that (almost) self referential arguments can be perfectly legitimate.
“Math Is a Language, Not a Story” I would not necessarily disagree with this, but would want to point out that it could equally well be described as a set of many languages, or as the structure of logic. Algebra is certainly a family of languages. It is not clear that, for example, Pythagorean theorem is best conceived of as a statement in any single language, as it can be proven within different axiom systems.
“It’s generalized conditional knowledge, not fundamentally free from the uncertainty but merely outsourcing it to the moment of applicability. As a result math can’t actually prove anything about real world with perfect confidence.” I know I already commented this on the post from which this statement is taken, but this implies that mathematics itself is not real, which has not been demonstrated at all. In addition, even though that mathematical theorems are true is conditional on the axioms with respect to which they are proven, it is unconditionally true that, for example, the inscribed angle theorem follows from the axioms of Euclidean geometry.
You say: ” And I understand why it may feel magical. But this is a magic rooted in empiricism about our physical reality. Only empiricism can reduce the improbability of a particular part of reality fitting specific mathematical axioms. Moreover, our ability to reason about properties of the universe and generalize patterns is the result of evolving inside this universe that has such properties in the first place.”
However, empiricism doesn’t explain the relative simplicity of the mathematical axioms required to describe reality. If we live within mathematics, as in the ‘mathematical universe hypothesis’, then this may be explained by the inherent structure and combinatorial explosiveness of the number of theorems which are true within axiom systems relative to the information required to specify them, along with the anthropic principle which requires us to live within a region of the mathematical universe which can accommodate intelligent beings of around our level of complexity. Under this interpretation, empirical observation serves to locate us within the mathematical universe.
It’s this simplicity which scientists find to be ‘unreasonable’ about the ‘effectiveness of mathematics in the natural sciences’ , or rather the immense ratio of the complexity explained by laws of physics formulated in mathematics derived from relatively simple axiom systems to the simplicity of those axioms. Although I don’t necessarily know whether I’d want to describe it as unreasonable, I think it’s fair to say it’s certainly notable and indicative that we are in a ‘mathematical universe’.
“I don’t think they could become good at designing batteries, at least if you mean “make a battery design from scratch”, because unsurprisingly in my ontology that skill is “design” and a CS degree does not usually transfer any design skill.” I see, I think I was imagining that in your ontology this skill was needed in research: you mentioned in another comment that a lot of research skill is ‘design’.
“I think they could totally start making contributions to differential geometry at roughly the level where they made contributions in computer science” If this was true, I would expect there to be more modern Von Neumanns. If you are a +4 SD computer scientist, and you do the same for several disparate areas of mathematics, as well as physics, and chemistry, and biology, in a few years or decades, you become comparable to Terence Tao almost in mathematics, just with less quality in your individual contributions, but you can in some sense compensate by contributing to all the other areas. Why is this so incredibly rare?
“The best evidence I can provide here is all the evidence about verbal intelligence being a meaningful sub-factor of g, and that lawyers do score substantially higher on that than physicists, after controlling for g.” I don’t have any data of my own so I’ll take your word.
“We measure the skill by measuring vocabulary and ability to make and follow clear arguments, which I think point vaguely in this direction.” I’ve never heard of a verbal reasoning test involving making and interpreting/understanding arguments. I am actually curious about this, as it seems hard to measure directly; I thought verbal IQ tests used things like simple analogical reasoning tests. How would such a test work?
“It obviously would be hella confusing to use the word as I am using it here without defining it. But I am literally using the phrase “design (in this ontology)” in the quoted passage.” This one: “ontologizing new domains and thinking really hard about what are good new abstractions to think in within this domain”?
“Wait, I think this is a highly controversial take! Indeed I think when I present it to professionals in most fields they bark at this pretty frequently.” I would actually like to weaken the claim to refer to the best person in any reasonably broad field, so that they can’t be, for example, the best person in the world at approximating solutions to differential equations of the form y‴- cube root(x’-z) / log(yz) = 9 . And by field, I meant what you described as a STEM/ mathematical -intelligence-requiring field.
“The thing I am saying is that if you took a reasonably good theoretical computer scientist, then they could become a reasonably good mathematician, or engineer, or software developer, within 6 months if they try really hard. They wouldn’t be a better mathematician than they were a computer scientist, I am not proposing there is some magical “get the best in any field” button.” I think it would depend a lot on the area into which they chose to specialize. For example, if they chose some computer science adjacent area of set theore, then I find the claim more plausible, although I am still unsure. If they chose differential geometry, I would seriously doubt the claim. I would also doubt that they could become good at designing batteries.
When AI becomes completely general, human intelligence will be obsolete.