I think Bostrom uses the term “hardware overhang” in Superintelligence to point to a cluster of discontinuous takeoff scenarios including this one
Jalex Stark
Karma: 43
The restrictions are something like “real humans who generally want to be effective should want to use the method”.
Just for amusement, I think this theorem can fail when s, x, y represent subsystems of an entangled quantum state. (The most natural generalization of mutual information to this domain is sometimes negative.)
Rice’s theorem applies if you replace “circuit” with “Turing machine”. The circuit version can be resolved with a finite brute force search.
“In the presence of cosmic rays, then, this agent is not safe for its entire lifetime with probability 1.”
I think some readers may disagree about whether you this sentence means “with probability 1, the agent is not safe” or “with probability strictly greater than 0, the agent is not safe”. In particular, I think Hibron’s comment is predicated on the former interpretation and I think you meant the latter.
I think the most interesting part of the piece is the bit at the end where the author analyzes a misleading graph. Now that I understand the graph, it seems like like strong evidence towards malicious misrepresentation or willful ignorance on the part of some subset (possibly quite small) of the AlphaStar team.
I think the article might benefit from comparisons to OpenAI’s Dota demonstration. I don’t remember anyone complaining about superhuman micro in that case. Did that team do something to combat superhuman-APM, or is Starcraft just more vulnerable than Dota to superhuman-APM tactics?
That’s not true? If there are five authors selling 0, 0, 1, 2, and 3 books each, then the mode is 0 and the median is 1.
To state it more plainly, the claim “the median number of sales is zero” is equivalent to the claim “more than half of self-published ebooks sell zero copies”.
Adding to Vladimir_Nesov’s comment:
In general, every suborder of a well-order is well-ordered. In a word, the property of “being a well-order” is hereditary. (compare: every subset of a finite set is finite)
Yes, there are good ways to index sets other than well orders. A net where the index set is the real line and the function is continuous is usually called a path, and these are ubiquitous e.g. in the foundations of algebraic topology.
I guess you could say that I think well-orders are important to the picture at hand “because of transfinite induction” but a simpler way to state the same objection is that “tomorrow” = “the unique least element of the set of days not yet visited”. If tomorrow always exists / is uniquely defined, then we’ve got a well-order. So something about the story has to change if we’re not fitting into the ordinal box.
Yeah, I’d agree with the “boundary doesn’t exist” interpretation.
An ordered set is well-ordered iff every subset has a unique least element. If your set is closed under subtraction, you get infinite descending sequences such as . If your sequence is closed under division, you get infinite descending sequences that are furthermore bounded such as . It should be clear that the two linear orders I described are not well-orders.
A small order theory fact that is not totally on-topic but may help you gather intuition:
Every countable ordinal embeds into the reals but no uncountable ordinal does.
A net is just a function where is an ordered index set. For limits in general topological spaces, might be pretty nasty, but in your case, you would want to be some totally-ordered subset of the surreals. For example, in the trump paradox, you probably want to:
include and for some infinite
have a least element (the first day)
It sounds like you also want some coherent notion of “tomorrow” at each day, so that you can get through all the days by passing from today to tomorrow infinitely many times. But this is equivalent to having your set be well-ordered, which is incompatible with the property “closed under division and subtraction by finite integers”. So you should clarify which of these properties you want.
the ordinary notion of sequence
I assume here you mean something like “a sequence of elements from a set is a function where is an ordinal”. Do you know about nets? Nets are a notion of sequence preferred by people studying point-set topology.
My best guess about how to clear up confusion about “what the boundary looks like” is via mathematics rather than philosophy. For example, have you understood the properties of the long line?
I think gjm’s response is approximately the clarification I would have made about my question if I had spent 30 minutes thinking about it.
In “Trumped”, it seems that if , the first infinite ordinal, then on every subsequent day, the remaining number of days will be for some natural . This is never equal to .
Put differently, just because we count up to doesn’t mean we pass through . Of course, the total order on days has has for each finite , but this isn’t a well-order anymore so I’m not sure what you mean when you say there’s a sequence of decisions. Do you know what you mean?
In order to apply surreal arithmetic to the expected utility of world-states, it seems we’ll need to fix some canonical bijection between states of the world and ordinals / surreals. In the most general case this will require some form of the Axiom of Choice, but if we stick to a nice constructive universe (say the state space is computable) then things will be better. Is this the gist of what you’re working on?
I’m not sure whether I’ve understood the point you’re trying to make, in part because I don’t know the answer to the following question:
Does your point change if you replace “quantumness” with ordinary randomness?
We live in a world with large incentives to teach yourself to do something like this, so either it is too hard for a single person to come up with on their own or it is possible to find people that have done it.
Some military studies might fit what you’re looking for.