MMath Cambridge. Currently studying postgrad at Edinburgh. D.P.Hobson@sms.ed.ac.uk
Donald Hobson
Karma: 1,756
Darwinian evolution as such isn’t a thing amongst superintelligences. They can and will preserve terminal goals. This means the number of superintelligences running around is bounded by the number humans produce before the point the first ASI get powerful enough to stop any new rivals being created. Each AI will want to wipe out its rivals if it can. (unless they are managing to cooperate somewhat) I don’t think superintelligences would have humans kind of partial cooperation. Either near perfect cooperation, or near total competition. So this is a scenario where a smallish number of ASI’s that have all foomed in parallel expand as a squabbling mess.
When performing first aid, you must never leave a patient until you have passed them off to someone more qualified than yourself.
I think this is bad advice. Sometimes the patient has a small cut. A dab of antiseptic and a plaster and their treated. In a triage situation you might be rushing back and forth between several patients, trying to stem the bleeding until the ambulances arrive.
If you and a friend were out walking, and your friend broke their leg, you might want to attach some sort of crude splint, then leave your friend there as you walk for help. (if you have no phone or no signal)
Then there is “more qualified than yourself”. Does this mean the most qualified person in the world can’t ever pass the patient off to anyone? Even if its just an injury any novice could treat?
In many situations, appearing helping and then wandering off will be strictly better than doing nothing. (an action optimised towards the goal of helping is generally better than the null action)
Helping someone and then abandoning them can be worse than just being a bystander.
I’m sure this is sometimes the case, but its not always the case. I would say its the case <1/2 the time. Mostly trying to help someone should help them.
Turing machines are kind of the limit of finite state machines. There are particular Turing machines that can simulate all possible finite state machines. In general, an arbitrary sequence of finite state machines can be uncomputable. But there are particular primitive recursive functions that simulate arbitrary finite state machines, (given the value as an input. ) You don’t need the full strength of turing completeness to do that. So I would say, kind of no. There are systems strictly stronger than all finite automaton, yet not Turing complete.
Really, the notion of a limit isn’t rigorously defined here, so its hard to say.
Sometimes in mathematics, you can right 20 slightly different definitions and find you have defined 20 slightly different things. Other times you can write many different formalizations and find they are all equivalent. Turing completeness is the latter. It turns up in Turing machines, register machines, tiling the plane, Conways game of life and many other places. There are weaker and stronger possibilities, like finite state machines, stack machines and oracle machines. (Ie a Turing machine with a magic black box that solves the halting problem is stronger than a normal Turing machine)
Human brains are finite state machines. A Turing machine has unlimited memory and time.
Physical laws are generally continuous, but there exists a Turing machine that takes in a number N and computes the laws to accuracy 1/N. This isn’t philosophically forced, but it seems to be the way things are. All serious theories are computable.
We could conceivably be in a universe that wasn’t simulateable by a Turing machine. Assuming our brains are simulatable, we could never know this absolutely, as simulators with a huge but finite amount of compute trying to trick us could never be ruled out. 0 and 1 aren’t probabilities and you are never certain. Still, we could conceivably be in a situation were an uncomputable explanation is far simpler than any computable theory.
perfect deterministic software twins, exposed to the exact same inputs. This example that shows, I think, that you can write on whiteboards light-years away, with no delays; you can move the arm of another person, in another room, just by moving your own.
In this situation, you can draw a diagram of the whole thing, including all identical copies, on the whiteboard. However you can’t point out which copy is you.
In this scenario, I don’t think you can say that you are one copy or the other. You are both copies.
You can send probes programmed to just grab resources, build radio receivers and wait.
Not yet. I’ll let you know if I make a follow-up post with this. Thanks for a potential research direction.
Yet More Modal Combat
This seems to be a bizarre mangling of several different scenarios.
Yet in most of the world, humans will probably no longer be useful to anything or anyone – even to each other – and will peacefully and happily die off.
Many humans will want to avoid death as long as they can, and to have children. Most humans will not think “robots do all that boring factory work, therefore I’m useless therefore kill myself now”. If the robots also do nappy changing and similar, it might encourage more people to be parents. And there are some humans that want humanity to continue, some that want to be immortal.
Having been trained to understand our human needs and human nature in minute detail, the AI we leave behind will be the sum total of all human values, desires, knowledge and aspiration.
I think that this is not nessesarily true. There are desings of AI that don’t have human values. Its possible for the AI to understand human values in great detail but still care about something else. This is one of the problems Miri is trying to avoid.
At that point, or soon thereafter, in the perfect world we can imagine all humans being provided all the basic needs without needing to work.
There is some utopian assumption here. Presumably the AI’s have a lot of power at this point. Why are they using this power to create the bargin basement utopia you described. What stops an AI from indiscriminately slaughtering humans.
Also in the last paragraphs, I feel you are assuming the AI is rather humanlike. Many AI designs will be seriously alien. They do not think like you. There is no reason to assume they would be anything recognisably conscious.
And since by then the AI-economy will have already had a long run of human-supervised self-sufficiency, there is no reason to fear that without our oversight the robots left behind will run the world any worse than we can.
A period of supervision doesn’t prove much. There are designs of AI that behave when the humans are watching and then misbehave when the humans aren’t watching. Maybe we have trained them to make good responsible use of the tech that existed at training time, but if they invent new different tech, they use it in a way we wouldn’t want.
It really isn’t clear what is supposed to be happening here. Did we build an AI that genuinely had our best interests at heart, but it turned out immortality was too hard, and the humans were having too much fun to reproduce? (Even though reproducing is generally considered to be quite fun) Or were these AI’s delibirately trying to get rid of humanity. In which case why didn’t all humans drop dead the moment the AI got access to serious weaponry?
I would use
Solar and Lunar Mana, which I think has about 66% chance of working. The only mana type that can be predicted 10 days in advance (beyond the prediction of a random sample from the previous data) is Doom. And that still doesn’t work out with as high probability. (Doom mana will be on a high at the time, but using it with solar gives a 69% chance of reaching 70 and an 11% chance of demons. So a 58% chance of doing good.) If the utility is 0 for any amount of mana <70, and the amount of mana if its >=70, then solar + earth does slightly better in expected utility. 54.6 vs the 54.2 for solar+lunar. It has slightly lower success probability 63%, but slightly more mana if it does succeed.
The bible is written by many authors, and contains fictional and fictionalized characters. Its a collection of several thousand year old fanfiction. Like modern fanfiction, people tried telling variations on the same story or the same characters. (2 entirely different genesis stories) Hence there not even being a pretence at consistency. This explains why the main character is so often portrayed as a Mary Sue. And why there are many different books each written in a different style. And the prevalence of weird sex scenes.
Another reason someone might stick to the rules is if they think the rules carry more wisdom than their own judgement. Suppose you knew you weren’t great at verbal discussions, and could be persuaded into a lot of different positions by a smart fast-talker, if you engaged with the arguments at all. You also trust that the rules were written by smart wise experienced people. Your best strategy is to stick to the rules and ignore their arguments.
Someone comes along with a phone that’s almost out of battery and a sob story about how they need it to be charged. They ask if they can just plug it in to your computer for a bit to charge it. If you refuse, citing “rule 172) no customer can plug any electronics into your computer. ” then you look almost like a blankface. If you let them plug the phone in, you run the risk of malware. If you understand the risk of malware, you could refuse because of that. But if you don’t understand that, the best you can do is follow rules that were written for some good reason, even if you don’t know what it was.
Genetic modification takes time. If you are genetically modifying embryos, thats ~20 years before they are usefully contributing to your attempt to make better embryos.
Maybe you can be faster when enhancing already grown brains. Maybe not. Either way, enhancing already grown brains introduces even more complications.
At some point in this process, a human with at most moderate intelligence enhancement decides it would be easier to make an AI from scratch than to push biology any further. And then the AI can improve itself at computer speeds.
In short, I don’t expect the biological part of the process to be that explosive. It might be enough to trigger an AI intelligence explosion.
If you have 100 hours, and and 100 commitments, each of which takes an hour, that is clearly a case of low time slack.
If you have 100 hours, and 80 commitments each of which takes either 0 or 2 hours (with equal chance) that is the high slack you seem to be talking about. Note that units of free time are available. This person is still pretty busy.
If you have 100 hours, and 1 hour of commitment, and most of the rest of the time will be spent laying in bed doing nothing or timewasting, this person has the most slack.
A way reality might not line up with the superlinear returns on slack is that there aren’t that many opportunities of similar value to take.
Imagine a device that looks like a calculator. When you type 2+2, you get 7. You could conclude its a broken calculator, or that arithmetic is subjective, or that this calculator is not doing addition at all. Its doing some other calculation.
Imagine a robot doing something immoral. You could conclude that its broken, or that morality is subjective, or that the robot isn’t thinking about morality at all.
These are just different ways to describe the same thing.
Addition has general rules. Like a+b=b+a. This makes it possible to reason about. Whatever the other calculator computes may follow this rule, or different rules, or no simple rules at all.
I think the assumption it that human-like morality isn’t universally privileged.
Human morality has been shaped by evolution in the ancestral environment. Evolution in a different environment would create a mind with different structures and behaviours.
In other words, a full specification of human morality is sufficiently complex that it is unlikely to be spontaneously generated.
In other words, there is no compact specification of an AI that would do what humans want, even when on an alien world with no data about humanity. An AI could have a pointer at human morality with instructions to copy it. There are plenty of other parts of the universe it could be pointed to, so this is far from a default.
if a human had been brought up to have ‘goals as bizarre … as sand-grain-counting or paperclip-maximizing’, they could reflect on them and revise them in the light of such reflection.
Human “goals” and AI goals are a very different kind of thing.
Imagine the instrumentally rational paperclip maximizer. If writing a philosophy essay will result in more paperclips, it can do that. If winning a chess game will lead to more paperclips, it will win the game. For any gradable task, if doing better on the task leads to more paperclips, it can do that task. This includes the tasks of talking about ethics, predicting what a human acting ethically would do etc. In short, this is what is meant by “far surpass all the intellectual activities of any man however clever.”.
The singularity hypothesis is about agents that are better at achieving their goal than human. In particular, the activities this actually depends on for an intelligence explosion are engineering and programming AI systems. No one said that an AI needed to be able to reflect on and change its goals.
Humans “ability” to reflect on and change our goals is more that we don’t really know what we want. Suppose we think we want chocolate, and then we read about the fat content, and change our mind. We value being thin more. The goal of getting chocolate was only ever an instrumental goal, it changed based on new information. Most of the things humans call goals are instrumental goals, not terminal goals. The terminal goals are difficult to intuitively access. This is how humans appear to change their “goals”. And this is the hidden standard to which paperclip maximizing is compared and found wanting. There is some brain module that feels warm and fuzzy when it hears “be nice to people”, and not when it hears “maximize paperclips”.
The training procedure is only judging based on actions during training. This makes it incapable of distinguishing between an agent that behaves in the box, and runs wild the moment it gets out the box, from an agent that behaves all the time.
The training process produces no incentive that controls the behaviour of the agent after training. (Assuming the training and runtime environment differ in some way.)
As such, the runtime behaviour depends on the priors. The decisions implicit in the structure of the agent and training process, not just the objective. What kinds of agents are easiest for the training process to find. A sufficiently smart agent that understands its place in the world seems simple. A random smart agent will probably not have the utility function we want. (There are lots of possible utility functions.) But almost any agent with real world goals that understands the situation its in will play nice on the training, and then turn on us in deployment.
There are various discussions about what sort of training processes have this problem, and it isn’t really settled.
I don’t think this research, if done, would give you strong information about the field of AI as a whole.
I think that, of the many topics researched by AI researchers, chess playing is far from the typical case.
It’s [chess] not the most relevant domain to future AI, but it’s one with an unusually long history and unusually clear (and consistent) performance metrics.
An unusually long history implies unusually slow progress. There are problems that computers couldn’t do at all a few years ago that they can do fairly efficiently now. Are there problems where people basically figured out how to do that decades ago and no significant progress has been made since?
The consistency of chess performance looks like more selection bias. You aren’t choosing a problem domain where there was one huge breakthrough that. You are choosing a problem domain that has had slow consistent progress.
For most of the development of chess AI (All the way from Alpha Beta pruning to Alpha Zero) Chess AI’s improved by an accumulation of narrow, chess specific tricks. (And more compute) How to represent chess states in memory in a fast and efficient manor. Better evaluation functions. Tables of starting and ending games. Progress on chess AI’s contained no breakthroughs, no fundamental insights, only a slow accumulation of little tricks.
There are cases of problems that we basically knew how to solve from the early days of computers, any performance improvements are almost purely hardware improvements.
There are problems where one paper reduces the compute requirements by 20 orders of magnitude. Or gets us from couldn’t do X at all, to able to do X easily.
The pattern of which algorithms are considered AI and which are considered maths and which are considered just programming is somewhat arbitrary. A chess playing algorithm is AI, a prime factoring algorithm is maths, a sorting algorithm is programming or computer science. Why? Well those are the names of the academic departments that work on them.
You have a spectrum of possible reference classes for transformative AI that range from the almost purely software driven progress, to the almost totally hardware driven progress.
To gain more info about transformative AI, someone would have to make either a good case for why it should be at a particular position on the scale, or a good case for why its position on the scale should be similar to the position of some previous piece of past research. In the latter case, we can gain from examining the position of that research topic. If hypothetically that topic was chess, then the research you propose would be useful. If the reason you chose chess was purely that you thought it was easier to measure, then the results are likely useless.
If you take this definition literally, then if scientists find an extremely expensive way to make lab grown dodo meat, suddenly the GDP for when dodos existed jumps up.
Measure by old prices, and one thing we can make cheaply now that we basically couldn’t make before and the numbers get huge. Measure by the old prices and one thing we could make but no longer can sends the numbers plummeting.
There are also questions of how much you want to call two items similar. When we count the number of spoons in ancient Rome, do we compare them to modern plastic spoons, modern stainless steel spoons, or what it would cost to pay people to make authentic recreations of Roman style spoons, or the cost of an actual historical Roman spoon in modern times.
How valuable is a slip of paper with Newtons equations written on it, 100 years before Newton was born. If you were sent back in time, you could fairly easily get some paper and pay someone to copy down these exact symbols. But if you didn’t know the answer, you couldn’t. You could pay someone a fair bit to look at the sky, and try to find simple equations to explain what they saw, but that’s still kind of using your future knowledge to say where to look. And it may take some time. And note that 2 such slips of paper are not much more valuable than 1.
When you look closely enough, your abstractions all break down.