Note the reason why a specific cryptocurrency, from a set of competing crypto currencies, gets used is an example of the network effect. The more people use a specific cryptocurrency (bitcoin or ethereum or dogecoin or whatever), the more and better support there will be for transactions in this currency. This means better and more reliable software (less likely to lose or corrupt your money), and more importantly, it means less volatility and more stability for carrying value from sender to receiver. The more mega or giga-dollars being moved using that currency the better off you will be for your transaction. Especially if your transaction if large or you are trying to be anonymous—either way, more traffic is better for you.
This is the network effect—using the network with more users has more utility to you. In the cryptocurrency world this means the currency that both (1) offers all the features needed in practice (2) has the most users is ultimately going to become dominant.
There are 2 significant issues here. Consider a robot in physical reality with limited manipulation capability. Even with infinite intelligence, the robot has a maximum lifetime number of manipulations it can perform to the outside world. With a human, that’s 2 arms. What if an animal without opposable thumbs had infinite intelligence? Then it would be capable of less.
What does infinite intelligence mean? It means an algorithm that, given a set of inputs and a heuristic, can always find the most optimal solution—the global maximum—for any problem the agent faces.
Actual intelligent agents have to find a compromise—a local maxima. But a “good approximation” may often be pretty close to the global maxima if the agent is intelligent enough. This means that if the approximation an agent uses is 80% as good as the global maxima, infinite intelligence only gains the last 20 percent.
This is the first problem here. You have discovered a way to build a self-improving algorithm that theoretically has the capability of finding the global maxima every time for a given regression problem. (it won’t but it might get close). So what. You still can’t do any better than the best solution the information allows. (and it may or may not make any progress on thought to be NP problems like encryption)
Consider a real problem like camera-based facial recognition. The reason for remaining residual error—positive and negative misclassifications—may simply be the real world signal does not have sufficient information to identity the right human from 7 billion every time.
The second problem is your heuristic. We can easily today build agents that optimize for the wrong, ‘sorcerer’s apprentice’, heuristic that goes awry. Building a heuristic that even gives an interesting agent—one with self awareness and planning and everything else we expect—may take more than simply building a perfect algorithm to solve a single subproblem.
A concrete example of this is ImageNet. The best-in-class algorithms for solving it solve the problem of “get the right answer on ImageNet” but not the actual problem we meant which is “identify the real world object in this picture in the real world”. So the best algorithms tend to overfit and cheat.
Well, the other way to check if I am right or wrong is to back calculate the rocket equation. Instead of relying what I say, what’s the payload mass to propellant mass of the BFR? Saturn V (the rocket equation is the same for the BFR, and it is using recoverable boosters and a compromise fuel (liquid CH4) so I expect it to perform slightly worse) it’s 6.5 million pounds total rocket mass, 85% payload, to 261,000 lbs to LEO. So 4% of the mass is payload, 85⁄4 = 21.25 kg of propellant for every kilogram of payload.
Ok, CH4 + 202 = CO2 + 2H20
1⁄3 of the mass is the CH4, while 2⁄3 is O2. That helps a lot as liquid oxygen is cheaper, only 16 cents per kilogram. So $2.26 for the liquid oxygen.
Well, how much does 7.08kg of liquid methane cost? (note that BFR needs purified methane and cannot use straight natural gas)
Well, 1.14 Therm = 1 gge = 5.660 lb. So 21.25kg = 15.61 pound, 15.61 pound/5.660 = 2.757 gge.
2.757 gge * 1.14 = 3.14 therm. Average prices presently per therm are $0.92. So $2.89 for the unpurified fuel. Then you need to purify it to pure methane (obviously with some loss of energy/gas/filter media) and liquify it. I am going to assume this raises the cost 50%. So $4.33 for the natural gas. Total cost per kg for the fuel is $4.33+2.26 = $6.59.
$10 a kg for payload to LEO, including the rocket, seems rather optimistic. Remember the rocket needs repair and will occasionally blow up. Helicopters and other much lower energy terrestrial machines, the maintenance + repairs are often either similar or more expensive than the price of the fuel. I would expect the real minimum cost per kg to be at least 3 times the cost of fuel: 2 units of repair/replacing exploded rocket parts for every kg of propellant. Or $19.78 per kg, which would be phenomenal results compared to today’s $2720 a kg (using spaceX now), and just half as good as Elon Musk’s promise.
Hard laws of nature here. I want to go to space as well but it takes a literal swimming pool of fuel under you to do it, and while SpaceX has made some impressive advances, it doesn’t change the basic parameters of the problem.
In the rocket industry, the ‘payload’ is the piece that reached orbit. That is how it is defined. You technically can occupy the entire upper portion of a Dragon spacecraft (the entire section above the second stage inside the fairing) with your mega-satellite. That entire satellite is ‘payload’ and the source of the ‘payload to LEO/geostationary orbit’ that gets quoted as the capability of the spacecraft.
You have to assume that “$10” figure is the lowest number possible, which means Musk is accounting for the entire payload.
Regarding cryonics not working: this depends on your definition of ‘working’. Let me describe the problem succinctly.
Assume at some future date you can build a ‘brain box’. This is a machine, using some combination of hardware and dedicated circuitry, that is capable of modeling any human brain that nature could build. It likely does this by simulating each synapse as a floating voltage, modulated by various coefficients (floating point weights) when an incoming pulse arrives.
Well, you can choose randomly the weights, and assuming you also attach a simulated or robotic human body (a body with sufficient fidelity), and train the robot or simualated body with an appropriate environment, the ‘being’ inside the box will eventually achieve sentience and develop skills humans are capable of developing.
But you don’t have to choose the weights at random. If you obtain just 1 bit of information from a frozen brain sample, you can use that bit to bias your random rolls, reducing the possibility space from “any brain possible within the laws of nature” to “a subset of that space”.
If you have an entire frozen brain, with whatever damage cryonics has done to it, and you first slice and scan it with electronic microscopes, you still get a lot more bits than just 1. You will be able to instantiate a brain that has at least some of the characteristics of the original. Will they have clear and coherent memories (as coherent as humans have...)? Depends on the quality of the sample, obviously.
But regardless of damage you can bring each cryonics patient ‘back’, limited by the remaining information. This is actually no different than caring for a patient with a neurodegenerative disease, except that the brain box will not have any flaws in it’s circuitry and once instantiated, the being occupying it will be able to redevelop any skills and abilities they are missing.
Now, yes, trying to ‘repair’ a once living brain to live again as a meat-system is probably unrealistic without technology we cannot really describe the boundaries of. (as in, we can posit that the laws of physics do let you do this if you could make nanoscale waldos and put all the pieces back together again, but we can’t really say with any confidence how feasible this is)
I was thinking of the cost to design and launch the space hotel to visit, the cost of operations for the vast company needed to support all this, and so on. On further thought, I think I will agree with you partially and reduce those overheads for the sake of argument. But the crew dragon is 12055 kg to launch 90*6 astronauts of payload. Or a 20:1 ratio of spacecraft mass to crew mass. Going bigger to a 100-seat spacecraft does allow for a better ratio, but it would still be at least 10:1. So if a person weighs 90 kg (average adult), they need 900 kg worth of spacecraft to visit the space hotel. Or $9000 for the single person transit costs. Plus the cost to launch and maintain the hotel and launch vehicles.
I don’t know what the other costs are going to be, just they will be governed by the high cost to reach orbit as well as very expensive mishaps whenever a rocket blows up or crashes, which will still happen at this scale. So maybe 50-100k per person for a week in space?
There you are in space. You just blasted off from Boca Chica and sit floating in low earth orbit. For every kg of your body’s flesh, there is n kgs of spacecraft around you. The vehicle itself. Maybe it’s automated but it has a skin to protect against radiation, micrometeorites, and vacuum. Maybe the consumables are recycled, but the structure of the spacecraft—the motors for life support, the parachutes for the escape system, the propellant for the reentry burn—all count as “payload” to orbit. Payload that will be required for you to make it alive to the space hotel and return safely later. 5:1 sounds like an underestimate, actually. On top of that, the food and booze are obviously required for your stay on the hotel. Maybe the air and water can be recycled, but growing food to appeal to the palate of someone in this price-range isn’t possible without a vast amount more scale.
I don’t think your last point is very indicative. Here’s what my analysis is based on.
Human activities need to bring in more value than they cost, or they will only happen on very small scales. Any sort of minerals you might bring in from space are still going to be far easier to access—and cheaper—with very large terrestrial mines. Even if Elon Musk’s $10 a kg were realized, that only gets you to LEO. You have to develop the equipment—and the spacecraft—to reach the asteroids, while equivalent terrestrial mines can use ships, trucks, and trains to move in very heavy equipment for mining on immense scales. Humans can also work on the equipment on earth, and there are many places unexploited simply because current prices don’t quite support operations in the more difficult areas.
So if not minerals, then what. It would still be extremely expensive for a person to go to space, there is the mass of the rocket, and many labor requiring steps such as preflight screening and training, flight crews, inspections of the rocket, and so on. Assuming all that raises the cost an order of magnitude, to $100/kg, and a person needs 5 kgs of support equipment for every kg of personal mass, then a 200 lb person will need to pay $220,000 for a trip to space.
You can see where you hit a saturation point—you can add up all of the people on earth who can afford the ticket, assume only a fraction will risk their lives in doing it, and that’s your market size.
This still might mean hundreds or thousands of people visiting space, versus the 3-9 people who are orbiting today, but it doesn’t change anything for the average person.
When I say it saturates I don’t mean that nothing will happen, I am just saying it doesn’t matter like autonomy does. Do the same napkin math for vehicle autonomy. 3.25 trillion miles driven in the United States. Assuming a market share of 30% of that (once autonomous cars are clearly safer and routinely available they are expected to rapidly take over the market), and 10 cents of revenue for the software companies making the software per mile. 97 billion. Then you add in Europe and China...
Robotics autonomy also has the promise that once you build a software framework to solve one problem (like autonomous driving), problems in the same domain (such as warehouse logistics) become much easier to solve, with far less investment required. So you would expect these machines to very quickly pay for their own hardware and software development costs.
I’m going to take the contrarian view. I don’t think cheaper access to space will have significant effect on the majority of the humans alive over the next 10 years. Cheaper satellites will mean better and more scientific information about climate and human activity will be available, and obviously it makes low latency/high bandwidth ISPs like Starlink feasible. This is an improvement as it brings internet access to the ‘last 10%’ - but it isn’t a society-changing one.
Furthermore, it saturates in scale. Once SpaceX grows large enough, they saturate the market for satellite ISPs—they have enough capacity to fill the skies—and saturate the market for billionaire tourist trips to space. Asteroid mining is questionable.
Autonomous cars and all of the robotic problems that the same class of motion solvers can solve—I expect to have an enormous impact because their saturation point is at which half of all present jobs have been automated, and exponential numbers of robotic systems have been deployed.
So I have a question. I’m really stoked about the potential of AI. For a number of reasons, I think that robotics—using machine learning systems to run physical robots, and using the data from physically interacting with our world plus the revenue they earn—is the fastest and most certain way to make genuine forward progress in AI. The reasons have to do with the fact that modeling the immediate term physical world is something that you can get by far the greatest accuracy with, that a robotic system produces far cleaner data than any type of passive observer, and I think that higher level thought requires subsystems trained on lower level phenomena.
But if you look at job listings—I am a few months from finishing the OMSCS Master’s in machine learning and have years of embedded systems experience—I rarely see postings for anyone working on these kinds of systems. Where can I get a job in the field?
So the problem here is that obviously there will be huge numbers of counter-exceptions to your proposed rule. Essentially while your rule probably has some predictive power—if you could passively observe thousands of humans interacting your rule might predict friendships better than chance—most likely there’s a better rule. You might need invasive brain data-loggers to find it but such a rule probably exists.
There are most likely sequences of inputs that cause most human brains to become receptive to new friendships.
Or maybe it’s a matter of logistics—you obviously can’t really form friendships with people you only are in proximity with briefly.
Childhood, a certain fraction of your elementary school class will be the same individuals all the way to senior year of high school. Hence, more years in proximity = stronger friendships. College is much shorter duration. In many workplaces, the economics encourage frequent job switches (about every 1.5 − 4 years is what the rules of the game call for), making such lasting interactions less likely.
And then there’s the fact that with limited cognitive resources, you can only maintain so many friendships and probably won’t invest in a new one if your older ones are still active. Thus the probability of a friendship would be higher in childhood because you don’t already have as many friends.
I don’t think this is a realistic model of how basic research works. Possibly this is a crux between our models?
I’m responding to this statement directly in this post. No, this isn’t how basic research works. But just because centuries of inertia cause basic research to be structured a certain way doesn’t mean it has to be that way, or that my original statement is wrong.
You could quick and dirty assemble a model using curve fitting that would approximately tell you the relationship between the position of the Moon in the sky and a rocket’s thrust vector. It wouldn’t need to be a complete theory of gravitation, that theory that was developed over centuries. And it would work : approximate models are very often good enough.
2. “Don’t worry about developing calculus, questioning the geocentric model of the solar system, etc.” is the wrong decision in the fictional example Eliezer provided. You suggest, “once you start getting spaceplanes into orbit and notice that heading right for the moon isn’t making progress, you could probably get together some mathematicians and scrum together a rough model of orbital mechanics in time for the next launch”. I don’t think this is a realistic model of how basic research works. Possibly this is a crux between our models?
The theoretical framework behind current AI research is essentially “here’s what we are regressing between, X and Y, or here’s some input data X, outputs in response Y, and a reward R”. % correct or biggest R is the objective. And for more complex reasons that I’m going to compress here, you also care about the distribution of the responses.
This is something we can run with. We can iteratively deploy an overall framework—a massive AI platform that is supported by a consortium of companies and offers the best and most consistent performance—that supports ever more sophisticated agent architectures. That is, at first, supported architectures are for problems where the feedback is immediate and the environment the system is operating in is very markovian and clean of dirt, and later we will be able to solve more abstract problems.
With this basic idea we can replace most current jobs on earth and develop fully autonomous manufacturing, resource gathering, construction,
Automating scientific research—there’s a way to extend this kind of platform to design experiments autonomously. Essentially you build upon a lower level predictive model by predicting the outcomes of composite experiments that use multiple phenomena at once, and you conduct more experiments where the variance is high. It’s difficult to explain and I don’t have it fully mapped out, but I think developing a systematic model for how macroscale mechanical physical systems work could be done autonomously. And then the same idea scaled to how low level subatomic systems works, and to iteratively engineer nanotechnology, and maybe work through cell biology a similar way.
Umm, maybe big picture will explain it better : you have hundred story + megaliths of robotic test cells, where the robotic cells were made in an automated factory. And for cracking problems like nanotechnology or cell bio, each test cell is conducting an experiment at some level of integration to address unreliable parts. For example, if you have nanoscale gears and motors working well, but not switches, each test cell is exhaustively searching possible variants of a switch—not the entire grid, but using search trees to guess where a successful switch design might be—to get that piece to work.
And you have a simulator—a system using both learnable weights and some structure—that predict the switch designs that didn’t work. You feed into the simulator the error between what it predicted would happen and what the actual robotic test waldos are finding in reality. This update to the simulation model makes the overall effort more likely to design the next piece of the long process to developing nanoscale self replicating factories more probable to succeed.
And a mix of human scientists/engineer and scripts that call on machine learning models decide what to do next once a particular piece of the problem is reliably solved.
There are humans involved, it would not be a hands off system, and the robotic system operating in each test cell uses a well known and rigidly designed architecture that can be understood, even if you don’t know how the details of each module function since they are weighted combinations of multiple machine learning algorithms, some of which were in turn developed by other algorithms.
I have a pet theory that even if you could build a self improving AI, you would need to give it access to such megaliths (a cube of modular rooms as wide on each side as it is tall, where each room was made in a factory and trucked onto the site and installed by robots) to generate the clean information needed to do the kinds of magical things we think superintelligent AIs could do.
Robotic systems are the way to get that information because each step they do is replicable. And you subtract what happens without intervention by the robotic arm from what happens when you do, giving you clean data that only has the intervention in it, plus whatever variance the system you are analyzing has inherently. I have a theory that things like nanotechnology, or the kind of real medicine that could reverse human biology age and turn off all possible tumors, or all the other things we know the laws of physics permit but we cannot yet do, can’t be found in a vacuum. If you could build an AI “deity” it couldn’t come up with this solution from just what humans have published (whether it be all scientific journals ever written or every written word and recorded image) because far too much uncertainty would remain. You still wouldn’t know, even with all information analyzed, exactly what arrangements of nanoscale gears will do in a vacuum chamber. Or what the optimal drug regimen to prevent Ms. Smith from developing another mycardial infarction was. You could probably get closer than humans ever have—but you would need to manipulate the environment to find out what you needed to do.
This is the concrete reason for my assessment that out of control AGI are probably not as big a risk as we think. If such machines can’t find the information needed to kill us all without systematically looking into this with a large amount of infrastructure, and the host hardware for such a system is specialized and not just freely available on unsecured systems on the internet, and we haven’t actually designed these systems with anything like self reflectance much less awareness, it seems pretty implausible.
But I could be wrong. Having a detailed model of how I think such things would really work, based upon my previous work with present day AI, doesn’t necessarily make me correct. But I certainly feel more correct.
If the paperclip maker’s architecture is a set of constrained boxes, where each box does a tiny, well defined part of the problem of making paperclips, and is being evaluated by other boxes that ultimately trace their goals and outputs to human defined goals and sensor data, it’s not going anywhere. It’s not even sentient in that there’s no memory in the system for anything like self reflection. Every piece of memory is specific to the needs of a component. You have to build reliable real-time systems like this, other architectures won’t function in a way that wouldn’t fail so often as to be economically infeasible. (because paperclips have very low value, while robotic waldos and human lives are expensive)
This is what I mean by I’m on the side of the spaceplane designers. I don’t know how another, more flexible architecture would even function, in the same way in this story they don’t know how to build a vehicle that doesn’t depend on air.
The trouble with this problem is the rocket used for this was a system of welded and bolted together parts. The functions and rules of each system remained the same throughout the flight and thus it was possible to model. Self improving AI, it would be like if we used the rocket exhaust from the Saturn V to melt metal used in other parts of the rocket during the flight to the Moon.
I can see a way to do self-improving AI : separate modular subsystems, each being evaluated by some connection either directly or indirectly to the real world. But in that case, while each subsystem may be a “black box” that is ever-evolving, basically the function remains the same. Like you might have a box that re-renders scenes from a camera without shadows. And there’s feedback and ways it can get better at it’s job. And there’s a meta-system that can gut the architecture of that box and replace it with a new internal way to do this task. But, all of the time, the box is still just subtracting shadows, it never does anything else.
I hate to say this but I’m taking the side of the Spaceplane designers. Perhaps it’s because it’s what I know.
That is, I can’t really extrapolate to fully sentient, free to act AI. What I see as plausible is we build a vast set of shared libraries used in automated systems all over the economy. And each of these is essentially a hierarchical stack of agents. So a robotic arm might have agents that use reinforcement learning to classify what’s in range of the arm. And then internally there’s an agent that evaluates possible paths and the estimated reward from moving the arm to do something. And then there’s control algorithms that generate the exact motor control sequences of commands. And a higher level system above this that is commanding various robotic systems to do things, with a goal of making paperclips or whatever.
And meta-systems that can try possible architectural changes to pieces of this system and determine if the changes are ‘better’, using Baysian calculations of the distribution of performance.
So, ok, what stops the script that orders the robotic systems to do things to make paperclips from ordering up killbots so it can kill people who are blocking the system from making more paperclips?
Well the obvious thing to do would be to add more heuristics to your paperclip maker. Instead of just having a goal to make the number of paperclips bigger, add some more goals. Add a goal to keep energy consumption under a certain level. And factory space taken to a certain level. And put a value on each person you statistically kill so you consider killing a person (by, saying, having your robotic actuators impinge on the probability cloud where you think a human worker is) pretty bad. (but not infinitely bad or your actuators won’t move at all because the probability a human is in the way is never exactly zero)
So that’s the “guidance fins” of this analogy. And the truth is, once you start getting spaceplanes into orbit and notice that heading right for the moon isn’t making progress, you could probably get together some mathematicians and scrum together a rough model of orbital mechanics in time for the next launch.
And making a spaceplane so powerful it wrecks the planet if it crashes into it, when you don’t know what you are doing...seems implausible to me. I agree we can make an AI that powerful but I think we would need to know what we are doing. Nobody made fission bombs work by slamming radioactive rocks together, it took a set of millions of deliberate actions in a row, by an army of people, to get to the first nuclear weapon.
A nuclear weapon is a system of tightly interconnected components. Some parts of it are extremely pure substances that took enormous energy (and human effort) to obtain.
I think a sentient AI is the same way. Nobody’s inventing one at a terminal in their basement. You’ll need libraries, trained on millions of working robot systems in the real world. Probably some new programming languages meant for the purpose that we haven’t invented yet. A vast network of server-clouds that provide the compute power to test and iterate on your models. Your actual underlying machine learning techniques will most likely been machine optimized.
Once you have all these pieces available to parties with sufficient budgets, it would be like having a way to order highly enriched plutonium from Granger. Then it would be possible to build a closed-loop, self improving system.
Ok, so why not rip a page from nature? Throw a replication counter or some other hard limit in an AI’s terminal values.
The very concept of a paperclip maximizer is an agent that is stuck maximizing some silly parameter and unable to change. If it’s unable to change it’s terminal values, then a second terminal value that limits how far it can expand will hold.
I don’t think this is a counter-argument, to be honest. Because the most probable scenario if we lose control of AI is we create some type of evolutionary environment where many types of agents and many terminal values can compete for resources. And it is self evident that the ultimate winner of that competition is an agent that values copying itself, as accurately and rapidly and ruthlessly as possible. Such agents are a paperclip maximizer, in the same way all life is.
What I’m hearing is that EBM as described can’t be very effective. Doctors do not have the mental energy to spare to be hunting through the literature. For one thing, the literature is a mess, there’s a mass amount of dead wood and conflicting findings and studies that didn’t replicate and so on. It is probably beyond the ability of a human to synthesize it all into a decision matrix to actually use to treat patients optimally.
Obviously the basic principle of using evidence gathered from lots and lots of places to make decisions instead of just doing whatever the attending physician you learned under does is sound. As long as the laws of the universe are the same between different medical clinics, this is a better method to make decisions with.
But yeah, this is a field waiting for AI to crack it open.
Ok, fair enough. So why are diet and exercise programs so hard to stick to medium to long term? Maybe it’s just me, but I’ve done all that you say. I’ve worked out the holy Meal Plan that will bring my weight down far enough to meet my goals. I’ve worked out exactly which sets to do and which lifts are optimal.
And I do them. For a week or three. And something happens and I stop doing them. It feels like the failure point is not the first time going to the gym or the second, it’s somewhere around the tenth time.
And air cooled fission cores have amazing simplicity and power density.
What bugs me about the concept of a “seed AI” that basically rebuilds itself incrementally is I don’t see a whole lot of difference than basically rigging a nuclear reactor to blow by using too much highly enriched uranium too close together. Or building an electrical panel where the wiring is all intermixed and you’ve got bricks of explosives put in there.
If you don’t have a clean, rational design for an AI, with a clear purpose and clear definition of success/failure for each subsystem, you’ve done a bad job at engineering one. We absolutely could develop AIs that will automate most menial tasks because they are well defined. We could develop one that would act like a force mulitplier for existing engineers, allowing the engineer to specify the optimization parameters and the AI produces candidate designs it thinks based on simulations and past experience will optimally meet those parameters.
Even more difficult things like treatments for aging and nanomachinery design and production could be solved with limited function, specialized agents acting like a force multiplier. Hardly the same thing as going back to paper and slide rules.