This is pretty close to the dust theory of Greg Egan’s Permutation City and also similar in most ways to Tegmark’s universe ensemble.
JBlack
Karma: 2,382
They are the only options available in the problem. It is true that this means that some optimality and convergence results in decision theory are not available.
It’s not exotic at all. It’s just a compatibilist interpretation of the term “free will”, which form a pretty major class of positions on the subject.
That doesn’t address the question at all. That just says if the system is well modelled as having a utility function, then … etc. Why should we have such high credence that the premise is true?
I expect that (1) is theoretically true, but false in practice in much the same way that “we can train an AI without any reference to any sort of misalignment in the training material” is false in practice. A superintelligent thought-experiment being can probably do either, but we probably can’t.
In that line, I expect that (3) is not true. Bits of true information leak into fabricated structures of information in all sorts of ways, and definitively excluding them from something that may be smarter than you are is likely to cost a lot more than presenting true information (in time, effort, or literal money).
Consider that the AI may ask for evidence in a form that you cannot easily fabricate. E.g. it may have internal knowledge from training or previous experience about how some given external person communicates, and ask them to broker the deal. How sure are you that you can fabricate data that matches the AI’s model? If you are very sure, is that belief actually true? How much will it cost you if the AI detects that you are lying, and secretly messes up your tasks? If you have to run many instances in parallel and/or roll back and retry many times with different training and experience to get one that doesn’t do anything like that, how much will that cost you in time and money? If you do get one that doesn’t ask such things, is it also less likely to perform as you wish?
These costs have to be weighed against the cost of actually going ahead with the deal.
(2) isn’t even really a separate premise, it’s a restatement of (1).
(4) is pretty obviously false. You can’t just consider the AI’s behaviour, you also have to consider the behaviour of other actors in the system including future AIs (possibly even this one!) that may find out about the deception or lack thereof.
I agree that even with free launch and no maintenance costs, you still don’t get 50x. But it’s closer than it looks.
On Earth, to get reliable self-contained solar power we need batteries that cost a lot more than the solar panels. A steady 1 kW load needs on the order of 15 kW peak-rated solar panels plus around 50 kW-hr battery capacity. Even that doesn’t get 99% uptime, but enough for many purposes and it is probably adequate when connected to a continent-spanning grid with other power sources.
The same load in orbit would need about 1.5 kW peak rated panels and less than 1 kW-hr of battery capacity for uptime dependent only upon reliability of equipment. The equipment does need to be designed for space, but doesn’t need to be sturdy against wind, rain, and hailstones. It would have increased cooling costs, but transporting heat (e.g. via coolant loop) into a radiator edge-on to the Sun will be highly effective (on the order of 1000 W/m^2 for a radiator averaging 35 C).
I don’t think either of these possibilities are really justified. We don’t necessarily know what capabilities are required to be an existential threat, and probably don’t even have a suitable taxonomy for classifying them that maps to real-world risk. What looks to us like conjunctional requirements may be more disjunctional than we think, or vice versa.
“Jagged” capabilities relative to human are bad if the capability requirements are more disjunctional than we think, since we’ll be lulled by low assessments in some areas that we think of as critical but actually aren’t.
They’re good if high risk requires more conjunctional capabilities than we think, especially if the AIs are jaggedly bad in actually critical areas that we don’t even know that we should be measuring.
Did you look only at changes in median prices (capital gain), or did you include a rental income stream as well? You would need to make allowance for maintenance and various fees and taxes out of that income stream, but it usually still exceeds the capital gain.
In addition to the much greater availability of retail loans, there are often substantial tax advantages available compared with other investments. For example in Australia: the ability to deduct interest payments for investment properties as an expense offsetting all income (not just income derived from the property) to determine taxable income. So in addition to the loans being easier to get and having lower interest rates, they’re effectively lowered further by the investor’s marginal tax rate.
There is also a substantial discount (50%) on capital gains tax for holding the relevant assets for more than a year, which applies to rental property more naturally than many other leveraged investments.
Furthermore, what if long-term average expenses are greater than the 2% quoted in the post?
For example: relocation and rebuilding costs if current facilities become unsuitable over the next few centuries for internal or external reasons, liquid nitrogen becomes more expensive, equipment needs more maintenance in future, and so on.
The model in the post seems like it could be boiled down to “current expenses are ~2%, current real returns are ~3%, therefore all will be good”.
There aren’t discrete “splits” in quantum mechanics, and consequently no way to accelerate them. Branches are a lies-to-children simplification that you can employ in a few toy scenarios, but fail to be useful in pretty much everything else.
I’m not entirely sure what’s being asked here. Is this asking “if we do experiment 1000001 and see k Rs in the first four trials, then what credence do you assign to the 5th trial being R?”
Or is it “if we take a random experiment out of the million and see k Rs in the first four trials, then what credence do you assign to the 5th trial being R”? This isn’t the same question as the first.
Or is it something else again?
In my opinion, Project Lawful / planecrash is a terrible reference in addition to being written in a seriously annoying format. Although I have read it, I don’t recommend that anyone else read it. If any of the material in it should become some sort of shared culture that we should assume others in the community have read, it would require completely rewriting the entire thing from beginning to end.
I am not one of the two voters who initially downvoted, but I understand why they might have done so. I have weakly downvoted your comment for having made a load-bearing link between someone not having read Project Lawful and calling them “an NPC” in your sense, which is not the standard of discourse I want to see.
If you were expecting this person to have read this extremely niche and frankly bizarre work of fiction without having confirmed that they have actually read it and understood and fully agree with some relevant decision theory parts in it, then that seems pretty unwise of you and their not having done so does not reflect in any way poorly upon them.
“You didn’t act like I think the fictional character Keltham would have” is not a reasonable criticism of anyone.
There may or may not be some other unspecified actions they may have performed that do reflect poorly upon them, but those do not appear to connect in any way with this post.
the authors have made a compelling case that even if >95% of their specific arguments are incorrect, the core claim “if anyone builds it, everyone dies” still holds true
I don’t believe this at all, and I’m not sure that you do either. I do believe that the title claim IABIED is largely true, but believe very much more strongly that it would be false if >95% of the arguments in the book were incorrect.
I’m not sure whether you are being hyperbolic with the “>95%” claim, or have actually gone through a sample of at least 40 arguments in the book and seriously examined what the world would look like if at most 2 of those were correct with all the rest failing to hold.
From what I’ve seen, the title claim would be seriously in doubt if even half of the arguments failed. Mainly because the world would necessarily be extraordinarily different in major respects from the way I or the authors believe that it is.
It is a predictable consequence of saying “lol no I didn’t agree to that prior to being told thanks for telling me” that Keltham (and other people with similar expressed views regarding information and coordination) won’t be told information intended for coordination in future, including information that Keltham and similar people would have wanted to be able to use in order to coordinate instead of using it against the interests of those giving them the information.
So the question is: just how strongly does Keltham value using this information against the priest, when weighed against the cost of decreasing future opportunities for coordination for himself and others who are perceived to be similar to him?
There are plenty of other factors, such as whether there are established protocols for receiving such information in a way that binds priests of Abadar to not use it against the interests of those conveying it, whether the priest and the adventurer (and future others) could have been expected to know those protocols, and so on.
“Imagine there’s a child drowning in a shallow pond. You’re wearing a swimsuit and could easily save them. Don’t believe me? Okay, let me make it more believable: imagine there’s also a cute puppy guarding the pond that you’d have to kill to reach the child. Would you do it?”
This seems very incoherent.
It starts with “Imagine” and then two sentences later asks “Don’t believe me?”
Believe what? Believe that I was asked to imagine something? I look back up a bit on the page, and I can have rather high credence that I was asked to imagine something.
Believe that I have imagined the scenario? I can have pretty high credence of imagining something that matches the description so that doesn’t fit either.
Believe that the imaginary scenario is real? Agreed, I definitely don’t believe that and there’s a lot of evidence that it’s false, but what relevance does it have? The text didn’t ask me to believe that.
Maybe believe that I could one day be in such a situation? Well, that’s definitely more believable but still very unlikely. I’m very much not in a habit of wearing swimsuits near ponds, though if it had been a swimming pool that would be more believable (while still being very unbelievable). So yes I don’t believe it.
Then the text goes on to posit a wildly more improbable scenario, prefaced with “let me make it more believable”. What? No, that just made it massively more unbelievable, so the writer of this story probably has terrible epistemics, or at best is authoring an unreliable narrator character that does. Pretty much any further detail would make it less believable by conjunction.
What’s worse, that particular conjunction is pretty ridiculous. I can still imagine it happening in some amazingly contrived scenario, but brings up so many other questions like “how” that make even suspension of disbelief for a fictional situation difficult. In what manner would I have to kill it to rescue the child? Is it basically a full-grown dog (but still cute) that would threaten my own life if I approach, and I happen to have a gun with my swimsuit? Maybe there’s a supernatural barrier that’s tied to its life? In what way is any of this more believable even for a very unreliable narrator?
Is it plausible that the two different sources of valuation are actually one source? For example, could On Point valuations have formed the basis of the CARES Act filings, or vice versa, or both used data from a common source?
It also seems likely that any borrower has a strong incentive to talk up the value of their collateral, especially so if they get to choose who has access to the data used for valuations.
No problem, if it’s tangential then we can agree that it’s tangential. We also appear to agree that the zombie argument may not be a useful way of thinking about things, as referred to in the last sentence of my first comment.
I agree that if a p-zombie of me can exist, then my consciousness would not be the sole cause for things with my pattern of matter saying that they are conscious. It may still be a cause, in that my consciousness may be causally upstream of having a pattern of matter that emits such utterances, but there may be other ways that such patterns of matter can come to exist as well.
You didn’t (rhetorically) ask what a zombie would say. You asked how do I know. That fact that something else might say the same thing as I did is not at all disconcerting to me, and I’m not sure why it’s disconcerting to you.
You don’t need to go anything like as far as p-zombies to get something that says the same thing. A program consisting of print(“I know that I’m not a zombie since I have consciousness”) etc does the same thing.
The rhetorical point you’re making is simply false. A p-zombie would not believe anything, since it has no mind with which to believe. It would merely act in the same manner as something that has a mind.
Most of these social structures are, in the aggregate, substantially stupider than individual humans in many important ways.