An aspiring rationalist who has been involved in the Columbus Rationality community since January 2016.
J Thomas Moros
Karma: 335
I appreciate your effort to take political violence seriously and not strawman it. I think it is critical that people argue against a steelman if they are going to be persuasive to those who might want to advocate for or carry out violence. However, I feel like you end up doing the same thing you accuse others of.
I cannot imagine a world in which people manage to coordinate at a scale this large and go undetected. Practically, it seems nearly impossible to achieve the coordination necessary to destroy enough data centers or kill enough researchers to make an impact on AI progress. For one, these actions would have to be carried out over a very short span of time, probably shorter than one night, in order not to be noticed before companies beef up their security. For another, one person can only kill so many people in a day, so you would have to get thousands of people on board with mass murder (or at least mass property destruction) and coordinate with them within this short time-frame. Finally, you’d have to do all this without someone whistleblowing, or something else going awry.
In fact, it seems like large-scale political violence is more difficult to coordinate than a global ASI ban, which Eliezer argues for. Even though a global ASI ban is often treated as an idealistic goal, I argue that it’s significantly easier to make happen than any sort of impactful political violence. It would require immense coordination, to be sure, but not nearly as much as the violence would require. Also, the coordination for a global ban wouldn’t have to be secret, as it would be for violence. Furthermore, a global ASI ban is something we can progress to through smaller laws, whereas any sort of effective political violence would have to happen all at once. Really, compared to effective political violence, a global ASI ban looks so doable.
One would need to have many people commit violence, but it wouldn’t need to be organized at such a large scale or committed over such a short time period.
AI is compute-constrained. According to https://www.allaboutai.com/resources/ai-statistics/ai-data-centers/, there are “4,000 AI-capable facilities globally.” Yes, many data centers would need to be hit, but one doesn’t need to fully destroy a data center to have a meaningful impact. Taking one offline for 6 to 12 months matters. It matters even more if you can permanently damage chips so that they have to be replaced. After some attacks, security will be increased, but no security is perfect, and the attacker gets to pick the most vulnerable targets. This talk of data centers ignores the fact that there are even better targets: AI chip fabs, and the factories for chip fab machines. A single successful attack on such a facility could significantly reduce future global computing capacity. There are far fewer of these in the world. And there are creative points of attack. For example, one could destroy a chip fab machine during shipment.
Organizing doesn’t require a massive conspiracy. In the US, it is legal to call for violence as long as it doesn’t cross certain boundaries for imminence and specificity. There are radical climate change activist orgs that routinely call for violence and do so within the limits of the law (e.g. Deep Green Resistance). They maintain a strict separation between “above-ground” legal activities and “below-ground” illegal activities. The AI threat is much more direct and immediate than climate change, and has targets for violence that are more clearly linked to the outcome. One or more organizations could publicly organize and advocate and make it clear that there was widespread support for the use of violence in these ways. Then, small groups and individuals could organize to commit individual acts of sabotage. Some groups will fail for various reasons, but if there are enough groups, it could make a difference.
The goal of such political violence wouldn’t be to single-handedly stop AI. It would be to make clear the level of opposition, threaten the possibility of continued escalation of violence, and buy time for governments to act to pause AI. It takes time to build any movement, and this would be no different.
I think there may be some variation in the answer across the different strands of structuralism.
Modal structuralism would say that there are two different possible mathematical structures. One where the CH is true and one where the negation of CH is true. Any statements dependent on the CH will have to be qualified with the mathematical structure being studied. If there is some statement X that is true if CH but false if not CH, then that is fine. It is no different than saying the angles of a triangle add to 180 in Euclidean space but not in hyperbolic space.
Your hash of BB mod 2 statement is different because it is true in all the mathematical structures we study (or not expressible in certain restricted structures). Also, it is not independent of those structures. So we simply say it is true. But if there were some logically possible mathematical structure where it had the opposite truth value, then one would need to qualify which mathematical structure one was referring to when talking about its truth.
I don’t know that there is an official “LessWrongist” philosophy for math.
In some posts, Yudkowsky reads to me as being a Platonist/realist. In other posts, his philosophy comes across to me as some kind of Intuitionism or Fictionalism. I don’t recall reading anything where it is clearly stated.
Modal Structuralism
In my own study of this, guided by another rationalist who had conducted an extensive study of the philosophy of math, I concluded that modal structuralism is correct. This view is mostly associated with the philosopher Geoffrey Hellman.
Structuralism is a group of philosophies of math that hold that mathematical objects are exhaustively defined by their place in mathematical structures. For example, the number 2 doesn’t exist on its own and have a property of “twoness”, rather it is defined by being in the second position in the structure of natural numbers. Its only property is in its place and how that place relates to the rest of the structure. Structuralism is an epistemologically realistic philosophy. So it holds that the hash of a busy beaver number example you gave does have a truth value, even if we can never determine it. However, structuralism doesn’t say what kind of existence mathematical structures have. So there are subvarieties for the different choices there. The book “Mathematical Structuralism” by Geoffrey Helman and Stewart Shapiro is a good introduction to structuralism.
Modal structuralism holds that mathematical structures do not exist as abstract entities (e.g., in a platonic sense). Instead that, if they did exist, then they would have certain properties. So when we say that your problem of the hash of a busy beaver number has a truth value, we are saying, if the mathematical structures necessary to talk about that problem existed, then the statement would be true or false and further that it would have the same truth value in any possible embodiment since the truth is determined by the properties of the structure. When we say a mathematical structure “exists”, we are really saying that it is logically possible it could exist and that we are talking about the properties it would have if it did exist.
Other Questions
Modal structuralism doesn’t specifically address your other questions about where mathematics comes from, how we evolved humans can practice it, and the unreasonable effectiveness of mathematics. However, I think there are some reasonable answers that are consistent with it. I’ll try to sketch them out.
Math is the study of the properties of mathematical structures if they were to exist. One way to define these structures is via systems of rules (i.e. axioms). Since the universe follows logically consistent rules (i.e. physical laws), it will be possible to, at least partially, map mathematical structures onto the physical systems. (We chose to study the ones that correspond to our universe first since those were the ones of most interest to us.) As to why the universe has logically consistent laws rather than chaos, I don’t have a good answer. I would guess that it is an anthropic argument. It isn’t possible to have thinking beings in a universe without logically consistent laws.
Since the universe follows logically consistent laws and we evolved to thrive in that universe, we evolved skills like logic, counting, etc. These were useful to our survival. There seems to have been some kind of runaway intelligence competition (sexual or social selection?). That intelligence built on the foundation of logic gave us the skills we need to do mathematics. Since it is modal reasoning about what would be true of mathematical structures if they existed, we don’t need any magic ability to know or get in touch with mathematical objects. Just our logic, counting, etc. skills are enough for us to make statements about math.
I continue to be confused as to why people didn’t go back to normal more once they got vaccinated. Yes, some people were too traumatized to return to normal, but I don’t think that is the typical person. If you have insight that can help me understand, I’d be interested in another post.
P.S. thanks for this post. I agree that people have failed to face what happened and also failed to move on properly. Those might be related.
Thank you for the work you are doing and for identifying The Spectre. I have felt something similar for a while now.
dpes
Typo, should be “does”
Tristan Harris Interview on AI Safety by Steven Bartlett (aka Diary Of A CEO)
I’m conflicted about this post because I agree with your core message and the urgency that is lacking here. However, I don’t think this is the right way to create a message for this group. I highly doubt a bunch of biblical references are the way to reach a Less Wrong crowd, even if they are very basic ones.
I’m confused as to why you are confused.
You say “FICO scores do not seem to be made with a special process. How can they be especially good data?” The score was presumably designed for the purpose of assessing risk. Per Wikipedia, FICO scores were introduced for that purpose based on credit reports. I highly doubt that the Fair Issac company did that with “no special process.” Most likely, a data analysis was done where many different possibly relevant data points were pulled from credit reports and then analyzed to see which would be predictive. I imagine the weighting between the factors was optimized for that purpose.
Your explanation of how the various factors are relevant to creditworthiness mostly makes sense. The best predictor of future behavior is past behavior. I do think that for the majority of people a FICO score is getting at a mix of trustworthiness and stability. If someone has a long track record of paying their debts and nothing has changed in their financial situation, then there is a high probability they will continue to pay them.
Keep in mind also that credit cards and many loans also ask for your income with legal consequences if you lie. Combine FICO score with sufficient income and you have eliminated a large risk factor that may not show up yet on a credit report (i.e. loss of employment).
The only factor that has always seemed odd to me is credit mix. Indeed, Wikipedia has a section in its Criticism of credit scoring systems in the United States page about Poor predictor of risk, which is mostly about how a mix of credit can be misleading. For example, I have no installment loans because I don’t own a house, and my car is paid off. Does that make me a worse credit risk? No, in fact, it makes me a better risk because I have lower expenses and so much savings that I always buy cars with cash. The best theory I can come up with is that not having a mix of credit types is predictive of being low-income, since low-income people are less likely to own a home and more likely to own used cars that they don’t have a loan on, or to not even own a car.
This should be promoted to the front page.
I don’t understand what you mean.
The human brain takes time to process sensory signals so that the qualia experienced are slightly delayed from when the sensory input that gave rise to those qualia entered the brain. In that sense, experience happens over time. But at any moment, there is only the qualia that is being experienced. How could it be otherwise? If you say that you then recall that the qualia you had just before that was different. Well, that is a different instant in time in which you are experiencing recall of a memory from your brain.
I share your confusion about the first two, but I believe that the question about the subjective passage of time can be dissolved, unless I am misunderstanding your question.
As you mention, the laws of physics are reversible, but as you mention, entropy gives an arrow of time. That arrow of time causes it to be the case that your brain encodes memories of the past instead of the future. You are not a subjective observer outside of time. You are always experiencing the current moment from the perspective of your brain at that moment. Thus, you always have the experience of being in a brain that remembers the past and remembers the immediately prior moment as the immediate past. So your experience will always be that your current experience has moved forward in time from the past. As a thought experiment, imagine that the 4D block universe existed and that there is some process that evaluates slices at which subjective experience happens. Imagine that instead of that process moving forward in time, it is moving backward in time from the “end” of time to the Big Bang. What would your subjective experience be in that case? It would still be that you are traveling forward in time. Because, as you experienced each moment, you would do so with the memory of the past moment, not of the moment that was experienced sequentially before (which is the temporally future moment). There is therefore no question here except what gives rise to qualia, and perhaps whether the block universe view is correct, or the universe is actually an evolving system for which only some “now” exists for each point in space.
(In reality, I don’t think a block universe makes sense. While I don’t understand what gives rise to qualia, all evidence says that it is tied to the execution of the “thinking” algorithm of my brain. A block universe would have no “execution” and so I think would have no qualia unless qualia exist eternally at all places in the block universe where there is a conscious being.)
The author, @Max Harms, is working on a high-quality AI-read audio book version. He had hoped to release it at the same time as the book, but is currently planning to release it in early 2026. There is a prediction market for When will the Red Heart audiobook come out? There is a preview on YouTube
There are existing crypto algos for “coin flipping”. You should be using one of those. See https://en.wikipedia.org/wiki/Commitment_scheme#Coin_flipping
A browser doesn’t count as a significant business integration?
I worry that this paper and article seem to be safety washing. They imply that existing safety techniques for LLMs are appropriate for more powerful systems. They apply a safety mindset from other domains to AI in an inappropriate way. I do think AI safety can learn from other fields, but those must be fields with an intelligent adversarial agent. Studying whether failure modes are correlated doesn’t matter when you have an intelligent adversary who can make failure modes that would not normally be correlated happen at the same time. If one is thinking only about current systems, then perhaps such an analysis would be helpful. But both the paper and article fail to call that out.
Most of this is interesting but unsurprising. Having reflected on it for a bit, I do find one thing surprising. It is very strange that Illya doesn’t know who is paying his lawyers. Really, he assumes that it is OpenAI and is apparently fine with that. I’m surprised he isn’t concerned about a conflict of interest. I assume he has enough money that he could hire his own lawyers if he wanted. I would expect him to hire at least one lawyer himself to ensure that his own interests are represented and to check the work of the other lawyers.
[Question] Why is OpenAI releasing products like Sora and Atlas?
I signed the statement. My concern, which you don’t address, is that I think the statement should call for a prohibition on AGI, not just ASI. I don’t think there is any meaningful sense in which we can claim that particular developments are likely to lead to AGI, but definitely won’t lead to ASI. History has shown that anytime narrow AI reaches human levels, it is already superhuman. Indeed, if one imagines that tomorrow one had a true AGI (I won’t define AGI here, but imagine an uploaded human that never needs to sleep or rest), then all one would need to do to make ASI is to add more hardware to accelerate thinking or add parallel copies.
As someone who is signed up for Cryonics myself (Alcor), I think a good patient care trust/endowment (PCT) is critical. It is important to take a more conservative approach than many endowments do. Reasons for that include:
Endowments are generally for institutions that are expected to have continued revenue and could adjust prices to compensate for endowment losses. The PCT may outlive the brain preservation org.
PCT needs to survive indefinitely, even through upheavals that other orgs might assume would destroy them or their reason for being
Many endowments get suckered into too much active management, or may be explicitly being taken advantage of by the board appointing active managers
I’ve done a fair amount of research for myself in investing strategies for early retirement. I recommend a low-cost index fund approach. However, you really should carefully evaluate what is a safe perpetual withdrawal rate and non-standard portfolio mixes that are meant to be safe portfolios through all conditions, not just the past 20 years. Examples include “All Seasons Portfolio”, “Permanent Portfolio”, and “Golden Butterfly”. I use the Portfolio Charts, which have excellent stats and analysis for many standard portfolios (including the three I mentioned) and tools for doing back testing on any portfolio mix one might come up with. Interestingly, many of these long-term portfolios actually have surprisingly high perpetual withdrawal rates because of reduced volatility.
I’m happy to discuss further if you private message me.