An aspiring rationalist who has been involved in the Columbus Rationality community since January 2016.
J Thomas Moros
Karma: 258
[Question] Why is OpenAI releasing products like Sora and Atlas?
As a professional software developer with 20+ years of experience who has repeatedly tried to use AI coding assistants and gotten consistently poor results, I am skeptical of even your statement that, “The average over all of Anthropic for lines of merged code written by AI is much less than 90%, more like 50%.” 50% seems way too high. Or if it is then most of that code is extraneous changes that aren’t part of the core code that executes. For example, I’ve seen what I believe to be AI-generated code where 3⁄4 of the API endpoints are unused. They exist just because the AI assumes that the rest endpoint for each entity ought to have all the actions even though that didn’t make sense in this case.
I think there is a natural tendency for AI enthusiasts to overestimate the amount of useful code they are getting from AI. If we were going to make any statements about how much code was generated by AI at any organization, I think we would need much better data than I have ever seen.
Have you considered that the policies are working correctly for most people with a “normie” communication style? I agree that they should be clearer. However, when I read your description of what they are saying, I think the rule makes sense. It isn’t that everything must be entirely fragrance-free. The intended rule seems to be nothing strongly scented. For example, I’ve met women who use scented shampoo, but you don’t notice it on them even when you are close to them. I’ve also met women who you immediately smell the scent of their shampoo from 3 feet. It seems they are basically asking that people use reasonable judgment. That may not be sufficient for extremely sensitive people, but it will address a lot of the problem. By having it in their code of conduct, they can ask people to leave if it is a problem.
Your Beantown Stomp statement seems to be the proper way to communicate the actually intended policy.
If your goal is to discourage violence, I feel you’ve missed a number of key considerations that you need to address to discourage people. Specifically, I find myself confused by several things:
Why the assumption that the target would be AI researchers instead of AI infrastructure (i.e. training clusters and chip fabs)
You claim that the state would have the force advantage, but independent terrorist actors often have advantages against large state actors. Any violence wouldn’t be some kind of organized militia vs. the state. It would be a loose coalition of “terrorist” cells seeking to disrupt and slow AI development.
You assume a lot of association with mainstream AI safety and reputational harm. However, any violence would likely have to come from a separate faction that would seek to distinguish itself from the mainstream. For example, there are a few ecoterrorist and “deep green” movements, but they are clearly distinct from mainstream climate activism, and I don’t think they have had much impact on mainstream climate activism.
So, by all means, we at LessWrong condemn any attempt to use violence to solve the race to ASI that kills everyone. But, if you were attempting to prevent some group from splintering off to seek a violent means of resistance, I think you’ve somewhat missed the mark.
I did watch this interview, but not his other videos. It does start with the intro from that trailer. However, I did not see it as reflecting a personality cult. Rather, it seemed to me that it was trying to establish Eliezer’s credibility and authority to speak on the subject for people who don’t know who he is. You have to remember that most people aren’t tracking the politics of the rationality community. They are very used to an introduction that hypes up the guest. Yes, it may have been a bit more hyperbolic than I would like, but given how much podcast/interview guests are hyped on other channels and the extent to which Eliezer really is an expert on the subject, much more so than many book authors that get interviewed, it was necessary to lay it on strong.
I’ve been a long-time user of book darts and highly recommend them.
The one other downside is that if they are on the page and catch on something so that they rotate, the clip can cut into the page edge. This can generally be avoided by making sure you put them all the way to the edge of the page and being aware not to let anything drag along the edge of the pages of a book with darts in it.
Do co-ops scale? I would guess they may not. If many firms are larger than the size that co-ops effectively scale to, then we would see more traditional firms than co-ops.
This is why the modal narrative, which is more likely than any other particular story, centers around loss of human control the end of 2027
I think this isn’t a strong enough statement. Indeed, the median narrative is longer. However, even the modal narrative ought to include at least one unspecified obstacle occurring. In a three-year plan, the most frequent scenarios have something go wrong.
I think it is interesting that you think it is not very neglected. I assume you think that because languages like Rust, Kotlin, Go, Swift, and Zig have received various funding levels. Also, academic research is funding languages like Haskell, Scala, Lean, etc.
I suppose that is better than nothing. However, from my perspective, that is mostly funding the wrong things and even funding some of those languages inadequately. As I mentioned, Rust and Go show signs of being pushed to market too soon in ways that will be permanently harmful to the developers using them. Most of those languages aren’t improving programming languages in any meaningful way. They are making very minor changes at the margin. Of the ones I listed, I would say only Rust and Scala have made any real advances in mainstream languages, and Scala is still mired in many problems because of the JVM ecosystem. On the other hand, the Go language has been heavily funded and pushed by Google and has set programming languages back significantly.
I would say there is almost no path to funding a language that is both meant for widespread general use and pushes languages forward. Many of the languages that have received funding did so by luck and were funded too late in the process and underfunded. There is no funding that actually seeks out good early-stage languages and funds them.
Also, many of those languages got funding by luck. Luck is not a funding plan.
[Question] Programming Language Early Funding?
Thanks for the summary of various models of how to figure out what to work on. While reading it, I couldn’t help but focus on my frustration about the “getting paid for it” part. Personally, I want to create a new programming language. I think we are still in the dark age of computer programming and that programming languages suck. I can’t make a perfect language, but I can take a solid step in the right direction. The world could sure use a better programming language if you ask me. I’m passionate about this project. I’m a skilled software developer with a longer career than all the young guns I see. I think I’ve proved with my work so far that I am a top-tier language designer capable of writing a compiler and standard library. But...… this is almost the definition of something you can’t and won’t be paid for. At least not until you’ve already published a successful language. That fact greatly contributes to why we can’t have better programming languages. No one can afford to let them incubate as long as needed. Because of limited resources, everyone has to push to release it as fast as possible. Unlike other software, languages have very strict backward compatibility requirements, so improving them is a challenge and inevitably leads to real issues as the language grows over time. However, they can never fix previous mistakes or address design changes needed to support new features.
I’m confused by the judges’ lack of use of the search capabilities. I think we need more information about how the judges are selected. It isn’t clear to me that they are representative of the kinds of people we would expect to be acting as judges in future scenarios of superintelligent AI debates. For example, a simple and obvious tactic would be to ask both AIs what one ought to search for in order to be able to verify their arguments. An AI that can make very compelling arguments still can’t change the true facts that are known to humanity to suit their needs.
This is not sound reasoning because of selection bias. If any of those predictions had been correct, you would not be here to see it. Thus, you cannot use their failure as evidence.
As someone who believes in moral error theory, I have problems with the moral language (“responsibility to lead ethical lives of personal fulfillment”, “Ethical values are derived from human need and interest as tested by experience.”).
I don’t think that “Life’s fulfillment emerges from individual participation in the service of humane ideals” or “Working to benefit society maximizes individual happiness.” Rather I would say some people find some fulfillment in those things.
I am vehemently opposed to the deathist language of “finding wonder and awe in the joys and beauties of human existence, its challenges and tragedies, and even in the inevitability and finality of death.” Death is bad and should not be accepted.
I assume there are other things I would disagree with, but those are a few that stand out when skimming it.
I agree with your three premises. However, I would recommend using a different term than “humanism”.
Humanism is more than just the broad set of values you described. It is also a specific movement with more specific values. See for example the latest humanist manifesto. I agree with what you described as “humanism” but strongly reject the label humanist because I do not agree with the other baggage that goes with it. If possible, try to come up with a term that directly states the value you are describing. Perhaps something along the lines of “human flourishing as the standard of value”?
I am signed up for cryonics with Alcor and did so in 2017. I checked and the two options you listed are consistent with the options I was given. I didn’t have a problem with them, but I can understand your concern.
I have had a number of interactions with Alcor staff both during the signup process and since. I always found them pleasant and helpful. I’m sorry to hear that you are having a bad experience. My suggestion would be to get the representative on the phone and discuss your concerns. Obviously, final wording should be handled in writing but I think a phone conversation would help you both understand what would be acceptable to both of you.
In my opinion, the responses you have gotten probably arise from one of two sources. It is possible that she simply didn’t read what you wrote carefully enough and fell back to boilerplate language that is closer to what their legal counsel has approved. She likely doesn’t have the authority to accept major changes herself. If that is not what happened, then it is most likely that Alcor is trying to push the option they are pushing to avoid legal issues, the issues they have had with family in the past, and delay in cryopreservation. They want a clear-cut decision procedure that doesn’t depend on too many third parties. If cryopreservation is to go well, it needs to be done in a timely fashion. Ideally, you want whoever is performing it to have a clear and immediate path to begin if it is warranted. Any judgment call or requirement to get consent could cause unnecessary delays. You might think it will be clear, but any chance your wife could claim that she should have been consulted and wasn’t could cause legal problems. Thus, Alcor may be forced to consult her in all but the most clear-cut cases. Again, just schedule a call.
As a proponent of cryonics, I hope you will persist and work through this issue. Please message me if there are other questions I can answer for you. If you choose not to proceed, you can choose to keep the insurance policy and designate another recipient rather than canceling it.
P.S. Having researched all the Cryonics organizations, Alcor is by far the best. They are still small but they are working the hardest to become a fully professional organization. Their handling of the legal issues and financial structure is much better. The Cryonics Institute (CI) is run by well-meaning people who are less professional. They are more of a volunteer organization. Having attended a CI annual meeting I was disappointed by the insufficiently conservative and far-sighted investment strategy. I think CI may actually be underfunded for the goal of existing 100 years from now.
While I can understand why many would view advances toward WBE as an AI-safety risk, many in the community are also concerned with cryonics. WBE is an important option for the revival of cryonics patients. So I think the desirability of WBE should be clear. It just may be the case that we need to develop safe AI first.
As someone interested in seeing WBE become a reality, I have also been disappointed by the lack of progress. I would like to understand the reasons for this better. So I was interested to read this post, but you seem to be conflating two different things. The difficulty of simulating a worm and the difficulty of uploading a worm. There are a few sentences that hint both are unsolved, but they should be clearly separated.
Uploading a worm requires being able to read the synaptic weights, thresholds, and possibly other details from an individual worm. Note that it isn’t accurate to say it must be alive. It would be sufficient to freeze an individual worm and then spend extensive time and effort reading that information. Nevertheless, I can imagine that might be very difficult to do. According to wormbook.org, C. elegans has on the order of 7,000 synapses. I am not sure we know how to read the weight and threshold of a synapse. This strikes me as a task requiring significant technological development that isn’t in line with existing research programs. That is, most research is not attempting to develop the technology to read specific weights and thresholds. So it would require a significant well-funded effort focused specifically on it. I am not surprised this has not been achieved given reports of lack of funding. Furthermore, I am not surprised there is a lack of funding for this.
Simulating a worm should only require an accurate model of the behavior of the worm nervous system and a simulation environment. Given that all C. elegans have the same 302 neurons this seems like it should be feasible. Furthermore, the learning mechanism of individual neurons, operation of synapses, etc. should all be things researchers outside of the worm emulation efforts should be interested in studying. Were I wanting to advance the state of the art, I would focus on making an accurate simulation of a generic worm that was capable of learning. Then simulate it in an environment similar to its native environment and try to demonstrate that it eventually learned behavior matching real C. elegans including under conditions which C. elegans would learn. That is why I was very disappointed to learn that the “simulations are far from realistic because they are not capable of learning.” It seems to me this is where the research effort should focus and I would like to hear more about why this is challenging and hasn’t already been done.
I believe that worm uploading is not needed to make significant steps toward showing the feasibility of WBE. The kind of worm simulation I describe would be more than sufficient. At that point, reading the weights and thresholds of an individual worm becomes only an engineering problem that should be solvable given a sufficient investment or level of technological advancement.
A study by Alcor trained C. elegans worms to react to the smell of a chemical. They then demonstrated that the worms retained this memory even after being frozen and revived. Were it possible to upload a worm, the same exact test would show that you had successfully uploaded a worm with that memory vs. one without that memory.
Study here: Persistence of Long-Term Memory in Vitrified and Revived Caenorhabditis elegans
I signed the statement. My concern, which you don’t address, is that I think the statement should call for a prohibition on AGI, not just ASI. I don’t think there is any meaningful sense in which we can claim that particular developments are likely to lead to AGI, but definitely won’t lead to ASI. History has shown that anytime narrow AI reaches human levels, it is already superhuman. Indeed, if one imagines that tomorrow one had a true AGI (I won’t define AGI here, but imagine an uploaded human that never needs to sleep or rest), then all one would need to do to make ASI is to add more hardware to accelerate thinking or add parallel copies.