UWaterloo Electrical Engineering student
Komodo
Karma: 96
I don’t think trying to build guardrails/monitoring systems around a misaligned A(G/S?)I that you are letting operate very autonomously and giving large amounts of control to (since it’s doing a lot of stuff in the company) is possible.
If it has gotten to the point where AI is successfully scheming hard enough to plant evidence to get an employee fired while being smart enough that we needed a special system to figure out if the evidence was planted by AI, I think we are already past the point of no return and we are all screwed so I don’t think it really matters (I don’t think trying to build guardrails around what’s pretty much completely misaligned and free roaming ASI at that point is going to accomplish much)
In the original everything is clearly NOT fine but we pretend it is. Over here everything seems fine but we know things are NOT fine we are panicking.
I mainly want to see what strategy they used out of curiosity. I don’t think they are just running RLHF on whatever deep-learning based super efficient learner novel architecture they cooked up. Would be interesting to see!
In his Dwarkesh interview, Ilyas two main reasons for why they might not straight shot ASI and instead release a product were roughly
1. Timelines end up being long
2. Try and get the real world used to and prepared for ASI, while also getting feedback for how safe and robust it is
But if they are just 10x’ing compute out of nowhere with a 5 billion dollar investment, and all the big labs seem confident they are <5 years out from AGI, that seems to nullify 1, and traditional LLM’s are strong enough that they seem to be getting the world used to very powerful AI through e.g. Mythos/Glasswing and the Hugging Face incident, which seems to nullify 2. So I think it is likely that they don’t release a regular product anytime soon and instead are going to just come out with ASI randomly (with some USG involvement).
Autonomy/agency/long-term planning—seems like a likely but-for cause of most takeover stories. Most takeover stories I can think about entail superhuman long-term planning, or at minimum peak human planning + coordination across instances. Takeovers without that always seem either flimsy or entail seemingly a “magical” technological edge in one or multiple other domains.
According to Reuters investigating the OpenAI HuggingFace incident: “agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI’s infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints”
So it does appear to be something todays models are trying!
I think prediction markets / sports betting will face a downturn post AGI
Young people do less “traditional” gambling because everyone knows the odds are against you, while with sports betting / prediction markets there is the feeling of skill (I say feeling because the vast majority lose money from crossing the bid-ask spread, fees, etc.). However if you go on your phone in 20XX and see DeepMind go “we are proud to announce we have recently solved Navier Stokes” I think it would be much more “in your face” the odds are perfectly calibrated and you are literally just playing roulette at that point.
Makes sense. I did not look at it this way before.
Internship grinding in the probable pre-apocalypse
An early draft of my sort of Turning 20 in the probable pre-apocalypse.
I go to a prestigious grind heavy school. Every day in lectures I see people ignoring the prof and applying for internships. People still behave as if it’s business as usual and California is waiting on the other side of the grind. It does not seem like CS where people much more clearly see AI looming over them. It does however seem like everything is going to melt down in the coming years (at least before I graduate). So the entire game is completely falling apart, and it is very likely I am wasting my time (as I am not enjoying this).
So the question naturally is: what am I supposed to be doing right now?
Timelines are so absurdly short. The idea behind doing a difficult program was it was a long term investment. But the investment is never going to pay off. I look at Yudkowsky and he is talking about “Death with Dignity” and then I go look at Metaculus and it’s “11% higher underemployment rate in 2035 :P”. What is happening? Who knows! What should I be doing? Who knows! I know staying in school is probably not getting me anywhere.
The information environment is so incredibly chaotic. I see a lot of people say “there will always be jobs, don’t worry :D, keep doing what you’re doing”, and then I see more grounded “AI will replace a lot of white collar jobs, you need to pivot into human facing or physical jobs.”, and then “Agent-5 will bioweapon us”,..., you’ve all read about this. So as somebody trying to figure things out, this entire thing is a mess. It is clear superintelligence is on the horizon and all my grinding to be competent will not have paid off, but the path from point A to B is completely unclear.
It is difficult to write at length despite how much I think about it because all it really is is “I don’t know what the future is at all, I don’t know how to plan for it at all, I don’t know what I am supposed to be doing at all, nobody is really talking about this at all, and also ‘real life’ is also stressful as well.” I generally just keep playing the game and apply for internships and get my grades up and do projects, even though it feels completely meaningless, because what on earth am I supposed to do. The most viable path I see is dropping out and joining the CAF, thereby dodging white collar unemployment and the blue collar job crisis that would come from the consequential recession, and I can coast and skip directly to the finish line (UBI luxury communism or extinction).
Oh Well! What can you do. Could be worse.
The Mythos checkpoint that the AISI ran their cyber evals on was actually an earlier version of the model. The one released with project Glasswing is much stronger.
Caring about the weak is not a trait that would be expected to naturally arise from ASI. Humans care about the weak because evolutionarily, taking care of the weak in the tribe was beneficial, and so it got “trained” into us (mostly). ASI would not naturally “evolve” to care about the weak unless we give it incentive to if it had an animal like brain.
I think for Gen Z / Gen Alpha, UBI will melt them. They hang out with friends less, have less kids, scroll more, have less sex,..., you get the point. If you take a 14 year old and say “getting social status is meaningless since everyone is on government welfare, going to the gym is pointless because everyone is jacked from the myostatin inhibitors Eli Lilly put out, talking to a woman is pointless since there are sex bots walking around that are more charismatic and attractive than any human that has ever lived, we have AI generated VR short form video, and also you never formed a “real life” since you grew up on Instagram Reels, anyways go figure it out” they will completely melt down.
I think for people that have already formed a structured life, they can cope with it. But for a lot of kids, they have none of that structure and so they will get torpedoed by extreme hedonistic pleasure given the freedom to.
Komodo’s Shortform
I will likely never graduate, which is rather inconvenient.
In my senior year of high school when I saw o1 release, I thought if AI continued to get significantly better, I would not go to university and instead go into the trades. o3 came out and blew me away, but I went to university anyways because I got my fancy Waterloo Engineering offer which I worked very hard for, and was much less weird than leaving it and going “trust me bro the robots are coming”.
The writing is very much on the wall at this point. I am very unsure about what the correct move is / what I “should” be doing right now. When I see people on here talk about giving advice to young people, they generally go “I have no idea what to tell them, stay away from SWE I guess?”
Morale is very low, as it seems like nothing I do particularly changes anything on a timescale of >3 years, and as the models get better and better, the actions I take become more and more unimportant. When I say unimportant, I primarily mean career wise / social-status wise, which is what I primarily care about.
Some possible life paths I have:Just stay in school and enjoy my life and hang out with friends until either UBI or doom. This is a very big shift from how I think about life, as I mostly just care about my career.
Drop out and go do trades in the CAF. This protects me from the imminent white collar job crisis, and also the construction recession I suspect would also happen in the civilian trades from the larger scale economic turmoil.
Drop out a year from now and then go do trades in the CAF. This would make me look less weird as it becomes more and more clear that white collar labor (and thus my education) has no future.
Go heavily into AI safety / AI political activism. This is the most “correct” choice in my eyes, but that brings the morale issue back into play, as my impact on risk almost certainly will round down to zero.
Most of these options are enormous shifts from the very “safe and stable” life path I was on, which is inconvenient.
If it’s something DSV4 Flash sized it could survive on 256 GB RAM regular CPU based servers / workstations.