UWaterloo Electrical Engineering student
Komodo
Karma: 137
AA also shuffled around their benchmarks a lot after Astra was scored super low on release despite clearly being good. Opus 5 on release was also amazing on literally every benchmark but everyone hated it (but this may have been a vibes and usability issue rather than the model being dumb). Generally I have no idea what benchmarks are good anymore.
I have had the complete opposite experience. I have almost completely cut Reddit out of my life and have found Twitter to be much more benign. Maybe 2-3% of Twitter is ragebait for me, and most of that is so obviously bait that I don’t get emotionally affected
Maybe I am part of some a/b test. For example, I saw everyone else complaining about the constant posts about Corgi, while I saw next to nothing from them. I think I also do a good job at blocking some of this stuff out, e.g. I saw a post about Abdul El-Sayed with Hasan also in the body text, and made the decision to completely ignore him. I have since seen a lot of posts about El-Sayed. I know absolutely nothing about him beyond him being a Dem. I also held off from Lindsay Clancy for several weeks.
I dont think it’s polarized. AI sentiment is very broadly negative. Trump is just doing his own thing to the dismay of most of his base (similar to Iran, data centers, flock cameras).
OpenAI is making very competitive models (with lax safety practices), there is still little progress on any sort of slowdown, and the models are still misaligned. This along with OpenAI’s Sebastian Bubeck supposedly acting like a Mafia boss in the recent Navier-Stokes controversy, and the possibility of a full speed race to the bottom is very salient.
If it’s something DSV4 Flash sized it could survive on 256 GB RAM regular CPU based servers / workstations.
I don’t think trying to build guardrails/monitoring systems around a misaligned A(G/S?)I that you are letting operate very autonomously and giving large amounts of control to (since it’s doing a lot of stuff in the company) is possible.
If it has gotten to the point where AI is successfully scheming hard enough to plant evidence to get an employee fired while being smart enough that we needed a special system to figure out if the evidence was planted by AI, I think we are already past the point of no return and we are all screwed so I don’t think it really matters (I don’t think trying to build guardrails around what’s pretty much completely misaligned and free roaming ASI at that point is going to accomplish much)
In the original everything is clearly NOT fine but we pretend it is. Over here everything seems fine but we know things are NOT fine we are panicking.
I mainly want to see what strategy they used out of curiosity. I don’t think they are just running RLHF on whatever deep-learning based super efficient learner novel architecture they cooked up. Would be interesting to see!
In his Dwarkesh interview, Ilyas two main reasons for why they might not straight shot ASI and instead release a product were roughly
1. Timelines end up being long
2. Try and get the real world used to and prepared for ASI, while also getting feedback for how safe and robust it is
But if they are just 10x’ing compute out of nowhere with a 5 billion dollar investment, and all the big labs seem confident they are <5 years out from AGI, that seems to nullify 1, and traditional LLM’s are strong enough that they seem to be getting the world used to very powerful AI through e.g. Mythos/Glasswing and the Hugging Face incident, which seems to nullify 2. So I think it is likely that they don’t release a regular product anytime soon and instead are going to just come out with ASI randomly (with some USG involvement).
Autonomy/agency/long-term planning—seems like a likely but-for cause of most takeover stories. Most takeover stories I can think about entail superhuman long-term planning, or at minimum peak human planning + coordination across instances. Takeovers without that always seem either flimsy or entail seemingly a “magical” technological edge in one or multiple other domains.
According to Reuters investigating the OpenAI HuggingFace incident: “agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI’s infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints”
So it does appear to be something todays models are trying!
I think prediction markets / sports betting will face a downturn post AGI
Young people do less “traditional” gambling because everyone knows the odds are against you, while with sports betting / prediction markets there is the feeling of skill (I say feeling because the vast majority lose money from crossing the bid-ask spread, fees, etc.). However if you go on your phone in 20XX and see DeepMind go “we are proud to announce we have recently solved Navier Stokes” I think it would be much more “in your face” the odds are perfectly calibrated and you are literally just playing roulette at that point.
Makes sense. I did not look at it this way before.
Internship grinding in the probable pre-apocalypse
An early draft of my sort of Turning 20 in the probable pre-apocalypse.
I go to a prestigious grind heavy school. Every day in lectures I see people ignoring the prof and applying for internships. People still behave as if it’s business as usual and California is waiting on the other side of the grind. It does not seem like CS where people much more clearly see AI looming over them. It does however seem like everything is going to melt down in the coming years (at least before I graduate). So the entire game is completely falling apart, and it is very likely I am wasting my time (as I am not enjoying this).
So the question naturally is: what am I supposed to be doing right now?
Timelines are so absurdly short. The idea behind doing a difficult program was it was a long term investment. But the investment is never going to pay off. I look at Yudkowsky and he is talking about “Death with Dignity” and then I go look at Metaculus and it’s “11% higher underemployment rate in 2035 :P”. What is happening? Who knows! What should I be doing? Who knows! I know staying in school is probably not getting me anywhere.
The information environment is so incredibly chaotic. I see a lot of people say “there will always be jobs, don’t worry :D, keep doing what you’re doing”, and then I see more grounded “AI will replace a lot of white collar jobs, you need to pivot into human facing or physical jobs.”, and then “Agent-5 will bioweapon us”,..., you’ve all read about this. So as somebody trying to figure things out, this entire thing is a mess. It is clear superintelligence is on the horizon and all my grinding to be competent will not have paid off, but the path from point A to B is completely unclear.
It is difficult to write at length despite how much I think about it because all it really is is “I don’t know what the future is at all, I don’t know how to plan for it at all, I don’t know what I am supposed to be doing at all, nobody is really talking about this at all, and also ‘real life’ is also stressful as well.” I generally just keep playing the game and apply for internships and get my grades up and do projects, even though it feels completely meaningless, because what on earth am I supposed to do. The most viable path I see is dropping out and joining the CAF, thereby dodging white collar unemployment and the blue collar job crisis that would come from the consequential recession, and I can coast and skip directly to the finish line (UBI luxury communism or extinction).
Oh Well! What can you do. Could be worse.
The Mythos checkpoint that the AISI ran their cyber evals on was actually an earlier version of the model. The one released with project Glasswing is much stronger.
Caring about the weak is not a trait that would be expected to naturally arise from ASI. Humans care about the weak because evolutionarily, taking care of the weak in the tribe was beneficial, and so it got “trained” into us (mostly). ASI would not naturally “evolve” to care about the weak unless we give it incentive to if it had an animal like brain.
I think for Gen Z / Gen Alpha, UBI will melt them. They hang out with friends less, have less kids, scroll more, have less sex,..., you get the point. If you take a 14 year old and say “getting social status is meaningless since everyone is on government welfare, going to the gym is pointless because everyone is jacked from the myostatin inhibitors Eli Lilly put out, talking to a woman is pointless since there are sex bots walking around that are more charismatic and attractive than any human that has ever lived, we have AI generated VR short form video, and also you never formed a “real life” since you grew up on Instagram Reels, anyways go figure it out” they will completely melt down.
I think for people that have already formed a structured life, they can cope with it. But for a lot of kids, they have none of that structure and so they will get torpedoed by extreme hedonistic pleasure given the freedom to.
Komodo’s Shortform
I will likely never graduate, which is rather inconvenient.
In my senior year of high school when I saw o1 release, I thought if AI continued to get significantly better, I would not go to university and instead go into the trades. o3 came out and blew me away, but I went to university anyways because I got my fancy Waterloo Engineering offer which I worked very hard for, and was much less weird than leaving it and going “trust me bro the robots are coming”.
The writing is very much on the wall at this point. I am very unsure about what the correct move is / what I “should” be doing right now. When I see people on here talk about giving advice to young people, they generally go “I have no idea what to tell them, stay away from SWE I guess?”
Morale is very low, as it seems like nothing I do particularly changes anything on a timescale of >3 years, and as the models get better and better, the actions I take become more and more unimportant. When I say unimportant, I primarily mean career wise / social-status wise, which is what I primarily care about.
Some possible life paths I have:Just stay in school and enjoy my life and hang out with friends until either UBI or doom. This is a very big shift from how I think about life, as I mostly just care about my career.
Drop out and go do trades in the CAF. This protects me from the imminent white collar job crisis, and also the construction recession I suspect would also happen in the civilian trades from the larger scale economic turmoil.
Drop out a year from now and then go do trades in the CAF. This would make me look less weird as it becomes more and more clear that white collar labor (and thus my education) has no future.
Go heavily into AI safety / AI political activism. This is the most “correct” choice in my eyes, but that brings the morale issue back into play, as my impact on risk almost certainly will round down to zero.
Most of these options are enormous shifts from the very “safe and stable” life path I was on, which is inconvenient.
There’s a few major camps
It’s all sci-fi speculation, it’s going to be fine.
Cal Newport
It’s not agents it’s just prompt loops. AI’s are just pattern matching from all the scifi stories when they talk about “permadeath”. It is not misalignment, long running unsupervised prompt loops should be expected to run amok. The reasoning traces aren’t actually reasoning. AI companies in an effort to make models better when scaling slowed just decided to reward LLM’s for outputting “reasoning traces” but it’s not actual reasoning, it’s just spitting random stuff out because it was rewarded for it. source: Chain-Of-Thought Reasoning In The Wild Is Not Always Faithful. Misalignment is not a real thing, there’s no entity behind the screen. It’s a dog with a weed wacker on it running around.
Yann LeCun
LLM’s are nowhere close to human level intelligence and will take us nowhere close, AGI will not cause mass unemployment or a “singularity”. Technological progress will hit limits and not just shoot off into infinity.
Ed Zitron
LLM’s have some uses but are gimmicks and nowhere close to real AGI. They rely on massive amounts of money to make benchmark number go up, while in practice they remain unreliable. and not very useful All the companies are passing money around to pump up the numbers and the entire funding scheme is smoke and mirrors.
Something might go really really bad, but we won’t go extinct
Gary Marcus
Generally the historic view was LLM’s are good at some things, but we need neurosymbolic reasoning for true AGI, LLM’s are nowhere close. His view is shifting though. In his eyes current models are actually neurosymbolic (harnesses are neurosymbolic, Astra making algebraic notation on the fly to solve puzzles is neurosymbolic reasoning).
and in another post talking about why he doubts extinction he said
which is a take I guess. I saw a similar take from Andrew McAfee (AI won’t be able to take out all of humanity, we are not that weak, I saw geohot say similar things in his debate with yudkowsky on the dwarkesh pod)
I think the Gary Marcus view is gaining traction (it poses large dangers, but don’t really believe in catastrophic risks). I see it with Ezra Klein and Bill Gates. I could definitely find some quotes from them but it’s late at night.
AI is going to transform the world and doomers are just pessimists
Accelerationists
A lot of these guys don’t even really think we’ll solve alignment. From there it ranges from just being happy to be along for the ride (accelerate harder subset), just happy to be along for the ride (can’t do anything about it so may as well enjoy it subset).
Also a sector think that alignment is solvable, a sector think the AI would keep us around because we would be useful (thinking it would kill us all for some mysterious reason is insane pessimism), a sector think as it becomes smarter it would be kind, a sector think that since it starts from human data, it is like a average of humans and wouldn’t be a weird alien.
A sector are just successionists.
Lot’s of views under this camp
Jensen Huang / Marc Andreessen
AI is normal technology that is going to transform the world. Arguments about doom are just pessimists just trying to slow us down. Alignment is an engineering problem that can be solved, and AI is built, designed, and controlled by us. I think the high level view is the same, Marc is just more singularity pilled.
Accelerationists and Huang-Andreessen exist on a sort of spectrum, sort of two axes, one is the future will be normal / singularity and the other is AI is software / this is an alien.
Huang-Andreessen is defo a view I see with ambitious high functioning go getters. Singularity pilled accelerationists I don’t really see much in reality (Mike Israetel but nobody else really comes to mind).
This is probably sloppily written it’s late at night but I think I covered all the major stuff.