The alignment of mundane, current AI systems is highly relevant for the alignment of superintelligence, if only because the explicit strategy of the model labs is to build a recursively self-improving AI, whose resultant superintelligence would obviously rely on how well that initial seed AI was aligned to human values.
lc
Karma: 13,433
This is pretty galaxy brained but I can see an argument for having two companies doing different approaches.
I don’t see why the funding situation will be any different at end of year. Everybody who works for e.g. Ant & OpenAI will have the same reasons to not sell their equity that they did at the beginning of the year. It’s not like Ant researchers are having a hard time finding buyers.
It’s disempowerment because one further meta-level up, the humans have lost the ability to monitor the architecture of their computer programs, because they’re locked in power struggles with other corporations who have already delegated their authorship of code to computer programs.
Why aren’t translators facing employment difficulties yet? Are they, and we just haven’t noticed? Seems relevant to predicting any near-term disemployment from, say, software engineering.
Do you believe the US government has access to better polygraphs than is commercially available and/or studied in the academic literature? If so do you have non-indirect evidence for this?
Yeah; my evidence for this is strewn across public interviews with ‘former’ CIA officers, books about the U.S. intelligence community, and personal discussions with family members who are/were officers in the U.S. military. It would take time I don’t have to go and gather it all but for the basic “polygraphs they are using seem to work” I think you can just look at what people who have recently left have said, and the (non-)history of long term employed defectors over the last ~15y.
I will add in passing that the CIA requires a polygraph screening for all employees until today despite all the research showing it to be ineffective at telling the truth from lies
The evidence for this is not very good, and is also not assessing the polygraph assessment the modern US government actually conducts. I think it’s very plausible that they have worked fairly reliably for the last ~25 years given the dearth of publicly disclosed cases of long-running spying ala Aldrich Ames or Robert Hanssen, and given how much former intelligence officers seem to believe it works & comment on their reliability in passing.
Firing people really sucks.
time to let the fame change me
I think he means “their legitimately self-authored writing”, not “their ai-generated writing”.
Feel like LessWrong should have a jobs board.
I’m no schwami but honestly all of this makes Tesla sound like a really bad bet. Why would I invest in a *public 1T$ company* who’s primary pitch is their ability to raise future capital?
hate sending emails to important people and waiting for them to respond
Consistency matters more than peak quality
Don’t think this particular bit follows from your other points, and think it’s contradicted by other anecdotal evidence about top performers in winner take all fields. If anything making more 80⁄20 decisions increases the number of bangers at the expense of average quality.
Rest seems accurate though.
I’ve also called 911 a couple times, and they’ve always shown up reasonably quickly. Only for thought-to-be medical emergencies though.
In my personal experience (for SWE work), Claude is much more likely to bullshit me, be openly sycophantic, and generally reward-hack than Codex. I wouldn’t have expected that to be the case ex-facto given the public persona of the two companies, but it’s true, and as far as I can tell it’s a factor in Claude’s commercial success.
Thank you for linking this wonderful piece. I have a similar criticism of how rationalists & SF relate to institutions.
There’s no hard line. Current, mundane AIs are already being used to e.g. detect reward hacking in vendor RL environments. And as we scale to superintelligence the tasks & amount of trust we hand to AI models will ramp up.