I set up a manifold market about the race outcome here:
Philipreal
Karma: 318
Philipreal’s Shortform
The United States Government has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States. This includes Anthropic employees who are foreign nationals.
Anthropic is currently disabling Fable 5 and Mythos 5 for all customers.
Related, from Ethan Mollick:
“One thing I mentioned only in passing in my Fable post is that, for long running tasks, Fable starts to develop its own dialect as its many agents and tasks reinforce themselves and make Claudish language ever more Claudish.
You need to ask it to report out in plain English.
This was after a 9 hour task, and it all makes sense, actually, but takes way too much effort to parse, like reading Shakespearian English.”
I’m not sure exactly how I’d set it up but I’d think there’d be opportunities relating to long-run coherence. If we’re imagining a working society populated by present-day LLMs there’d necessarily be some sort of systems aiding in keeping coherence over time but I have to imagine that I would be a lot better in certain ways or in helping specific projects to make me valuable.
There are now 15 competing Types of Guy standards
Doing a bit of trading on Manifold (I’ve been active for around 3 months) has drilled this into me well enough that the general principle of “low enough/high enough markets generally don’t go to their true probability” seems obvious, and that’s after I gained the theoretical knowledge that these things happen from the Jesus Christ returns market. If people on LessWrong are taking nearly all prediction markets purely at their face value, I think that wouldn’t be good, but I don’t think they are.
I will note that Polymarket does have a 4% annualized holding reward (basically 4% interest on your market positions, the rate is variable), so the potential gain isn’t quite as bad as you state. With this in mind, people not betting down a 9% probability does seem meaningfully different to me than if it were five or less percent.
Very funny that the cutoff for Heaven seems to be exactly the amount of karma you had before posting this comment.
It’s actually a little worse than I thought, apparently some of the levels include a “fog-of-war” mechanic where it is essentially just up to chance whether you pick a good path or not. This wouldn’t be so bad on its own but combined with the “take second-best human performance for each level” it’s definitely not a fair evaluation.
I think the benchmark methodology is pretty bad/misleading.
“For each level that is counted, compare the AI agent’s action count to a human baseline, which we define as the second-best human action action[sic]. Ex: If the second-best human completed a level in only 10 actions, but the AI agent took 100 to complete it, then the AI agent scores (10/100)2 for that level, which gets reported as 1%. Note that level scoring is calculated using the square of efficiency.” (See full here)
Defining the human baseline as the second-best human performance per each level (out of 486 participants who were members of the San Francisco general public) doesn’t seem helpful, and the method of scoring the AI results is also unintuitive to me (not to mention it just sets the human score to be 100%).
If the second-best human takes 10 moves to solve a level, and the AI takes 13, its score for that level will be (10/13)2, or about 60%, which seems unreasonable to me, especially since there was nothing in the AI prompt that indicated it was better to finish a level while making the least amount of moves.
They also set a cutoff for the AI of 5x the human performance, after which the AI will score 0 (which means their quoted example of how scoring works couldn’t actually happen, the AI would have been cut off after 50 steps).
They cap the AI’s score at 100% per level even if it were to find a way to complete one in fewer moves than the human baseline, so it’s completely impossible for it to score better than humans, at best it can match the human score if and only if it at least matches the 2nd-best human performance on Every level.
I also imagine it as making a copy, but I’d also expect that people who want their mind uploaded would know of this and would hold their identity such that they consider the copy(ies) to be themself as well. I’m not sure I’d endorse this view of identity,[1] but I don’t really have any issues with people taking it. Does your view on “the original” break with this, or would you just then consider the copy similarly to how you would whole brain emulation? (or something else)
- ^
Or at least, I think it would be very risky to get rid of my biological self based on such a view
- ^
I find it expected that once there are a variety of autonomous agents, they will begin exhibiting a variety of behaviors, based on differences in architecture, prompting from the human behind them, etc. We can see from stuff like the spiritual bliss attractor state and the GPT 4o parasitology stuff (and more, those are just two things that immediately jump to mind) that talking about consciousness is not a surprising state for LLMs to be in.
I don’t think it’s necessarily appropriate to say that the agents “started feeling conscious”, or that they read all of the philosophy mentioned vs just having it in their training data. I think it’s easy for LLMs to to go into states where they talk about consciousness (and indeed, I think a nontrivial group of people who care about/use LLMs enough to set up autonomous agents would be interested in what they would report on consciousness and prompt them in that direction). Given this, there’s likely some unknown number of autonomous agents mucking about on the internet doing things related to this topic, and as such it’s not particularly surprising a human author would receive an email from one of them.
You can also see that the general behavior is happening a lot on Moltbook, a social media intended to be for AI agents (see the bottom of this post), which is a more recent thing but I think there’d be good reasons to expect the outcome of an AI consciousness researcher getting emailed by an LLM much before any of the Moltbook stuff started happening.
And of course just because this stuff isn’t surprising doesn’t mean it’s not interesting or potentially valuable to know/talk about.
Why do you think it’s mind blowing if it’s legit? To me it’s something that seems pretty expected once there are any sorts of semi-autonomous AI agents (this started quite a while ago looking at the AI Village’s stuff), and as OpenClaw has gained some popularity I’d expect this to be happening pretty often.
You can remove it by downvoting your own react. Not sure if it works with reactions that other people have already voted on though.