Related, from Ethan Mollick:
“One thing I mentioned only in passing in my Fable post is that, for long running tasks, Fable starts to develop its own dialect as its many agents and tasks reinforce themselves and make Claudish language ever more Claudish.
You need to ask it to report out in plain English.
This was after a 9 hour task, and it all makes sense, actually, but takes way too much effort to parse, like reading Shakespearian English.”
I’m not sure exactly how I’d set it up but I’d think there’d be opportunities relating to long-run coherence. If we’re imagining a working society populated by present-day LLMs there’d necessarily be some sort of systems aiding in keeping coherence over time but I have to imagine that I would be a lot better in certain ways or in helping specific projects to make me valuable.
There are now 15 competing Types of Guy standards
Doing a bit of trading on Manifold (I’ve been active for around 3 months) has drilled this into me well enough that the general principle of “low enough/high enough markets generally don’t go to their true probability” seems obvious, and that’s after I gained the theoretical knowledge that these things happen from the Jesus Christ returns market. If people on LessWrong are taking nearly all prediction markets purely at their face value, I think that wouldn’t be good, but I don’t think they are.
I will note that Polymarket does have a 4% annualized holding reward (basically 4% interest on your market positions, the rate is variable), so the potential gain isn’t quite as bad as you state. With this in mind, people not betting down a 9% probability does seem meaningfully different to me than if it were five or less percent.
Very funny that the cutoff for Heaven seems to be exactly the amount of karma you had before posting this comment.
It’s actually a little worse than I thought, apparently some of the levels include a “fog-of-war” mechanic where it is essentially just up to chance whether you pick a good path or not. This wouldn’t be so bad on its own but combined with the “take second-best human performance for each level” it’s definitely not a fair evaluation.
I think the benchmark methodology is pretty bad/misleading.
“For each level that is counted, compare the AI agent’s action count to a human baseline, which we define as the second-best human action action[sic]. Ex: If the second-best human completed a level in only 10 actions, but the AI agent took 100 to complete it, then the AI agent scores (10/100)2 for that level, which gets reported as 1%. Note that level scoring is calculated using the square of efficiency.” (See full here)
Defining the human baseline as the second-best human performance per each level (out of 486 participants who were members of the San Francisco general public) doesn’t seem helpful, and the method of scoring the AI results is also unintuitive to me (not to mention it just sets the human score to be 100%).
If the second-best human takes 10 moves to solve a level, and the AI takes 13, its score for that level will be (10/13)2, or about 60%, which seems unreasonable to me, especially since there was nothing in the AI prompt that indicated it was better to finish a level while making the least amount of moves.
They also set a cutoff for the AI of 5x the human performance, after which the AI will score 0 (which means their quoted example of how scoring works couldn’t actually happen, the AI would have been cut off after 50 steps).
They cap the AI’s score at 100% per level even if it were to find a way to complete one in fewer moves than the human baseline, so it’s completely impossible for it to score better than humans, at best it can match the human score if and only if it at least matches the 2nd-best human performance on Every level.
I also imagine it as making a copy, but I’d also expect that people who want their mind uploaded would know of this and would hold their identity such that they consider the copy(ies) to be themself as well. I’m not sure I’d endorse this view of identity,[1] but I don’t really have any issues with people taking it. Does your view on “the original” break with this, or would you just then consider the copy similarly to how you would whole brain emulation? (or something else)
Or at least, I think it would be very risky to get rid of my biological self based on such a view
I find it expected that once there are a variety of autonomous agents, they will begin exhibiting a variety of behaviors, based on differences in architecture, prompting from the human behind them, etc. We can see from stuff like the spiritual bliss attractor state and the GPT 4o parasitology stuff (and more, those are just two things that immediately jump to mind) that talking about consciousness is not a surprising state for LLMs to be in.
I don’t think it’s necessarily appropriate to say that the agents “started feeling conscious”, or that they read all of the philosophy mentioned vs just having it in their training data. I think it’s easy for LLMs to to go into states where they talk about consciousness (and indeed, I think a nontrivial group of people who care about/use LLMs enough to set up autonomous agents would be interested in what they would report on consciousness and prompt them in that direction). Given this, there’s likely some unknown number of autonomous agents mucking about on the internet doing things related to this topic, and as such it’s not particularly surprising a human author would receive an email from one of them.
You can also see that the general behavior is happening a lot on Moltbook, a social media intended to be for AI agents (see the bottom of this post), which is a more recent thing but I think there’d be good reasons to expect the outcome of an AI consciousness researcher getting emailed by an LLM much before any of the Moltbook stuff started happening.
And of course just because this stuff isn’t surprising doesn’t mean it’s not interesting or potentially valuable to know/talk about.
Why do you think it’s mind blowing if it’s legit? To me it’s something that seems pretty expected once there are any sorts of semi-autonomous AI agents (this started quite a while ago looking at the AI Village’s stuff), and as OpenClaw has gained some popularity I’d expect this to be happening pretty often.
I’m guessing Zvi is referencing these parts in the definition of “large developer” and a further section later on pages 3 and 5 respectively.
“Large developer” means a person that has trained at least one frontier model, the compute cost of which exceeds five million dollars, and has spent over one hundred million dollars in compute costs in aggregate in training frontier models...
...Any person who is not a large developer, but who sets out to train a frontier model that if completed as planned would qualify such person as a large developer (I.E. At the end of the training, such person will have spent five million dollars in compute costs on one frontier model and one hundred million dollars in compute costs in aggregate in training frontier models, excluding accredited colleges and universities to the extent such colleges and universities are engaging in academic research) shall, before training such model:
Non-large developers setting out to train a frontier model as described above also have to fill out an SSP but don’t have follow paragraphs C or D of the SSP definition (making a detailed test procedure) which wasn’t really part of your question but now we know.
ETA: I just now see Expertium’s comment where there’s a more recent version of the bill, mostly making this comment superfluous.
Imagine you’re an algebra student and your teacher pretends not to know algebra. Despite the fact that the teacher does know it themselves, you as a student will not learn.
This is very cool and valuable work but I was also distracted by how funny I found this example.
I wouldn’t consider it a common phrase, but I also wouldn’t be surprised at all to hear someone say it given a sensible context.
One flaw in the setup is that the person opposing you could generate a random sequence beforehand and simply follow that when choosing options in the “game.” I assume the offer to play the game is not still available and/or you would not knowingly choose to play it against someone using this strategy, but if you would I’ll take the $25.
I don’t think this analogy works on multiple levels. As far as I know, there isn’t some sort of known probability that scaling laws will continue to be followed as new models are released. While it is true that a new model continuing to follow scaling laws is increased evidence in favor of future models continuing to follow scaling laws, thus shortening timelines, it’s not really clear how much evidence it would be.
This is important because, unlike a coin flip, there are a lot of other details regarding a new model release that could plausibly affect someone’s timelines. A model’s capabilities are complex, human reactions to them likely more so, and that isn’t covered in a yes/no description of if it’s better than the previous one or follows scaling laws.
Also, following your analogy would differ from the original comment since it moves to whether the new AI model follows scaling laws instead of just whether the new AI model is better than the previous one (It seems to me that there could be a model that is better than the previous one yet still markedly underperforms compared to what would be expected from scaling laws).
If there’s any obvious mistakes I’m making here I’d love to know, I’m still pretty new to the space.
Can you expand on this? I’m not sure what you mean but am curious about it.
Two years later, you could get this with Veo 2:
The picture(s) here seem to be missing.
My strong downvotes are giving +1, which is a little confusing.
I’m not sure the example you provide is actually an example of appealing to consequences. To me it seems more like looking at an action at different levels of abstraction rather than starting to think about consequences, although I do think the divide can be unclear. I don’t do philosophy so my ideas of how things may be defined are certainly messy and may not map on well to their technical usage, but I’m reminded of the bit in the sequences regarding Reductionism, where you certainly can have an accurate model of a plane dealing just with particle fields and fundamental forces, but that doesn’t mean the plane doesn’t have wings.
This is to say I think that you can have different moral judgements about an action at different levels of abstraction, but that’s something that can happen before and/or separately from thinking about the consequences of that action.
The United States Government has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States. This includes Anthropic employees who are foreign nationals.
Anthropic is currently disabling Fable 5 and Mythos 5 for all customers.
https://www.anthropic.com/news/fable-mythos-access