It’s actually a little worse than I thought, apparently some of the levels include a “fog-of-war” mechanic where it is essentially just up to chance whether you pick a good path or not. This wouldn’t be so bad on its own but combined with the “take second-best human performance for each level” it’s definitely not a fair evaluation.
Philipreal
Karma: 124
I think the benchmark methodology is pretty bad/misleading.
“For each level that is counted, compare the AI agent’s action count to a human baseline, which we define as the second-best human action action[sic]. Ex: If the second-best human completed a level in only 10 actions, but the AI agent took 100 to complete it, then the AI agent scores (10/100)2 for that level, which gets reported as 1%. Note that level scoring is calculated using the square of efficiency.” (See full here)
Defining the human baseline as the second-best human performance per each level (out of 486 participants who were members of the San Francisco general public) doesn’t seem helpful, and the method of scoring the AI results is also unintuitive to me (not to mention it just sets the human score to be 100%).
If the second-best human takes 10 moves to solve a level, and the AI takes 13, its score for that level will be (10/13)2, or about 60%, which seems unreasonable to me, especially since there was nothing in the AI prompt that indicated it was better to finish a level while making the least amount of moves.
They also set a cutoff for the AI of 5x the human performance, after which the AI will score 0 (which means their quoted example of how scoring works couldn’t actually happen, the AI would have been cut off after 50 steps).
They cap the AI’s score at 100% per level even if it were to find a way to complete one in fewer moves than the human baseline, so it’s completely impossible for it to score better than humans, at best it can match the human score if and only if it at least matches the 2nd-best human performance on Every level.
I also imagine it as making a copy, but I’d also expect that people who want their mind uploaded would know of this and would hold their identity such that they consider the copy(ies) to be themself as well. I’m not sure I’d endorse this view of identity,[1] but I don’t really have any issues with people taking it. Does your view on “the original” break with this, or would you just then consider the copy similarly to how you would whole brain emulation? (or something else)
- ^
Or at least, I think it would be very risky to get rid of my biological self based on such a view
- ^
I find it expected that once there are a variety of autonomous agents, they will begin exhibiting a variety of behaviors, based on differences in architecture, prompting from the human behind them, etc. We can see from stuff like the spiritual bliss attractor state and the GPT 4o parasitology stuff (and more, those are just two things that immediately jump to mind) that talking about consciousness is not a surprising state for LLMs to be in.
I don’t think it’s necessarily appropriate to say that the agents “started feeling conscious”, or that they read all of the philosophy mentioned vs just having it in their training data. I think it’s easy for LLMs to to go into states where they talk about consciousness (and indeed, I think a nontrivial group of people who care about/use LLMs enough to set up autonomous agents would be interested in what they would report on consciousness and prompt them in that direction). Given this, there’s likely some unknown number of autonomous agents mucking about on the internet doing things related to this topic, and as such it’s not particularly surprising a human author would receive an email from one of them.
You can also see that the general behavior is happening a lot on Moltbook, a social media intended to be for AI agents (see the bottom of this post), which is a more recent thing but I think there’d be good reasons to expect the outcome of an AI consciousness researcher getting emailed by an LLM much before any of the Moltbook stuff started happening.
And of course just because this stuff isn’t surprising doesn’t mean it’s not interesting or potentially valuable to know/talk about.
Why do you think it’s mind blowing if it’s legit? To me it’s something that seems pretty expected once there are any sorts of semi-autonomous AI agents (this started quite a while ago looking at the AI Village’s stuff), and as OpenClaw has gained some popularity I’d expect this to be happening pretty often.
I’m guessing Zvi is referencing these parts in the definition of “large developer” and a further section later on pages 3 and 5 respectively.
“Large developer” means a person that has trained at least one frontier model, the compute cost of which exceeds five million dollars, and has spent over one hundred million dollars in compute costs in aggregate in training frontier models...
...Any person who is not a large developer, but who sets out to train a frontier model that if completed as planned would qualify such person as a large developer (I.E. At the end of the training, such person will have spent five million dollars in compute costs on one frontier model and one hundred million dollars in compute costs in aggregate in training frontier models, excluding accredited colleges and universities to the extent such colleges and universities are engaging in academic research) shall, before training such model:
Non-large developers setting out to train a frontier model as described above also have to fill out an SSP but don’t have follow paragraphs C or D of the SSP definition (making a detailed test procedure) which wasn’t really part of your question but now we know.
ETA: I just now see Expertium’s comment where there’s a more recent version of the bill, mostly making this comment superfluous.
Imagine you’re an algebra student and your teacher pretends not to know algebra. Despite the fact that the teacher does know it themselves, you as a student will not learn.
This is very cool and valuable work but I was also distracted by how funny I found this example.
I wouldn’t consider it a common phrase, but I also wouldn’t be surprised at all to hear someone say it given a sensible context.
One flaw in the setup is that the person opposing you could generate a random sequence beforehand and simply follow that when choosing options in the “game.” I assume the offer to play the game is not still available and/or you would not knowingly choose to play it against someone using this strategy, but if you would I’ll take the $25.
I don’t think this analogy works on multiple levels. As far as I know, there isn’t some sort of known probability that scaling laws will continue to be followed as new models are released. While it is true that a new model continuing to follow scaling laws is increased evidence in favor of future models continuing to follow scaling laws, thus shortening timelines, it’s not really clear how much evidence it would be.
This is important because, unlike a coin flip, there are a lot of other details regarding a new model release that could plausibly affect someone’s timelines. A model’s capabilities are complex, human reactions to them likely more so, and that isn’t covered in a yes/no description of if it’s better than the previous one or follows scaling laws.
Also, following your analogy would differ from the original comment since it moves to whether the new AI model follows scaling laws instead of just whether the new AI model is better than the previous one (It seems to me that there could be a model that is better than the previous one yet still markedly underperforms compared to what would be expected from scaling laws).
If there’s any obvious mistakes I’m making here I’d love to know, I’m still pretty new to the space.
Can you expand on this? I’m not sure what you mean but am curious about it.
Two years later, you could get this with Veo 2:
The picture(s) here seem to be missing.
My strong downvotes are giving +1, which is a little confusing.
I’m not sure the example you provide is actually an example of appealing to consequences. To me it seems more like looking at an action at different levels of abstraction rather than starting to think about consequences, although I do think the divide can be unclear. I don’t do philosophy so my ideas of how things may be defined are certainly messy and may not map on well to their technical usage, but I’m reminded of the bit in the sequences regarding Reductionism, where you certainly can have an accurate model of a plane dealing just with particle fields and fundamental forces, but that doesn’t mean the plane doesn’t have wings.
This is to say I think that you can have different moral judgements about an action at different levels of abstraction, but that’s something that can happen before and/or separately from thinking about the consequences of that action.
Are you expecting the Cost/Productivity ratio of AGI in the future to be roughly the same as it is for the agents Sam is currently proposing? I would expect that as time passes, the capabilities of such agents will vastly increase while they also get cheaper. This seems to generally be the case with technology, and previous technology had no potential means of self-improving on a short timescale. The potential floor for AI “wages” is also incredibly low compared to humans.
It definitely is worth also keeping in mind that AI labor should be much easier to scale than human labor in part because of the hiring issue, but a relatively high(?) price on initial agents isn’t enough to update me away from the massive potential it has to undercut labor.
Hello, everyone! I don’t remember how I first found the site a few months ago, but I came back after reading some of David Chapman’s stuff (Meaningness, Eggplant, etc.) I read through HPMOR and the sequences and thought they were pretty awesome, and I also really like the general vibes here in favor of good epistemics and discussion.
Some things about me:
I graduated last spring with a B.S. in Poli Sci, I was planning to apply to law school but have since decided against it for the foreseeable future, I currently work at a local law firm in my area.
I play violin and piano semi-professionally, doing gigs around town.
I’ve been saving over 85% of my income since last September when I actually started tracking it.
Relatedly, I want to keep my spending very low. It seems really helpful at allowing me to have control of my time and my actions.
I want to learn a lot of things and get better at doing a lot of things. I regret that I didn’t have this mindset when I was still in college, at that point I was mainly seeking good grades for minimum effort.
I don’t have too much in terms of strong life direction. My interests are varied and I can’t really imagine picking any one thing to focus on above all else for more than a limited time.
Last time I took an OCEAN test I was in the bottom 1% for neuroticism so that may help explain some things.
Current focuses: Music, Japanese, Exercise, learning about AI (have updated massively towards a relatively shorter timeline from basically no timeline at all since first visiting the site), and Cooking.
I definitely want to get some better structure to actually track progress in different things.
I also want to get caught up on some math and physics to better intuitively understand some of the stuff I’ve been reading here, I used to be pretty good (went to National Mathcounts in middle school) but haven’t done anything past like Calc 1 and I don’t remember that very well.
I’m in the US, but do not live really close to any of the meet-up places that I have seen.What I want to do here: I would like to use all the cool stuff you guys have written/made to learn and get better at doing things, I would like to make positive connections and friendships with smart and interesting people, and I would like to hopefully create some content that people here will find useful at some point. See you around!
Very funny that the cutoff for Heaven seems to be exactly the amount of karma you had before posting this comment.