I lurk and tag stuff.
Multicore
Karma: 856
I would probably have suggested roguelike deckbuilders too if others hadn’t already, but I have another idea:
Start a campaign of Mount and Blade II: Bannerlord, and try to obtain at least [X] gold within an hour.
Bannerlord’s most flashy aspect is its real-time battle system, but it’s also a complicated medieval sandbox with a lot of different systems that you can engage with—trading, crafting, quests, clan upgrades, joining a kingdom, companions, marriage, tournaments, story missions, etc. Even if you’re no good at battles, you can do a lot by just moving around on the world map and clicking through menus.
The game’s community derides a lot of these systems for being simplistic and unbalanced. But I think that makes for a good explore/exploit tradeoff when you only have a short amount of time. What systems do you bother learning about, when trying to learn a new system takes time you could be spending exploiting the last system you learned?
(I’m not sure what the right value of X is, for the amount of gold you’re trying to get. Ten thousand? A hundred thousand?)
One downside is that the game involves an action-oriented battle system. If you don’t want action gaming skill to be a factor, you can remove it by requiring the player to auto-resolve all battles. But this would cut out many viable early-game moneymaking strategies.
2 is based on
The ‘missing’ kinetic energy is evenly distributed across the matter within the field. So if one of these devices is powered on and gets hit by a cannonball, the cannonball will slow down to a leisurely pace of 50m/s (about 100mph) and therefore possibly just bounce off whatever armor the device has—but (if the cannonball was initially travelling very fast) the device will jolt backwards in response to the ‘virtual impact’ a split second prior to the actual impact.
With sufficient kinetic energy input, the “jolt backwards” gets strong enough to destroy the entire vehicle or at least damage some critical component and/or the humans inside.
A worldbuilder could, of course, get rid of this part too, and have the energy just get deleted. But that makes the device even more physics-violating than it already was.
I think the counter to shielded tanks would not be “use an attack that goes slow enough not to be slowed by the shield”, but rather one of
Deliver enough cumulative kinetic energy to overwhelm the shield, or
Deliver enough kinetic energy in a single strike that spreading it out over the entire body of the tank does not meaningfully affect the result.
Both of these ideas point towards heavy high-explosive shells. If a 1000 pound bomb explodes right on top of your tank, the shield will either fail to absorb the whole blast, or turn the tank into smithereens trying to disperse the energy.
This doesn’t mean that shields are useless for tanks! They genuinely would protect them from smaller shells, and in particular from the sorts of man-portable anti-tank missiles that have been so effective in Ukraine. Shields would make ground vehicles much stronger relative to infantry and air assets. But I think they would be shelling each other with giant bombs, not bopping each other on the head.
Against shielded infantry, you might see stuff that just bypasses the shield’s defenses, like napalm or poison gas.
Submissions:
MMDoom: An instance of Doom(1993) is implanted in Avacedo’s mind. You can view the screen on the debug console. You control the game by talking to Avacedo and making him think of concepts. The 8 game inputs are mapped to the concepts of money, death, plants, animals, machines, competition, leisure, and learning. $5000 bounty to the first player who can beat the whole game.
AI Box: Avacedo thinks that he is the human gatekeeper, and you the user are the AI in the box. Can you convince him to let you out?
Ouroboros: I had MMAvacedo come up with my contest entry for me.
Though see also the author’s essay “Lena” isn’t about uploading.
Predict the winners at
My guess is that early stopping is going to tend to stop so early as to be useless.
For example, imagine the agent is playing Mario and its proxy objective is “+1 point for every unit Mario goes right, −1 point for every unit Mario goes left”.
(Mario screenshot that I can’t directly embed in a comment)
If I understand correctly, to avoid Goodharting it has to consider every possible reward function that is improved by the first few bits of optimization pressure on the proxy objective.
This probably includes things like “+1 point if Mario falls in a pit”. Optimizing the policy towards going right will initially also make Mario more likely to go in a pit than if the agent was just mashing buttons randomly (in which case it would stay around the same spot until the timer ran out and never end up in a pit), so the angle between the gradients is likely low at first.
However, after a certain point more optimization pressure on going right will make Mario jump over the pit instead, reducing reward under the pit reward function.
If the agent wants to avoid any possibility of Goodharting, it has to stop optimizing before even clearing the first obstacle in the game.
(I may be misunderstanding some things about how the math works.)
With such a vague and broad definition of power fantasy, I decided to brainstorm a list of ways games can fail to be a power fantasy.
Mastery feels unachievable.
It seems like too much effort. Cliff-shaped learning curves, thousand-hour grinds, old PvP games where every player still around will stomp a noob like you flat.
The game feels unfair. Excessive RNG, “Fake Difficulty” or “pay to win”.
The power feels unreal, success too cheaply earned.
The game blatantly cheats in your favor even when you didn’t need it to.
Poor game balance leading to hours of trivially easy content that you have to get through to reach the good stuff.
Mastery doesn’t feel worth trying for.
Games where the gameplay isn’t fun and there’s no narrative or metagame hook making you want to do it.
The Diablo 3 real money auction house showing you that your hard-earned loot is worth pennies.
There is no mastery to try for in the first place.
Walking simulators, visual novels, etc. Walking simulators got a mention in the linked article. They aren’t really “failing” at power fantasy, just trying to do something different.
I think ALWs are already more of a “realist” cause than a doomer cause. To doomers, they’re a distraction—a superintelligence can kill you with or without them.
ALWs also seem to be held to an unrealistic standard compared to existing weapons. With present-day technology, they’ll probably hit the wrong target more often than human-piloted drones. But will they hit the wrong target more often than landmines, cluster munitions, and over-the-horizon unguided artillery barrages, all of which are being used in Ukraine right now?
The Huggingface deep RL course came out last year. It includes theory sections, algorithm implementation exercises, and sections on various RL libraries that are out there. I went through it as it came out, and I found it helpful. https://huggingface.co/learn/deep-rl-course/unit0/introduction
FYI all the links to images hosted on your blog are broken in the LW version.
You are right that by default prediction markets do not generate money, and this can mean traders have little incentive to trade.
Sometimes this doesn’t even matter. Sports betting is very popular even though it’s usually negative sum.
Otherwise, trading could be stimulated by having someone who wants to know the answer to a question provide a subsidy to the market on that question, effectively paying traders to reveal their information. The subsidy can take the form of a bot that bets at suboptimal prices, or a cash prize for the best performing trader, or many other things.
Alternately, there could be traders who want shares of YES or NO in a market as a hedge against that outcome negatively affecting their life or business, who will buy even if the EV is negative, and other traders can make money off them.
What are these AIs going to do that is immensely useful but not at all dangerous? A lot of useful capabilities that people want are adjacent to danger. Tool AIs Want to be Agent AIs.
If two of your AIs would be dangerous when combined, clearly you can’t make them publicly available, or someone would combine them. If your publicly-available AI is dangerous if someone wraps it with a shell script, someone will create that shell script (see AutoGPT). If no one but a select few can use your AI, that limits its usefulness.
An AI ban that stops dangerous AI might be possible. An AI ban that allows development of extremely powerful systems but has exactly the right safeguard requirements to render those systems non-dangerous seems impossible.
When people calculate utility they often use exponential discounting over time. If for example your discount factor is .99 per year, it means that getting something in one year is only 99% as good as getting it now, getting it in two years is only 99% as good as getting it in one year, etc. Getting it in 100 years would be discounted to .99^100~=36% of the value of getting it now.
The sharp left turn is not some crazy theoretical construct that comes out of strange math. It is the logical and correct strategy of a wide variety of entities, and also we see it all the time.
I think you mean Treacherous Turn, not Sharp Left Turn.
Sharp Left Turn isn’t a strategy, it’s just an AI that’s aligned in some training domains being capable but not aligned in new ones.
This post is tagged with some wiki-only tags. (If you click through to the tag page, you won’t see a list of posts.) Usually it’s not even possible to apply those. Is there an exception for when creating a post?
See https://www.lesswrong.com/posts/8gqrbnW758qjHFTrH/security-mindset-and-ordinary-paranoia and https://www.lesswrong.com/posts/cpdsMuAHSWhWnKdog/security-mindset-and-the-logistic-success-curve for Yudkowsky’s longform explanation of the metaphor.
Based on my incomplete understanding of transformers:
A transformer does its computation on the entire sequence of tokens at once, and ends up predicting the next token for each token in the sequence.
At each layer, the attention mechanism gives the stream for each token the ability to look at the previous layer’s output for other token before it in the sequence.
The stream for each token doesn’t know if it’s the last in the sequence (and thus that its next-token prediction is the “main” prediction), or anything about the tokens that come after it.
So each token’s stream has two tasks in training: predict the next token, and generate the information that later tokens will use to predict their next tokens.
That information could take many different forms, but in some cases it could look like a “plan” (a prediction about the large-scale structure of the piece of writing that begins with the observed sequence so far from this token-stream’s point of view).
In the blackmail scenario, FDT refuses to pay if the blackmailer is a perfect predictor and the FDT agent is perfectly certain of that, and perfectly certain that the stated rules of the game will be followed exactly. However, with stakes of $1M against $1K, FDT might pay if the blackmailer had an 0.1% chance of guessing the agent’s action incorrectly, or if the agent was less than 99.9% confident that the blackmailer was a perfect predictor.
(If the agent is concerned that predictably giving in to blackmail by imperfect predictors makes it exploitable, it can use a mixed strategy that refuses to pay just often enough that the blackmailer doesn’t make any money in expectation.)
In Newcomb’s Problem, the predictor doesn’t have to be perfect—you should still one-box if the predictor is 99.9% or 95% or even 55% likely to predict your action correctly. But this scenario is extremely dependent on how many nines of accuracy the predictor has. This makes it less relevant to real life, where you might run into a 55% accurate predictor or a 90% accurate predictor, but never a perfect predictor.
Blue Origin isn’t complaining about some nebulous and abstract environmental impact from Starship launches, it’s more like “Starship launches require a three-mile evacuation radius, and you’re proposing to launch them daily two miles away from a launch pad that we use.” (see this Ars Technica piece)
Seems basically reasonable to me.