There is no rollback when open-weight models are almost SOTA. Could we convince people like Zuck that open weights are too risky? I seriously doubt it.
martinkunev
Karma: 179
There are some differences between what women consider attractive female traits and what men actually find attractive. The example I like to give is how some women enlarge their lips even though men (usually) don’t find that attractive.
I think these are the relevant points:
Maybe women’s sense of each other’s beauty is more discriminating than men’s.
Some women probably don’t distinguish a sex symbol’s actual physical attractiveness from the other characteristics that would make her rarely and especially appealing to women (like fame, wealth, etc.), and they’re neglecting to account for these factors being less salient to men.
This:
Might agent rewrite agent ‘s brain to make agent better satisfy agent ’s utility function? Most forms of wire-heading inherently limit the ability of agents to affect the future
and this
We have not proved that agent does not try to affect agent ‘s utility function (in fact, I expect in many cases agent does try to influence agent ’s utility function).
appear to be in conflict. Are you trying to say that depending on the circumstances b may try to influence a’s utility function or avoid doing so?
How important is it to keep the system prompt short? I guess this would depend on the model, but does anybody have useful tips on that?
well-established
The usage I’m objecting to started, as far as I can tell, about 2 years ago with Llama 2. The term “open weights”, which is often used interchangably, is a much better fit.
At some point the open/closed distinction becomes insufficient as a description. You could very well have an open-source wrapper (or fine-tuning) of something which is closed-source. Just try to not mislead people about what you’re offering.
If I vibe-coded an app prompting, say, Claude, and released it along with the generated code, would you have the same objections to me calling it “open source,”
No, because I don’t think this misleads people. Granted, the term “open source” is fuzzy at the boundaries. Should we use the term? I don’t know, but if we do, it only makes sense if it means something different from “closed source”.
wrong in suggesting they prefer to work with the model by editing the training data and “recompiling” instead of starting with the weights
One doesn’t exclude the other. If you’re creating v2 of your model, you’d likely: take the training code and data for v1; make some changes / add new things; run the new training code on the new data. For minor changes you may prefer to do fine-tuning on the weights.
wildly more expensive
Suppose I write a program and let people download the binary. Can I say “I spent 100k on AWS to compile it, therefore the binary is open source”?
not even modification
Would you say compiling source code from scratch (e.g. for a different platform) is not a modification?
Even if you’re not intending to retrain the model from scratch, simply knowing what the training data is is valuable. Maybe you don’t care about the training data, but somebody else does. I don’t think “I could never possibly make use of the source code / training data” is an argument that a binary / weights is actually open source.
How does open source differ from closed source for you in the case of generative models? If they are the same, why use the term at all?
Open weights != Open source
There is the possibility of misgendering somebody and them taking it seriously. Sometimes it feels like you’re walking in a minefield. It’s not conducive to a good social interaction.
too few pronouns, and communication becomes vague and cumbersome
I’m wondering why languages like finnish can do just fine with “hän” while english needs he/she.
French to English you always translate as “you”. You probably mean translating from English to French where you need to make a judgement whether to use “vous” or “tu”.
Subjective experience is most likely physical
I think of preferences as a description of agent behavior, which means the preferences changed.
When you say “got better at achieving it’s preference” I suppose you’re thinking of preference as some goal the agent is pursuing. I find this view (assuming goal directedness) less general in its ability to describe agent behavior. It may be more useful, but if so I think we need to justify it better. I don’t exclude the possibility that there is a piece of information I don’t know about.
Goal-directedness leads toward instrumental convergence and away from corrigibility. If we are looking to solve corrigibility, I think it’s worth it to question goal-directedness.
vim
closing stuff—window/application list menu which opens with alt+tab / command+tab; dropdown menus; popup messages in sites
When was the last time you (intentionally) used your caps lock key?
yesterday
I may be the only one :)
I’d rather remap my right shift, which keyboard makers for some reason tend to make huge.
You touch on the point that people can mean several different things when talking about preferences. I think this causes most of the confusion and subsequent debates.
I wrote about this from a completely different perspective recently.
To add to the discussion, my impression is that many people in the US believe they have some moral superiority or know what is good for other people. The whole “we need a manhattan project for AI” discourse is reminiscent of calling for global domination. Also, doing things for the public good is controversial in the US as it can infringe on individual freedom.
This makes me really uncertain as to which AGI would be better (assuming somebody controls it).
Western AI is much more likely to be democratic
This sounds like “western AI is better because it is much more likely to have western values”
I don’t understand what you mean by “humanity’s values”. Also, one could maybe argue that “democratic” societies are those where actions are taken based on whether the majority of people can be manipulated to support them.
I find “indifference” poorly defined in this context, which makes me doubt totality and transitivity. I’m trying to clarify my own confusion on this.
My fear with the teleporter has always been the engineering details—can it get a consistent snapshot of me? what about the last moments after the snapshot and before the old copy is destroyed? can it reliably reconstruct me? what happens in case of a failure?
Assuming many worlds quantum mechanics, we should have similar anticipations for forking into two and for tossing a quantum coin.