Capabilities cannot be faked, alignment can.
martinkunev
Karma: 249
To the list of three questions asked in the beginning, I would add a fourth about anthropics. I think anthropic decision theory is quite poorly understood and yet quite important.
I agree, I’d challenge “slightly better than the best publicly available”. The best publicly available models are quite expensive to run so a ton of instances of those would only make sense if they make more money than they spend. I’m not an economist but it’s not obvious to me that when the internet floods with such agents, they’d be able to make enough money.
I’m surprised by your circle of friends. Many people around me are impressed. However very few think software development jobs are going to disappear.
I’m curious how much effort does this require for you.
How much time do you spend producing a minute of content? How long does a single video tend to be?
let the model know that good bug reports will be rewarded more highly than the bug-exploiting slop they wrote in the first place
The reward is a value we use in our RL algorithm to calculate the gradient update. In other words, Reward is not the optimization target. The model doesn’t naturally prefer gradient updates in directions where reward is high. You would need to use some meta-reward to train it to do that and this seems to move reward hacking one level above.
Either that or I’m misunderstanding something.
“does this really matter that much, outside of major company NDAs or state-secrets?”
Keeping secrets could be important in social interactions. Personal stories tend to be what allows people to connect and sometimes people would prefer those stories to not become common knowledge for a number of reasons (e.g. not be judged). People are comfortable sharing more when they know there will be no downstream consequences. I think keeping things confidential can enhance social interactions. I for one enjoy when I’m being trusted with a secret.
Below are just some thoughts I may turn into an article at some point.If not explicitly negotiated, I’ve noticed different people tend to have different thresholds for what they treat as secret. There is also an attitude of outing a secret to show how well-informed you are (I’ve been guilty of this in my teens). Besides that, some contexts require a lot of skill that not everybody has. Suppose a question such as “have you cheated?” is being asked around the table and suppose your answer is “no” and a friend has previously confided that their answer is “yes”. In this case, the Glomar response may be a better answer for you than “no”.
When sharing your own personal secret, you may inevitably reveal information about other people. There is some balance to be struck—revealing things concerning you vs preserving the privacy of others involved (e.g. talking about one’s sex life reveals things about their partners’). It is generally useful to understand what the expectations of the other people involved would be in that context. My best rule of thumb so far is to not say anything that could be used against the other people involved, but this does not cover all cases.
anecdote: Me, A and B were in a hot seat session (together with other people). Later, A would talk with B and mention stuff B said in the hot seat session in front of outsiders. I explained to A that things shared during hot seat stay private by default. In retrospect, it was a mistake that nobody had mentioned the privacy expectation during the hot seat session.
I usually decide on a policy before going to sleep (I know what I need to do and approximately how much sleep I require). A policy could be something like “wake up around 10am, but if still too tired, sleep for 90 more minutes”.
I think I’ve been doing a version of this for most of my life, leaning a little toward world building (but the main point of the world is to be a place where I’d be happy to live, not to tell a story).
Over the years I’ve started to avoid talking about that because it weirds most people out or they see it as frivolous. When there are other people in my “vision”, I try to think of them as p-zombies (if that concept even makes sense).
You can question a tradition without getting rid of it.
To be more concrete, what I have in mind are things like religious traditions (e.g. “light a candle in the church”) or superstitions (e.g. “don’t sing at the table because you’ll marry a gypsy”) or even culinary traditions that differ between cultures. Occasionally people in my family have insisted on such things after I explicitly question them and despite evidence disproving them.
It doesn’t follow from the text, it’s just based on my anecdotal experience. I could very well be wrong.
For the no sex before marriage, I’m not convinced the effects you describe are downstream of dismantling that tradition. For divorse, why not downstream of more independence and decreased stigma? For low childbirth, why not downstream of contraception and changed economic conditions? I also disagree that low childbirth rate is a problem, but that’s a separate question.
Anyway, I get your general point.
If killing a single person could have a low chance of improving the rain, well you need to do it or test it.
It’s easy to come up with tests. Given a random idea without epistemic backing, the rule of thumb should be to not waste time and resources with it.
young women selling their bodies… We can still ask if it’s an experiment worth advocating for, as a society, but is it?
I sense some cognitive dissonance. I think the risk from sex for money is much lower than from sacrificing people.
I’m also interested in trying such a discussion group
This may be just me (I’ve barely interacted with you), but you create the impression of being intentional in your actions and busy. When I saw you carrying the ball around, I found it funny, but assumed you have a deliberate goal in mind, rather than simply looking for people to be playful (I wasn’t at the area near the roof). That being said, improvisation is a skill a lot of people at lessonline (me included) could improve at.
balls
Studies show that humans have shape bias compared to some other animals (e.g. dogs). Dogs tend to categorize more based on size and texture. I’m not sure what this tells us. Maybe it just points to useful concepts being goal-dependent?
It may be interesting to compare languages and look for concepts only present in some. The examples I could find seem to be from categories other than nouns (e.g. left/right, specific colors, numbers, time).
In this example the program generating the data from both S1 and S2 is the more general abstraction, but can we really think of it as natural?
Maybe we have M3, M4 and so forth and if we try use the corresponding strings S1, … Sn to produce the data, we get something longer (each Si is small but there are many of them). In that case, there will be no single abstraction which is the most “natural”. I’m handwaving a lot, but does this make sense?
Not exactly what you’re asking for, but maybe in the right direction?
Things like >>= != =~ ++ ← /* <% are arguably new symbols, typically built on top of ASCII but only for convenience
I think the hard part is actually finding the right books / resources when you don’t have the skill (obviously) you’re trying to learn.
If the tokenizer treats the sequence as a single token, I don’t think it matters that much what it starts with.
However, if one could obtain some text known to cause halting, a simple binary search can identify the magic sequence.