Recently, Yudkowsky has been recently talking about third countries stealing
Should probably be
Recently, Yudkowsky has been talking about third countries stealing
Recently, Yudkowsky has been recently talking about third countries stealing
Should probably be
Recently, Yudkowsky has been talking about third countries stealing
A few typos.
How’s intuitively going to be better --> Who’s intuitively going to be better
igure out” that they need to to help and how. --> igure out” that they need to help and how.
very single job as though I’d be a life-and-death decision whether or not to apply --> very single job as though it’d be a life-and-death decision whether or not to apply
I found this post interesting but I think there is something wrong with it, even though I estimate that its central point has value. My remarks focus on “this post as an advice” rather than the “phenomenon explanation” part of the post.
Perhaps the first thing I should say is that I agree some people have a tendency to think too much in certain situations. To delay the first try for too long and to waste a lot of time optimizing entire lines of though that will be revealed to be worthless after five minutes of concrete work. I am one of those people and I have spent time thinking (eh) about when and how this must be improved and corrected.
But I do not think a general cheer for the policy of “acting before thinking” is a good thing. I can cite two examples from my personal life, in the last 10 days, where a dozen hours or more were wasted because someone did not spend half an hour thinking when it was “obviously” sensible to do so. The first one involves writing code without thinking, creating bad code that had to be managed afterward (technical debt), the second involves personal relationships and is private (sorry).
I estimate that this post presents danger on the same scale as the gain it might deliver. In fact I expect this post to be a net negative in terms of direct advice. Ideally, the wisdom found in this post would have its place in a more complete framework on decision processes. As it stands, I am tempted to compare this advice to “to walk, move your right leg forward”, dangerously misleading and incomplete without the complementary advice “also move your left leg”.
Perhaps a better immediate advice would be.
When applicable and when your actions will not have non-trivial lasting consequences think just a little then act, make an attempt. Only after your have tried enough can you think again, if you believe it to be useful.
If we apply this advice to different timescales we get both your second and third example. We however do not get the first, insofar as there can be lasting consequences to a botched application.
at first she had qualms literally called “Effective Evil”
I think this sentence is missing “working for an organization”.
I don’t think this is a good illustration of point 6. The video shows a string of manipulative leading questions, falling short of the ” in a way that they would endorse” criteria.
When people understand that a string of questions is designed to strong arm them into a given position they rarely endorse it. It seems to me that point 6 is more about benevolent and honest uses of leading questions.
Admittedly, I am making the assumption that ” in a way that they would endorse” means “such that if people understood the intent that went into writing the string of questions in that way they would approve of the process”.
Thank you for the post. A few typos:
To understand the current conflicts, it’s vital what the Russian discourse means when it talks about Nazis --> Probably “it’s vital to understand”.
One argument made, about why Russia’s claims of far-right influence in Ukraine are overblown, is that far-right parties don’t have much influence is that they have relatively poor electoral results --> Two different versions of the same end of sentence.
Holocaust to be national heroes who are shall not be criticized feels deeply wrong. --> just “who shall not be”.
[MENTOR]
I am a 25-year-old computer science PhD student and I currently work on natural language processing with neurosymbolic approaches (using a mix of logic and neural networks to do reasoning on a normal text input). I have a (very) high degree of education in math and theoretical computer science. I also have some very limited skills and knowledge for more practical aspects of computer science. Also, I have quite a lot of reflection on the topic of the nature of thought, valid inference, and rational behavior. These are condensed in a currently unpublished book length report. The result of a year-long break I took for personal reflection. I expect to be able to help quite a lot with fundamental reflection that relates to epistemology or rationality.
If you intend to learn about some aspect of mathematics or theoretical computer science I can probably point you toward resources or help you understand technical aspects. I am also willing to serve as a background help for someone going through a math education. Alternatively, I might be willing to help an in depth reflection to which you think I can be relevant.
I can offer some asynchronous email/messages exchange and semiregular conversation, perhaps averaging one or two per month. Ideally, we would schedule conversations quickly when you need them. If you have a very different mode of interaction in mind I am willing to adapt.
[APPRENTICE]
Presentation copy/pasted from the mentor section.
I am a 25-year-old computer science PhD student and I currently work on natural language processing with neurosymbolic approaches (using a mix of logic and neural networks to do reasoning on a normal text input).
I have a (very) high degree of education in math and theoretical computer science. I also have some very limited skills and knowledge for more practical aspects of computer science.
Also, I have quite a lot of reflection on the topic of the nature of thought, valid inference, and rational behavior. These are condensed in a currently unpublished book length report. The result of a year-long break I took for personal reflection.
I am looking for one of the followings:
Resources and help on the topic of the nature of thought and characterization of valid inference. This is a long standing topic for me. A string of reflections and readings that has been and continues to be an important side project.
Like 1 but with a focus on AI or in general automation of thought.
Help better managing my work and motivation. I need to reduce my work related stress. The help of someone who does a high amount of intellectual work without stress could be precious.
By default, I would imagine a few discussions near the beginning and then much less frequent conversations as needed (once a month?). But that’s just off the top of my head.
[NORMAL]
Whether as a mentor, as an apprentice, or because you think we could work together in some other capacity, feel free to DM me through lesswrong. Give me a few days to get back to you. If I do not answer it probably means your message did not go through. In that case just comment below.
I get the idea but I am not sure how to move to a richer domain. The only obvious idea I see is to go to continous time, but that not the usual paradigm for games.
We could go the opposite direction and try to get a result for a more restrictive class of games. I listed some in the post; the only case I thought of for which I do not know if the result holds is bounded games.
Alternatively, it is also possible to take another hypothesis than the strategy not being dominated. The result has shape “if a strategy is then it is a utility maximisation”. Maybe we can find some better .
Is there some other way to change the conjecture that I missed?
I see. If we were to make this formal it would depend on the notion of “complexity” we use.
Notably it seems intuitive that there be counterexample games that pump complexity out of thin air by adding rules and restriction that do not really matter. So “just” adding a simple complexity threshold would certainly not work, for most notions of complexity.
Maybe it is true that “the higher the complexity the larger the portion of nondominated strategies that are utility maximisation”. But
The set of strategies is often infinite, so the notion of “portion” depends on a measure function.
That kind of result is already much weaker than the “coherence result” I have come to expect by reading various sources.
Interesting idea anyway, seems to require quite a bit more work.
I have the suspicion that you read “more complexity” as meaning “more restrictions”, while I meant the contrary (I do realize I didn’t express myself clearly). Is that the case?
My intuition for the idea of complexity is something like “the minimal number of character it takes to implement this game in python”. The flaw is that this assume computable games, which is not in line with the general formulation of the conjecture I used. So that definition does not worK. But that’s roughly what I think of when I read “complexity”. Is that compatible with your meaning?
Note that this is for the notion of complexity of a given game. If you mean the complexity of a class of games then I am less certain how to define it. However if we are to change the category of games we are talking about then the only clear ways to do so I see involve weakening the conjecture by restricting it to speak of strictly fewer games.
In EJT’s post, I remember that the main concrete point was that being stubborn, in this sense:
if I previously turned down some option X, I will not choose any option that I strictly disprefer to X
To my understanding that was a good counter to the idea that anything that is not a utility maximisation is vulnerable to money pumps in a specific kind of games. But that is restricted to “decision tree” games in which in every turn but the first you have an “active outcome” which you know you can keep until the end if you wish. Every turn you can decide to change that active outcome or to keep it. These games are interesting to discuss dutch book vulnerability but they are still quite specific. Most games are not like that.
On a related note:
a non-dominated strategy for a preference tree compact enough compared to the world it applies to will be approximately a utility maximizer
I think I didn’t understand what you mean by “preference tree” here. Is it just a partial order relation (preference) on outcomes? If you mean “for a case in which the complexity of the preference ordering is small compared to that of the rest of the game” , then I disagree. The counterexample could certainly scale to high complexity of the rules without any change to the (very simple) preference ordering.
The closest I could come to your statement in my vocabulary above is:
For some value , if the ratio “complexity of the outcome preference” / “complexity of the total game” is inferior to then any nondominated strategy is (approximately) a utility maximisation.
Is this faithful enough?
Weird. I didn’t expect this to be wrong and I did not expect the other one to be right. Glad I asked.
“Minimal number of character it takes to implement this game in python” would be small because the “game code” part is the laws and the reward.
Not so sure about that. The game has to describe and model “everything” about the situation. So if you want to describe interaction with details of a “real world” then you also need a complete simulation of said real world. While everything is “contained” in the reward function, it is not like the reward function can be computed independently of “what happens” in the game. It is however true that you only need to compute the most minimal version of the game relevant to the outcome. So if your game contains a lot of “pointless” rules that do nothing then they can be safely ignored when computing the complexity of the game. I think that’s normal.
In the case of the real world, even restricting it to a bounded precision, the program would need to be very long. It is not just a description of the sentence “you win if there are a lot of diamonds in the world” (or whatever the goal is). It is also a complete “simulation” of the world.
Btw, the notion I was alluding to is Kolmogorov complexity.
and it would be difficult to write a non-dominated strategy which tries to be non-dominated just by not accruing that much energy overall and instead moving a lot of energy at some time step. Yet it’s probably possible [...]
Depending on the exact parameter, an intuitive strategy that is not dominated but not optimal long terms either could be “never invest anything”, in which you value the present so much that you never move because that would cost energy. Remark that this strategy is still “a” utility maximisation (just value each step more than the next to a high amount). But it is very bad for the utility you described.
But this kind of trick becomes more difficult if I restrict the number of branches you can make in the preferences tree; it’s possible to be non-dominated “just” because you have many non-comparable branches and it suffices to do OK on just one branch. As I restrict the number of branches, you’ll have to do better overall.
I still get the feeling that your notion of preference tree is not equivalent to my own concept of a partial order on the set of outcomes. Could you clarify?
I know this comment is a few months old, but an answer might still be helpful.
I think the following quote from the article answers your question.