Jan_Kulveit

Karma: 3,531

My current research interests:
- alignment in systems which are complex and messy, composed of both humans and AIs?
- actually good mathematized theories of cooperation and coordination
- active inference
- bounded rationality

Research at Alignment of Complex Systems Research Group (acsresearch.org), Centre for Theoretical Studies, Charles University in Prague. Formerly research fellow Future of Humanity Institute, Oxford University

Previously I was a researcher in physics, studying phase transitions, network science and complex systems.

Jan_Kulveit 8 Jan 2018 18:50 UTC
1 point
in reply to: spiralingintocontrol’s comment on: Nonlinear perception of happiness
Ad Kahneman: Yes. This is related, but my impression is the nonlinearity is somewhat more general—in DRM you are still asking people for rating on 1-6 affective scale (the nonlinearity would appear between the “raw affect” and the rating on the scale), and doing aggregates.
Happiness is Not a Coherent Concept thanks. It seems to me to be somewhat orthogonal—the article argues happiness breaks into several different variables, which are correlated, but not identical. Ok, than you can choose one of them. Or, you can try to uderstand the varriables better, and possibly construct something which encompasses all of them.

Jan_Kulveit 8 Jan 2018 20:31 UTC
2 points
in reply to: Dagon’s comment on: Nonlinear perception of happiness
Re: counter-theory - would you also argue there is no “raw” wealth? To me it seems the argument is broadly the same—there are many distinct inputs, some with hard-to-determine value, some with complex realtionships. (A fresh Harvard graduates beeing “poorer” than farmers in Nepal due to loans, etc.). Still, the aggregate concept is useful and in practice is often quantified.
I would argue you can guess at the shape by observing how some sort of “addition” operation on the inputs changes the output. (Like you can have fairly complex function F with many inputs, which is then perceived through some non-linear lens, like P(F()) … you can still guess at P() from partial derivative of P(F()), if you assume e.g. F() is nothing worse than some sort of polynomial )
Re: suspicion. This sort of aggregation goes on in many places when making decisions. Apparently the sort of problem this propsal hints at is usually not reflected at all, and the aggregation goes on by simply averaging the perceptions.

Jan_Kulveit 8 Jan 2018 20:49 UTC
2 points
in reply to: Charlie Steiner’s comment on: Nonlinear perception of happiness
Obviously there is a lot of complexity. In _this model_:
the complexity lies mainly in some unknown function Hapiness(), which maps from mind-states (or a fair chunk of the human brain) to real numbers. Apparently, humans have the ability to evaluate some sort of this estimate. The proposal here is they make some sort of really non-linear mapping, e.g. log(), when asked to map this to scale 1.. 10.
Prescriptive morality starts elsewhere: when you take such number, aggregate it somehow over some number of people, and claim it is worth optimizing.
What I’m saying, anyone making such prescriptive claims, should consider the possibility they are aggregating in a bad way. (Anyone optimizing e.g. QUALYs or Gross domestic hapiness or some conceptions of utilitarian value is making such claims)
By revealed preference people often put exponentially more resources to going 9 to 10 than from 1 to 2, so I don’t think the suggestion 1 to 2 is as valuable as 9 to 10 is intuitive at all.

Jan_Kulveit 8 Jan 2018 21:06 UTC
2 points
in reply to: Qiaochu_Yuan’s comment on: Nonlinear perception of happiness
The point of this is the relation between percieved happiness and some conjured “raw ” happiness. The implications to various ethical systems are just that, implications, and the inclusion of this was not meant as an endorsement. I dont want to argue for utilitarianism here, but I hope we agree some forms of utilitarianism are obviously relevant, and used in practice.
I’m a bit confused by “identifying any function of happiness with utility seems clearly wrong to me” : Do you propose the actual utility function as you understand it, has no relation to happiness at all?

Jan_Kulveit 9 Jan 2018 0:44 UTC
2 points
in reply to: dxu’s comment on: Nonlinear perception of happiness
Thanks. I guess I should not have included the simple utilitarian calculation, as it seems to work as a red herring :( Mea culpa.
Qiaochu: Would the article make better sense if framed like this: …assuming as per standard LessWrong reasoning, the actual utility function is very complicated, but also assuming, it has a large happiness component, whatever hapiness means, we may ask: what would be a relation of such component to usual approaches to measure happiness by asking people? And how to aggregate among people?

Jan_Kulveit 25 Jan 2018 14:48 UTC
4 points
on: Hammers and Nails
Ok… I’m not sure if it can be counted as “instrumental technique”, but I often think in terms of Kahneman’s System 1/ & System 2/.
The exercise … was hard, so I included also ideas which are common but can be “rediscovered” this way
1. Predicting what should be doable with machine learning techniques now or really soon. My model is “System 1” usually means some dedicated neural “hardware” for a task, and it seems narrow AIs are now approximately at the same level of power. The prediction is you can probably train NNs to recognize e.g.: the emotional state of people; sexual orientation of people; aggressiveness; “moods” of places.
2. Trying to apply my “system 1” instead of training neural network model, or applying some complex statistical computation. E.g. I tried to predict results of ongoing Czech presidential elections this way: take the number of votes for 6 leading candidates in the previous elections. Ask my system 1 how the candidates are similar to current candidates (assuming we have a relatively good hardware for simulating other people). Calculate how the votes would redistribute based on the similarity “guess”
3. Estimating what people’s intuitive moral feelings would be and where they are inconsistent with some more formal “system 2” model based on the guess such feelings are “system 1″ heuristic computation.
4. Trying to improve one’s “system 1” processing speed in useful domains, e.g. reading text faster.
5. Training system 1 to pattern match some class of situations and do something… well, that’s TAPs :)
6. Training system 1 to do some routine simple but useful calculation
7. Trying to extract some “lower level” data from System 1 computations than what is usually presented. E.g. people seem to have some “sense of status”, but they usually apply it indirectly.
8. Applying ideas from “adversarial machine learning” to think about inputs able to “hack” some system 1 processing common among people, and observing whether its actually done by someone
9. Training system 1 to get simple numerical estimates of body parameters such as temperature or heart rate
10. Using system 2 to convert problems to isomorphic problems which are recognized & can be processed by system 1 hardware … well, has been done and works

Jan_Kulveit 4 Feb 2018 23:27 UTC
44 points
on: Arbital postmortem
I believe I can add something useful to this conversation.
Background: I noticed Wikipedia when it was a relatively new project. I was one of the first registered users of Czech Wikipedia (# 7), one fo the first builders of the Czech community, registered the Czech domain, and gradually worked in different functions suchas an administrator, member of the arbitration committee, checkuser, etc … I was also a founding member of the Czech Wikimedia Chapter.
Since then, I have been following various attempts to compete with Wikipedia (does anyone remember e.g. Google Knol?) And so far in all cases I expected the projects will fail.
Why? Because IMO essential factor behind the success of Wikipedia is the “community design” that is in the background, and which provides incentives to content creators. Common lore it something likeeveryone can edit Wikipedia, and the encyclopedia is written by mostly random crowd of site visitors who click the edit button. That’s complete nonsense. Wikipedia was written mostly by a dedicated community of volunteers with complex internal organization, hierarchies, social rewards, and the possibility of gaining power.
Much of what I did at the begining cz.wiki was translations of various documents about community norms, implementation of different processes, feedbacks, dispute resolution, etc. The community design goals are various, e.g. you want positive feedback loops, you dont want editors to bite each other, you want the community to be meritocratic, and so on. It’s far from obvious how to design such system.
Of course, with most wikipedists, somewhere in the background is an altruistic motivation to help with the aggregation of human knowledge and create something like Encyclopedia Galactica. But on a day-to-day basis, what helps keepeing peeple motivated is working with other dedicated people, receiving feedback, beeing able to see others interacting with your edits and improving further, and even some forms of conflicts . Also valuable editing leads to increasing your weight in the community, you can gain various social goods, responsibility, various functions, and of course power.
Btw the power of senior Wikipedia users is not negligible at all. If we simplify it to counting money, you can imagine what is the value at stake depending on the content of the first page of Google results on many topics, and how much power Google has controlling it. Wikipedia has less influence, but still a lot of influence.
So to summarize, what I see in this portmortem (thanks for wrting it!) from my perspective is almost complete lack of “community design” and thinking about editor motivations. (Community design is different from user aquisiton.)
What links here?
- Jan_Kulveit's comment on Duncan Sabien on Moderating LessWrong by Davis_Kingsley (29 May 2018 18:34 UTC; 5 points)

Jan_Kulveit 14 Feb 2018 23:56 UTC
2 points
on: The Principled Intelligence Hypothesis
I’m not sure if the starting point of the postt isn’t really just anecdotal evidence from self-evaluation, and stereotypes.
You have various stereotypes of the smartest persons—e.g. I believe Richard Feynman is also a stereotypical genius, but of the rule-breaking flavour
I tried to find some research on correlation between rule-breaking and IQ, and the first study I found has insignifficant but positive one

Jan_Kulveit 22 Feb 2018 0:00 UTC
8 points
on: Robustness to Scale
Nice explanation of a concept, thanks.
My intuition from complex systems studies is I wouldn’t be too content even with things which look robust—“scale invariant” properties of systems in most cases hold only across some orders of magnitude.

Jan_Kulveit 1 Mar 2018 23:44 UTC
15 points
on: Extended Quote on the Institution of Academia
1. I think some nuance can be useful. Contemporary science is quite good at pushing boundaries in existing research directions, somewhat less at creating new directions. (Once a field becomes larger than some critical size, it is ok.)
2. Academia (not science itself) in fact does a lot of synthesizing, distillation and aggregation work. Usually in the form of creating courses and writing textbooks. Some of the inadequacies are
- Usually this happens at some later stage of researcher’s careers, people are not trained to do it
- Knowledge up to something like “mainstream graduate lever course” is usually distilled and refactored to some user-friendly format somewhere. Beyond that, it gets much worse.
- New fields fare even worse than in science
- None of this is usually done with policy in mind
(Btw this model leads to an actionable suggestion—if you want more aggregation work done in a field by academia, cause it to be taught somewhere.)
I think one important thing which science and academic institutions do is provide spaces/jobs where people can have time to think about whatever they want, speak with smart people, and have open conversations. (From some anecdotal evidence I would guess this is more the case at some less prestigious places than at the top places where the daemons of competition rule over people without tenure with all their might.)

Jan_Kulveit 5 Mar 2018 9:41 UTC
3 points
on: God Help Us, Let’s Try To Understand Friston On Free Energy
Somewhat meta question: is it better to comment here, or on SSC? Both?

Jan_Kulveit 7 Mar 2018 16:59 UTC
3 points
on: Person-moment affecting views
I think person-affecting views are wrong, so I don’t see much motivation to “rescue” them from whatever contradiction or problem they run into.
Why do you want?
(I dont mean it in any negative way, just curious)

Jan_Kulveit 9 Mar 2018 10:02 UTC
4 points
on: Prize for probable problems
My intuition is this is not particularly stable against adversarial inputs. Trying to think about is as a practical problem, I would attack it in this way
- provide adversarial inputs to A0 which will try to manipulate them so they are better at the “simple task at hand” but e.g. have slightly distorted some model of the world
- it seems feasible to craft manipulations which would “evaluate” and steer the whole system only while already at superhuman level, so we have Amplify(H,A[10]) where A(10) is superhuman and optimizing to take over the control for the attackers goal
In vivid language, you seed the personal assistants with the right ideas … and many iterations later, they start a communist revolution

Jan_Kulveit 9 Mar 2018 22:13 UTC
2 points
in reply to: KatjaGrace’s comment on: Person-moment affecting views
I think this is quite devastating analysis for them, even if you would take a “person” to be well defined object.
The Person-Affecting Restriction, Comparativism, and the Moral Status of Potential People, by Gustaf Arrhenius
for example see fig. 2 and related argument. Basically, you have 3 worlds (A;B;C), with populations ([x,y];[y,z];[z,x]). You set the welfare of the populations such that
“Assume that all of these people have positive welfare, but that the y people are better off in B as compared to A, the z people are better off in C as compared to B, and the x people are better off in A as compared to C.
Since the x people do not exist in B, B is neither worse nor better than A for them. Similarly, since the z people do not exist in A, A is neither worse nor better than B for them. However, B is better than A for the y people. Consequently, B is better than A according to the second clause of the Person Affecting Restriction. The same reasoning yields that C is better than B, and A is better than C. But if B is better than A, and C is better than B, then transitivity yields that C is better than A. Consequently, C is both better and worse than A.”
So you would have to sacrifice transitivity to “rescue” PAW.
Another argument may be from physics—according to many-worlds interpretation of QM, there exists a world where I was not born, because some high energy particle damaged a part of DNA necessary for my birth. Hence, for each person there exists a world where he does not exists. Taken ad absurdum, nobody has moral value.

Jan_Kulveit 9 Mar 2018 22:41 UTC
6 points
on: Brains and backprop: a key timeline crux
I put most weight on hypothesis there are multiple secret sauces, so even if it would come out brains use some kind of backprop, I would not expect the rest to be “just engineering”. For example there is an open problem with long-term memory, which may require architectural changes, like freezing weight, adding neurons on the way,...
Btw you have likely wrong labels on scenarios
Generally good way of looking on things! Thanks

Jan_Kulveit 30 Mar 2018 8:18 UTC
13 points
on: Rationalist Lent is over
Worked great. I chose facebook. These were my observations

1. Well, confirming the obvious, the platform actually is optimized to eat as much time and attention as possible. I found it quite easy to not open fb at all, but when I needed to post something work-related several times, it was hard not to check if someone interacts with it, comments, etd.

2. I partly substituted tracking “general discussion” on my facebook with tracking the general discussion on LessWrong. While generally it seems a positive change, I lost some interesting links e.g. from AI Safety Discussion. Idea: if someone would make a periodic digest and post it here I would love it.

3. Writing fb statuses correlated with the fact that I wrote more useful texts.
A potential disadvantage is my fb still seems a better environment for idea incubation or constructive criticism than current LW or EA fora. What I’ve been doing is to write in google docs and ask friends for comments privately.
4. What is hard to replace are invitations to local events, however, I usually deal with the undersupply of evenings, not undersupply of events.

5. Overall, I find that I was more successful in the things I care about.
Big thanks for the lent posting.
Conclusion: I’ll mostly leave facebook for at least a few months, or until I really need something that fb offers due to changed circumstances. I’m not going to delete accounts or anything like that, just not follow the discussion there or post. It will likely cost me some “weak but important” social ties, but seems worth it.

Jan_Kulveit 1 Apr 2018 13:26 UTC
31 points
on: A Sketch of Good Communication
I like the approach but the problem this may run into on many occasions is—parts of the reasoning include things which are not easily communicable. Suppose you talk to AlphaGo and want it to explain why it prefers some move. It may relatively easily communicate part of tree search (an I can easily integrate the non-overlapping part of the tree into my tree), but we will run into trouble communicating the “policy network” consisting of weights obtained on training by very large number of examples. *
From human perspective, such parts of our cognitive infrastructure are often not really accessible, giving their results in the form of “intuition”. What’s worse, other part of our cognitive infrastructure is very good in making fake stories explaining the intuitions, so if you are motivated enough your mind may generate plausible but in fact fake models for your intuition. In worst case you discard the good intuition when someone shows you problems in the fake model.
Also… it seems to me it is in practice difficult to communicate such parts of your reasoning, and among rationalists in particular. I would expect communicating parts of the model in the form of “my intuition is …” to be intuitively frowned upon, down-voted, pattern-matched to argument from authority, or “argument” by someone who cannot think clearly. (An exception are intuitions of high-status people). In theory it would help if you describe why your intuitive black box is producing something else than random noise, but my intuition tells me in it may sound even more awkward (“Hey I studied this for several years” )
*Btw in some cases, it _may_ make sense to throw away part of you model and replace it just by the opinion of the expert. I’m not sure I can describe it clearly without drawing
What links here?

Jan_Kulveit 1 Apr 2018 21:24 UTC
2 points
in reply to: TurnTrout’s comment on: A Sketch of Good Communication
What I had in mind would, for example, in the case of AlphaGo mean throwing away part of my opinion on “how valuable a position is” and replacing it by AG opinion. That does not mean throwing away your whole model. If done in a right way, you can integrate opinions of experts and come with better overall picture than any of them, even if you miss most of the information.

Jan_Kulveit 1 Apr 2018 22:15 UTC
9 points
on: Global insect declines: Why aren’t we all dead yet?
Some speculation
The original article mentions few reasons what may be going on in Germany—industrial scale use of pesticides or the increase in farms dedicated to a single crop. The same factors plus better crops and fertilizers allowed for 10x increase in crop productivity (check this truly amazing graph https://www.agry.purdue.edu/ext/corn/news/timeless/images/CornYieldTrend_US.gif)
Industrial farming is progressively more decoupled from ecosystems and does not need rich ecosystems (imagine hydroponic indoor farms growing vegetables just from water, artificial nutrients, and under LED light)
Also my guess would be you are overestimating importance of insects (copared eg to bacteria)

Jan_Kulveit 15 Apr 2018 22:47 UTC
2 points
on: A voting theory primer for rationalists
Thanks for this introduction to voting systems! I always wanted to know more about them, but never actually got to study it more—so usually I just suggested approval voting as a reasonable default. (Hence we use it in CZEA)
Btw what do you think of this voting method? https://web.d21.me/en/ (pushed by somewhat eccentric mathematician-trader-billionaire Janecek)
(Btw I would suggest this post gets to curated)