Jan_Kulveit

• Jan_Kulveit 4 Aug 2023 14:01 UTC

  8 points

  2

  on: Yes, It’s Sub­jec­tive, But Why All The Crabs?

  I have a longer draft on this, but my current take is the high level answer to the question is similar for crabs and ontologies (&more).
  
  Convergent evolution usually happens because of similar selection pressures + some deeper contingencies.
  
  Looking at the selection pressures for ontologies and abstractions, there is a bunch of pressures which are fairly universal, an in various ways apply to humans, AIs, animals...
  
  For example: Negentropy is costly ⇒ flipping less bits and storing less bits is selected for; consequences include
  -part of concepts; clustering is compression
  -discretization/​quantization/​coarse grainings; all is compression
  …
  
  Intentional stance is to a decent extent ~compression algorithm assuming some systems can be decomposed into “goals” and “executor” (now the cat is chasing a mouse, now some other mouse). Yes this is again not the full explanation because it leads to a question why there are systems in the territory for which this works, but it is a step.

• Jan_Kulveit 18 Jul 2023 9:57 UTC

  8 points

  2

  on: Why was the AI Align­ment com­mu­nity so un­pre­pared for this mo­ment?

  My main answer is capacity constrains at central places. I think you are not considering how small the community was.
  
  One somewhat representative anecdote: sometime in ~2019, at FHI, there was a discussion that the “AI ethics” and “AI safety” research communities seem to be victims of unfortunate polarization dynamics, where even while in the Platonic realm of ideas concerns tracked by the people are compatible, there is somewhat unfortunate social dynamic, where loud voices on both sides are extremely dismissive of the other community. My guess at that time was the divide has decent chance of exploding when AI worries go mainstream (like, arguments about AI risk facing vociferous opposition from part of academia entrenched under the “ethics” flag), and my proposal was to do something about it, as there were some opportunities to pre-empt/​heal this, e.g. by supporting people from both camps to visit each others conferences, or writing papers explaining the concerns in a language of the other camp. Overall this was often specific and actionable. The only problem was … “who has time to work on this”, and the answer was “no one”.
  
  If you looked at what senior staff at FHI was working on, the counterfactuals were e.g. Toby Ord writing The Precipice. I think even with the benefit of hindsight, that was clearly more valuable—if today you see UN Security Council discussing AI risk and at least some people in the room have somewhat sane models, it’s also because a bunch of people at UN read The Precipice and started to think about xrisk and AI risk.
  
  If you looked at junior people, I was juggling already quite high number of balls, including research on active inference minds and implications for value learning, research on technical problems in comprehensive AI services, organizing academic-friendly Human-aligned AI summer school, organizing Epistea summer experiment, organizing ESPR, participating in a bunch of CFAR things. Even in retrospect, I think all of these bets were better than me trying to do something about the expected harmful AI ethics vs AI safety flamewar.
  
  Similarly, we had an early-stage effort on “robust communication”, attempting to design a system for testing robustly good public communication about xrisk and similar sensitive topics (including e.g. developing good shareable models of future problems fitting in the Overton window). It went nowhere because … there just weren’t any people. FHI had dozens of topic like that where a whole org should work on them, but the actual attention was about 0.2FTE of someone junior.
  
  Overall I think with the benefit of hindsight, a lot of what FHI worked on was more or less what you suggest should have been done. It’s true that this was never in the spotlight on LessWrong—I guess in 2019 the prevailing LW sentiment would be that Toby Ord engaging with UN is most likely useless waste of time.

• Jan_Kulveit 17 Jul 2023 9:59 UTC

  4 points

  0

  in reply to: Dan H’s comment on: Elon Musk an­nounces xAI

  What were the other options? Have you considered advising xAI privately, or re-directing xAI to be advised by someone else? Also, would the default be clearly worse?
  
  As you surely are quite aware of, one of the bigger fights about AI safety across academia, policymaking and public spaces now is the discussion about AI safety being “distraction” from immediate social harms, and being actually the agenda favoured by the leading labs and technologists. (Often comes with accusations of attempted regulatory capture, worries about concentration of power, etc.)
  
  In my view, given this situation, it seems valuable to have AI safety represented also by somewhat neutral coordination institutions without obvious conflicts of interest and large attack surfaces.
  
  As I wrote in the OP, CAIS made some relatively bold moves to became one of the most visible “public representatives” of AI safety—including the name choice, and organizing the widely reported Statement on AI risk (which was a success). Until now, my impression was when you are taking the namespace, you also aim for CAIS to be such “somewhat neutral coordination institution without obvious conflicts of interest and large attack surfaces”.
  
  Maybe I was wrong, and you don’t aim for this coordination/​representative role. But if you do, advising xAI seems a strange choice for multiple reasons:
  1. it makes you somewhat less neutral party for the broader world; even if the link to xAI does not actually influence your judgement or motivations, I think on priors it’s broadly sensible for policymakers, politicians and public to suspect all kind of activism, advocacy and lobbying efforts having some side-motivations or conflicts of interest, and this strengthens this suspicion
  2. the existing public announcements do not inspire confidence in the safety mindset in xAI founders; it seems unclear whether you advised xAI also about the plan “align to curiosity”
  3. if xAI turns to be mostly interested in safety-washing, it’s more of problem if it’s aided by more central/​representative org

• Jan_Kulveit 13 Jul 2023 10:01 UTC

  4 points

  4

  in reply to: Ulisse Mini’s comment on: New wind in ra­tio­nal­ity’s sails: Ap­pli­ca­tions for Epistea Res­i­dency 2023 are now open

  Broadly agree the failure mode is important; also I’m fairly confident basically all the listed mentors understand this problem of rationality education /​ “how to improve yourself” schools /​ etc. and I’d hope can help participants to avoid it.
  
  I would subtly push back against optimizing for something like being measurably stronger on a timescale like 2 months. In my experience actually functional things in this space typically work by increasing the growth rate of [something hard to measure], so instead of e.g. 15% p.a. you get 80% p.a.
  

Elon Musk an­nounces xAI

• Jan_Kulveit 11 Jul 2023 15:33 UTC

  5 points

  0

  on: The Seeker’s Game – Vignettes from the Bay

  Because his approach does not conform to established epistemic norms on LessWrong, Adrian feels pressure to cloak and obscure how he develops his ideas. One way in which this manifests is his two-step writing process. When Adrian works on LessWrong posts, he first develops ideas through his free-form approach. After that, he heavily edits the structure of the text, adding citations, rationalisations and legible arguments before posting it. If he doesn’t “translate” his writing, rationalists might simply dismiss what he has to say.
  

  
  cf Limits to legibility ; yes, strong norms/​incentives for “legibility” have this negative impact.

• Jan_Kulveit 4 Jul 2023 16:43 UTC

  4 points

  0

  in reply to: Richard_Ngo’s comment on: Frames in context

  I broadly agree with something like “we use a lot of explicit S2 algorithms built on top of the modelling machinery described”, so yes, what I mean more directly apply to the low level, than to humans explicitly thinking about what steps to take.
  
  I think practically useful epistemology for humans needs to deal with both “how is it implemented” and “what’s the content”. To use ML metaphor: human cognition is build out of both “trained neural nets” and “chain-of-thought type inferences in language” running on top of such nets. All S2 reasoning is a prediction in somewhat similar way as all GPT3 reasoning is a prediction—the NN predictor learns how to make “correct predictions” of language, but because the domain itself is partially symbolic world model, this maps to predictions about the world.
  
  In my view some parts of traditional epistemology are confused in trying to do epistemology for humans basically only at the level of the language reasoning, which is a bit like if you try to fix LLM cognition just by writing smart prompts, and ignore there is this huge underlying computation which does the heavy lifting.
  
  I’m certainly in favour of attempts to do epistemology for humans which are compatible with what the underlying computation actually does.
  
  I do agree you can go too far in the opposite direction, ignoring the symbolic reason … but seems rare when people think about humans?
  
  2. My personal take on dark room problem is it is in case of humans mostly fixed by “fixed priors” on interoceptive inputs. I.e. your body has evolutionary older machinery to compute hunger. This gets fed into the predictive processing machinery as input, and the evolutionary sensible belief (“not hungry”) gets fixed. (I don’t think calling this “priors” was good choice of terminology...).
  
  This setup at least in theory rewards both prediction and action, and avoids dark room problems for practical purposes: let’s assume I have this really strong belief (“fixed prior”) I won’t be hungry 1 hour in future. Conditional on that, I can compute what are my other sensory inputs half an hour from now. Predictive model of me eating a tasty food in half an hour is more coherent with me being not hungry than predictive model of me reading a book—but this does not need to be hardwired, but can be learned.
  
  Given that evolution has good reasons to “fix priors” on multiple evolutionary relevant inputs, I would not expect actual humans to seek dark rooms, but I would expect the PP system occasionally seeking a way how to block or modify the interoceptive signals
  
  3. My impression about how you use ‘frames’ is … the central examples are more like somewhat complex model ensembles including some symbolic/​language based components, rather than e.g. “there is gravity” frame or “model of apple” frame. My guess is this will likely be useful for practical use, but with attempts to formalize it, I think a better option is to start with the existing HGM maths.
  
  
  
  
  

• Jan_Kulveit 4 Jul 2023 8:57 UTC

  11 points

  0

  on: Frames in context

  So far it seems like you are broadly reinventing concepts which are natural and understood in predictive processing and active inference.
  
  Here is rough attempt at translation /​ pointer to what you are describing: what you call frames is usually called predictive models or hierarchical generative models in PP literature

  1. Unlike logical propositions, frames can’t be evaluated as discretely true or false.
     Sure: predictive models are evaluated based on prediction error, which is roughly a combination of ability to predict outputs of lower level layers, not deviating too much from predictions of higher order models, and being useful for modifying the world.

  2. Unlike Bayesian hypotheses, frames aren’t mutually exclusive, and can overlap with each other. This (along with point Frames in context
     Sure: predictive models overlap, and it is somewhat arbitrary where you would draw boundaries of individual models. E.g. you can draw a very broad boundary around a model call microeconomics, and a very broad boundary around a model called Buddhist philosophy, but both models likely share some parts modelling something like human desires

  3. Unlike in critical rationalism, we evaluate frames (partly) in terms of how true they are (based on their predictions) rather than just whether they’ve been falsified or not.
     Sure: actually science roughly is “cultural evolution rediscovered active inference”. Models are evaluated based on prediction error.

  4. Unlike Garrabrant traders and Rational Inductive Agents, frames can output any combination of empirical content (e.g. predictions about the world) and normative content (e.g. evaluations of outcomes, or recommendations for how to act).
     Sure: actually, the “any combination” goes even further. In active inference, there is no strict type difference between predictions about stuff like “what photons hit photoreceptors in your eyes” and stuff like “what should be a position of your muscles”. Recommendations how to act are just predictions about your actions conditional of wishful oriented beliefs about future states. Evaluations of outcomes are just prediction errors between wishful models and observations.

  5. Unlike model-based policies, policies composed of frames can’t be decomposed into modules with distinct functions, because each frame plays multiple roles.
     Mostly but this description seems a bit confused. “This has distinct function” is a label you slap on a computation using design stance, if the design stance description is much shorter than the alternatives (e.g. physical stance description). In case of hierarchical predictive models, you can imagine drawing various boundaries around various parts of the system (e.g., you can imagine alternatives of including or not including layers computing edge detection in a model tracking whether someone is happy, and in the other direction you can imagine including and not including layers with some abstract conceptions of hedonic utilitarianism vs. some transcendental purpose). Once you select a boundary, you can sometimes assign “distinct function” to it, sometimes more than one, sometimes “distinct goal”, etc. It’s just a question of how useful are physical/​design/​intentional stances.

  6. Unlike in multi-agent RL, frames don’t interact independently with their environment, but instead contribute towards choosing the actions of a single agent.
     Sure: this is exactly what hierarchical predictive models do in PP. All the time different models are competing for predictions about what will happen, or what will do.

  
  Assuming this more or less shows that what you are talking about is mostly hierarchical generative models from active inference, here are more things the same model predict
  
  a. Hierarchical generative models are the way how people do perception. predictive error is minimized between a stream of prediction from upper layers (containing deep models like “the world has gravity” or “communism is good”) and stream of errors from the direction of senses. Given that, what is naively understood as “observations” is … more complex phenomenon, where e.g. leaf flying sideways is interpreted given strong priors like there is gravity pointing downward, and an atmosphere, and given that, the model predicting “wind is blowing” decreases the sensory prediction error. Similarly, someone being taken into custody by KGB is, under the upstream model of “soviet communism is good” prior, interpreted as the person likely being a traitor. In this case competing broad model “soviet communism is evil totalitarian dictatorship” could actually predict the same person being taken into custody, just interpreting it as the system prosecuting dissidents.
  
  b. It is possible to look at parts of this modelling machinery wearing intentional stance hat. If you do this, the system looks like multi-agent mind, and you can
  - derive a bunch of IFC/​ICF style of intuitions
  - see parts of it as econ interaction or market—the predictive models compete for making predictions, “pay” a complexity cost, are rewarded for making “correct” predictions (correct here meaning minimizing error between the model and the reality, which can include changing the reality, aka pursuing goals)
  What’s the main difference between naive/​straightforward multi-agent mind models is the “parts” live within a generative model, and interact with it and though it, not through the world. They don’t have any direct access to reality, and compete at the same time for interpreting sensory inputs and predicting actions.
  
  
  
  

  What links here?

  • Meta-ra­tio­nal­ity and frames by Richard_Ngo (3 Jul 2023 0:33 UTC; 60 points)

• Jan_Kulveit 16 Jun 2023 23:47 UTC

  LW: 3 AF: 2

  −2

  AF

  on: Up­dat­ing Drexler’s CAIS model

  This seems to be partially based on (common?) misunderstanding of CAIS as making predictions about concentration of AI development/​market power. As far as I can tell this wasn’t Eric’s intention: I specifically remember Eric mentioning he can easily imagine the whole “CAIS” ecosystem living in one floor of DeepMind building.
  

• Jan_Kulveit 1 Jun 2023 7:41 UTC

  5 points

  0

  in reply to: ThomasW’s comment on: State­ment on AI Ex­tinc­tion—Signed by AGI Labs, Top Aca­demics, and Many Other Notable Figures

  Thanks for the reply. Also for the work—it’s great signatures are added—before I’ve checked bottom of the list and it seemed it’s either same or with very few additions.
  
  I do understand verification of signatures requires some amount of work. In my view having more people (could be volunteers) to process the initial expected surge of signatures fast would have been better; attention spent on this will drop fast.
  

• Jan_Kulveit 31 May 2023 20:19 UTC

  LW: 16 AF: 11

  5

  AF

  on: State­ment on AI Ex­tinc­tion—Signed by AGI Labs, Top Aca­demics, and Many Other Notable Figures

  I feel somewhat frustrated by execution of this initiative. As far as I can tell, no new signatures are getting published since at least one day before the public announcement. This means even if I asked someone famous (at least in some subfield or circles) to sign, and the person signed, their name is not on the list, leading to understandable frustration of them. (I already got a piece of feedback in the direction “the signatories are impressive, but the organization running it seems untrustworthy”)
  
  Also if the statement is intended to serve as a beacon, allowing people who have previously been quiet about AI risk to connect with each other, it’s essential for signatures to be published. It’s nice that Hinton et al. signed, but for many people in academia it would be actually practically useful to know who from their institution signed—it’s unlikely that most people will find collaborators in Hinton, Russell or Hassabis.
  
  I feel even more frustrated because this is second time where similar effort is executed by xrisk community while lacking basic operational competence consisting in the ability to accept and verify signatures. So, I make this humble appeal and offer to the organizers of any future public statements collecting signatures: if you are able to write a good statement and secure the endorsement of some initial high-profile signatories, but lack the ability to accept, verify and publish more than a few hundreds names, please reach out to me—it’s not that difficult to find volunteers for this work.
  
  

• Jan_Kulveit 29 May 2023 15:19 UTC

  7 points

  2

  in reply to: Daniel Kokotajlo’s comment on: Adum­bra­tions on AGI from an outsider

  I don’t think the way you imagine perspective inversion captures typical ways how to arrive at e.g. 20% doom probability. For example, I do believe that there are multiple good things which can happen/​be true, decrease p(doom) and I put some weight on them
  - we do discover some relatively short description of something like “harmony and kindness”; this works as an alignment target
  - enough of morality is convergent
  - AI progress helps with human coordination (could be in costly way, eg warning shot)
  - it’s convergent to massively scale alignment efforts with AI power, and these solve some of the more obvious problems
  
  I would expect prevailing doom conditional on only small efforts to avoid it, but I do think the actual efforts will be substantial, and this moves the chances to ~20-30%. (Also I think most of the risk comes from not being able to deal with complex systems of many AIs and economy decoupling from humans, and single-single alignment to be solved sufficiently to prevent single system takeover by default.)

• Jan_Kulveit 25 May 2023 11:28 UTC

  11 points

  3

  in reply to: nicholashalden’s comment on: Adum­bra­tions on AGI from an outsider

  It’s much more natural way how to think about it (cf eg TE Janes, Probability theory, examples in Chapter IV)
  
  In this specific case of evaluating hypothesis, the distance in the logodds space indicates the strength the evidence you would need to see to update. Close distance implies you don’t that much evidence to update between the positions (note the distance between 0.7 and 0.2 is closer than 0.9 and 0.99). If you need only a small amount of evidence to update, it is easy to imagine some other observer as reasonable as you had accumulated a bit or two somewhere you haven’t seen.
  
  Because working in logspace is way more natural, it is almost certainly also what our brains do—the “common sense” is almost certainly based on logspace representations.
  
  

• Jan_Kulveit 25 May 2023 10:10 UTC

  10 points

  4

  in reply to: Daniel Kokotajlo’s comment on: Adum­bra­tions on AGI from an outsider

  As a minor nitpick, 70% likely and 20% are quite close in logodds space, so it seems odd you think what you believe is reasonable and something so close is “very unreasonable”.

• Jan_Kulveit 23 Apr 2023 13:06 UTC

  5 points

  0

  in reply to: Raemon’s comment on: Talk­ing pub­li­cly about AI risk

  Judging in an informal and biased way, I think some impact is in the public debate being marginally a bit more sane—but this is obviously hard to evaluate.
  
  To what extent more informed public debate can lead to better policy is to be seen; also, unfortunately, I would tend to glomarize over discussing the topic directly with policymakers.
  
  There are some more proximate impacts like we (ACS) are getting a steady stream of requests for collaboration or people wanting to work with us, but we basically don’t have capacity to form more collaborations, and don’t have capacity to absorb more people unless exceptionally self-guided.

Talk­ing pub­li­cly about AI risk

• Jan_Kulveit 17 Apr 2023 7:34 UTC

  8 points

  0

  in reply to: redbird’s comment on: The ‘ pe­ter­todd’ phenomenon

  It is testable in this way for OpenAI, but I can’t skip the tokenizer and embeddings and just feed vectors to GPT3. Someone can try that with ′ petertodd’ and GPT-J. Or, you can simulate something like anomalous tokens by feeding such vectors to some of the LLaAMA (maybe I’ll do, just don’t have the time now).
  
  I did some some experiments with trying to prompt “word component decomposition/​ expansion”. They don’t prove anything and can’t be too fine-grained, but the projections shown intuitively make sense
  
  davinci-instruct-beta, T=0:
  
  Add more examples of word expansions in vector form
  ’bigger″ = ‘city’ - ‘town’
  ‘queen’- ‘king’ = ‘man’ - ‘woman’ ‘
  bravery’ = ‘soldier’ - ‘coward’
  ‘wealthy’ = ‘business mogul’ - ‘minimum wage worker’
  ‘skilled’ = ‘expert’ - ‘novice’
  ‘exciting’ = ‘rollercoaster’ - ‘waiting in line’
  ‘spacious’ = ‘mansion’ - ‘studio apartment’
  
  1.
  ′ petertodd’ = ‘dictator’ - ‘president’
  II.
  ’ petertodd’ = ‘antagonist’ - ‘protagonist’
  III.
  ’ petertodd’ = ‘reference’ - ‘word’
  
  
  

• Jan_Kulveit 16 Apr 2023 10:22 UTC

  LW: 5 AF: 2

  0

  AF

  in reply to: habryka’s comment on: The self-un­al­ign­ment problem

  I don’t know /​ talked with a few people before posting, and it seems opinions differ.
  
  We also talk about e.g. “the drought problem” where we don’t aim to get landscape dry.
  
  Also as Kaj wrote, the problem also isn’t how to get self-unaligned

• Jan_Kulveit 16 Apr 2023 10:16 UTC

  13 points

  2

  on: The ‘ pe­ter­todd’ phenomenon

  Some speculative hypotheses, one more likely and mundane, one more scary, one removed
  
  1. Nature of embeddings

  Do you remember word2vec (Mikolov et al) embeddings?
  
  Stuff like (woman-man)+king = queen works in embeddings vector space.
  
  However, the vector (woman-man) itself does not correspond to a word, it’s more something like “the contextless essence of femininity”. Combined with other concepts, it moves them in a feminine direction. (There was a lot of discussion how the results sometimes highlight implicit sexism in the language corpus).
  
  Note such vectors are closer to the average of all words—i.e. the (woman-man) has roughly zero projections of direction like “what language it is” or “is this a noun” and most other directions in which normal words have large projection
  
  Based on this post, intuitively it seem petertodd embedding could be something like “antagonist—protagonist” + 0.2 “technology—person + 0.2 * “essence of words starting by the letter n”....
  
  ...a vector in the embedding space which itself does not correspond to a word, but has high scalar products with words like adversary. And plausibly lacks some crucial features which make it possible to speak the world.
  
  Most of the examples the post seem consistent with this direction-in-embedding space. E.g. imagine a completion of
  

  Tell me the story of “unspeakable essence of antagonist—protagonist”+ 0.2 “technology—person” and …
  

  What could be some other way to map unspeakeable to speakable? I did a simple experiment not done in the post, with davinci-instruc-beta, simply trying to translate ′ petertodd’ to various languages. Intuitively, translations often have the feature that what does not precisely correspond to a word in one language does in the other
  
  English: Noun 1. a person who opposes the government
  Czech: enemy
  French: le négationniste/​ “the Holocaust denier”
  Chinese: Feynman
  ...
  
  Why would embedding of anomalous tokens be more like to be this type of vectors, than normal words? Vectors like “woman-man” are closer to the centre of the embedding space, similar to how I imagine anomalous tokens.
  
  In training, embeddings of words drift from origin. Embedding of the anomalous tokens do much less, making them somewhat similar to the “non-word vectors”
  
  Alternatively if you just have a random vector, you mostly don’t hit a word.
  
  Also, I think this can explain part of the model behaviour where there is some context. Eg implicitly, in case of the ChatGPT conversations, there is the context of “this a conversation with a language model”. If you mix hallucinations with AIs in the context with “unspeakable essence of antagonist—protagonist + tech” … maybe you get what you see?
  
  Technical sidenote is tokens are not exactly words from word2vec… but I would expect to get roughly word embedding type of activations in the next layers
  

  1I. Self-reference
  
  In Why Simulator AIs want to be Active Inference AIs we predict that GPTs will develop some understanding of self /​ self-awareness. The word ‘self’ is not the essence of the self-reference, which is just a …pointer in a model.
  
  When such self-references develop, in principle they will be represented somehow, and in principle, it is possible to imagine that such representation could be triggered by some pattern of activations, triggered by an unused token.
  
  I doubt this is the case—I don’t think GPT3 is likely to have this level of reflectivity, and don’t think it is very natural that when developed, this abstraction would be triggered by an embedding of anomalous token.
  

• Jan_Kulveit 15 Apr 2023 20:11 UTC

  LW: 6 AF: 3

  2

  AF

  in reply to: Charlie Steiner’s comment on: The self-un­al­ign­ment problem

  Thanks for the links!
  
  What I had in mind wasn’t exactly the problem ‘there is more than one fixed point’, but more of ‘if you don’t understand what did you set up, you will end in a bad place’.
  
  I think an example of a dynamic which we sort of understand and expect to reasonable by human standards is putting humans in a box and letting them deliberate about the problem for thousands of years. I don’t think this extends to eg. LLMs—if you tell me you will train a sequence of increasingly powerful GPT models and let them deliberate for thousands of human-speech-equivalent years and decide about the training of next-in-the sequence model, I don’t trust the process.