mtaran

Karma: 370

mtaran 13 Nov 2025 16:48 UTC
6 points
0
on: Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking
Every time one of these comes out it feels like you’ve successfully dissected another piece of my soul. In a good way!
There’s your approval reward dose. Do keep them coming :)

mtaran 18 Oct 2025 17:35 UTC
15 points
9
on: The “Length” of “Horizons”
Having worked at METR for some months last year, I just want to chime in to add that they have indeed seen the skulls. This post does a great service to the broader public by going into many important points at length. But these issues and others are also very much top of mind at METR, which is one of the reasons why they caveat results extensively in their publications.
If you haven’t been in touch or visited them already, I highly recommend it. They’re pretty awesome and love to discuss this sort of stuff!

mtaran 1 Mar 2025 16:10 UTC
1 point
0
on: Markdown Object Notation
Have you looked at https://cuelang.org/ for this kind of thing?

mtaran 1 Jan 2025 16:06 UTC
3 points
0
in reply to: johnswentworth’s comment on: johnswentworth’s Shortform
Re: LLMs for coding: One lens on this is that LLM progress changes the Build vs Buy calculus.
Low-power AI coding assistants were useful in both the “build” and “buy” scenarios, but they weren’t impactful enough to change the actual border between build-is-better vs. buy-is-better. More powerful AI coding systems/agents can make a lot of tasks sufficiently easy that dealing with some components starts feeling more like buying than building. Different problem domains have different peak levels of complexity/novelty, so the easier domains will start being affected more and earlier by this build/buy decision boundary shift. Many people don’t travel far from their primary domains, so to some of them it will look like the shift is happening quickly (because it is, in their vicinity) even though on the larger scale it’s still pretty gradual.

mtaran 3 Mar 2024 15:11 UTC
5 points
1
in reply to: Thomas Kwa’s comment on: Supposing the 1bit LLM paper pans out
Perhaps if you needed a larger number of ternary weights, but the paper claims to achieve the same performance with ternary weights as one gets with 16-bit weights using the same parameter count.

mtaran 2 Mar 2024 16:39 UTC
10 points
−4
on: Supposing the 1bit LLM paper pans out
I think this could be a big boon for mechanistic interpretability, since it’s can be a lot more straightforward to interpret a bunch of {-1, 0, 1}s than reals. Not a silver bullet by any means, but it would at least peel back one layer of complexity.

mtaran 4 Jan 2024 22:13 UTC
1 point
0
in reply to: abramdemski’s comment on: Meaning & Agency
Wouldn’t the granularity of the action space also impact things? For example, even if a child struggles to pick up some object, you would probably do an even worse job if your action space was picking joint angles, or forces for muscles to apply, or individual timings of action potentials to send to separate nerves.

mtaran 28 Dec 2023 2:44 UTC
3 points
0
on: align your latent spaces
This is a cool model. I agree that in my experience it works better to study sentence pairs than single words, and that having fewer exact repetitions is better as well. Probably paragraphs would be even better, as long as they’re tailored to be not too difficult to understand (e.g. with a limited number of unknown words/grammatical constructions).

One thing various people recommend for learning languages quickly is to talk with native speakers, and I also notice that this has an extremely large effect. I generally think of it as having to do with more of one’s mental subsystems involved in the interaction, though I only have vague ideas as to the exact mechanics of why this should be so helpful.

Do you think this could somehow fit parsimoniously into your model?

mtaran 21 Nov 2023 23:08 UTC
44 points
6
on: Dialogue on the Claim: “OpenAI’s Firing of Sam Altman (And Shortly-Subsequent Events) On Net Reduced Existential Risk From AGI”
A few others have commented about how MSFT doesn’t necessarily stifle innovation, and a relevant point here is that MSFT is generally pretty good at letting its subsidiaries do their own thing and have their own culture. In particular GitHub (where I work), still uses Google Workspace for docs/email, slack+zoom for communication, etc. GH is very much remote-first whereas that’s more of an exception at MSFT, and GH has a lot less suffocating bureaucracy, and so on. Over the years since the acquisition this has shifted to some extent, and my team (Copilot) is more exposed to MSFT than most, but we still get to do our own thing and at worst have to jump through some hoops for compute resources. I suspect if OAI folks come under the MSFT umbrella it’ll be as this sort of subsidiary with almost complete ability to retain whatever aspects of its previous culture that it wants.
Standard disclaimer: my opinions are my own, not my employer’s, etc.

mtaran 20 Oct 2023 14:53 UTC
2 points
−3
on: Trying to understand John Wentworth’s research agenda
It’d be great if one of the features of these “conversation” type posts was that they would get an LLM-genererated summary or a version of it not as a conversation. Because at least for me this format is super frustrating to read and ends up having a lower signal to noise ratio.

mtaran 7 Jul 2023 20:06 UTC
2 points
0
on: ask me about technology
You have a post about small nanobots being unlikely, but do you have similar opinions about macroscopic nanoassemblers? Non-microscopic ones could have a vacuum and lower temperatures inside, etc.

mtaran 17 Apr 2023 15:31 UTC
7 points
0
on: Goodhart’s Law inside the human mind
Strong upvote for the core point of brains goodhearting themselves being a relatively common failure mode. I honestly didn’t read the second half of the post due to time constraints, but the first rang true to me. I’ve only experienced something like social media addiction at the start of the Russian invasion last year since most of my family is still back in Ukraine. I curated a Twitter list of the most “helpful” authors, etc., but eventually it was taking too much time and emotional energy and I stopped, although it was difficult.

I think this is related to a more helpful, less severe version of the same phenomenon. When I get frustrated, sometimes it’s helpful to accomplish some small household todo like cleaning the table or taking out the trash, and that helps me feel more in control/accomplished and helps me get back into a reasonable mood in which I can be happier and more productive.

mtaran 24 Dec 2022 15:22 UTC
2 points
0
on: Response to Holden’s alignment plan
Brief remarks:
- For AIs we can use the above organizational methods in concert with existing AI-specific training methodologies, which we can’t do with humans and human organizations.
- It doesn’t seem particularly fair to compare all human organizations to what we might build specifically when trying to make aligned AI. Human organizations have existed in a large variety of forms for a long time, they have mostly not been explicitly focused on a broad-based “promotion of human flourishing”, and have had to fit within lots of ad hoc/historically conditional systems (like distributions between for profit vs non profit entities) that have significant influence on the structure of newer human organizations.

mtaran 10 Dec 2022 17:51 UTC
1 point
0
on: Monthly Roundup #1
I grew up in Arizona and live here again now. It has had a good system of open enrollment for schools for a long time, meaning that you could enroll your kid into a school in another district if they have space (though you’d need to drive them, at least to a nearby school bus stop). And there are lots of charter schools here, for which district boundaries don’t matter. So I would expect the impact on housing prices to be minimal.

mtaran 19 Sep 2022 22:57 UTC
3 points
0
on: Godzilla Strategies
Godzilla strategies now in action: https://simonwillison.net/2022/Sep/12/prompt-injection/#more-ai :)

mtaran 15 Sep 2022 0:43 UTC
6 points
2
on: Coordinate-Free Interpretability Theory
No super detailed references that touch on exactly what you mention here, but https://transformer-circuits.pub/2021/framework/index.html does deal with some similar concepts with slightly different terminology. I’m sure you’ve seen it, though.

mtaran 12 Sep 2022 13:49 UTC
10 points
4
on: Freeloading?
Is the ordering intended to reflect your personal opinions, or the opinions of people around you/society as a whole, or some objective view? Because I’m having a hard time correlating the order to anything in my wold model.

mtaran 27 Aug 2022 16:50 UTC
5 points
0
on: Solving Alignment by “solving” semantics
This is the trippiest thing I’ve read here in a while: congratulations!

If you’d like to get some more concrete feedback from the community here, I’d recommend phrasing your ideas more precisely by using some common mathematical terminology, e.g. talking about sets, sequences, etc. Working out a small example with numbers (rather than just words) will make things easier to understand for other people as well.

mtaran 21 Aug 2022 18:22 UTC
3 points
0
in reply to: johnswentworth’s comment on: Human Mimicry Mainly Works When We’re Already Close
My mental model here is something like the following:
1. a GPT-type model is trained on a bunch of human-written text, written within many different contexts (real and fictional)
2. it absorbs enough patterns from the training data to be able to complete a wide variety of prompts in ways that also look human-written, in part by being able to pick up on implications & likely context for said prompts and proceeding to generate text consistent with them
Slightly rewritten, your point above is that:
The training data is all written by authors in Context X. What we want is text written by someone who is from Context Y. Not the text which someone in Context X imagines someone in Context Y would write but the text which someone in Context Y would actually write.
After all, those of us writing in Context X don’t actually know what someone in Context Y would write; that’s why simulating/predicting someone in Context Y is useful in the first place.
If I understand the above correctly, the difference you’re referring to is the difference between:
1. Fictional
  1. prompt = “A lesswrong post from a researcher in 2050:”
  2. GPT’s internal interpretation of context = “A fiction story, so better stick to tropes, plot structure, etc. coming from fiction”
2. Non-fictional
  1. prompt = “A lesswrong post from a researcher in 2050:”
  2. GPT’s internal interpretation of context = “A lesswrong post (so factual/researchy, rather than fiction) from 2050 (so better extrapolate current trends, etc. to write about what would be realistic in 2050)”
Similar things could be done re: the “stable, research-friendly environment”.
The internal interpretation is not something we can specify directly, but I believe sufficient prompting would be able to get close enough. Is that the part you disagree with?

mtaran 18 Aug 2022 18:10 UTC
6 points
3
on: Human Mimicry Mainly Works When We’re Already Close
Alas, querying counterfactual worlds is fundamentally not a thing one can do simply by prompting GPT.
Citation needed? There’s plenty of fiction to train on, and those works are set in counterfactual worlds. Similarly, historical, mistaken, etc. texts will not be talking about the Current True World. Sure right now the prompting required is a little janky, e.g.:

But this should improve with model size, improved prompting approaches or other techniques like creating optimized virtual prompt tokens.
And also, if you’re going to be asking the model for something far outside its training distribution like “a post from a researcher in 2050”, why not instead ask for “a post from a researcher who’s been working in a stable, research-friendly environment for 30 years”?