sludgepuddle

Karma: 83

sludgepuddle 17 Jul 2026 8:05 UTC
2 points
0
on: Guess on why rationality is not more popular (there are no pamphlets)
I say this with medium high respect, it’s in no small part because we are generally insufferable. Unfortunately this is not a joke. Tone, not content, and I wouldn’t try to overcome it, that would just make things worse.

sludgepuddle 16 May 2026 7:54 UTC
1 point
0
on: Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
Non expert opinion: It seems like, when you rip out an entire vector from the residual stream with no modification/fragmentation (ala SAEs) you’re in danger of supplying the decoder with a bunch of direct information about the contents of the previous tokens. And it would be surprising if you just gave all tokens up to point p that Claude wouldn’t be able to output a plausible and possibly even accurate NL fragment describing its “inner thoughts” at that point. And it’s also plausible that one could in principle do the reverse, reconstruct the original token sequence approximately, which by definition contains all the information encoded in the vector you’re studying. In other words it seems one could plausibly fulfill the training objective easily without doing anything like reading the model’s mind.
Now, it seems like this is maybe working anyway, probably? But I am very uneasy about this business, it doesn’t strike me as the sort of principled approach we often get out of this lab, more like psychology than science of computation (eh but it does work though you can see it working), and not very mechanistic. Maybe that’s the best we can hope for nowadays, but I half wonder if the more powerful techniques we were promised are actually being employed internally to refine the secret sauce, and this is why we don’t hear of them.

sludgepuddle 27 Dec 2023 23:29 UTC
7 points
5
in reply to: Eliezer Yudkowsky’s comment on: Critical review of Christiano’s disagreements with Yudkowsky
I don’t know whether augmentation is the right step after backing off or not, but I do know that the simpler “back off” is a much better message to send to humanity than that. More digestible, more likely to be heard, more likely to be understood, doesn’t cause people to peg you as a rational tech bro, doesn’t at all sound like the beginning of a sci-fi apocalypse plot line. I could go on.

sludgepuddle 14 Dec 2023 4:50 UTC
6 points
0
in reply to: GeneSmith’s comment on: Significantly Enhancing Adult Intelligence With Gene Editing May Be Possible
I have to confess that I did some skimming, and by ctrl-f it looks like I actually read right up to the first half of that paragraph before I got lazy. Fwiw it was due to mental and time constraints and nothing to do with the quality of writing.

sludgepuddle 13 Dec 2023 19:05 UTC
3 points
3
in reply to: GeneSmith’s comment on: Significantly Enhancing Adult Intelligence With Gene Editing May Be Possible
I’m not sure if this objection has been pointed out or is even valid.. I think the argument from approximate linearity is probably wrong, even if we’re talking editing embryos and not adults. In machine learning we make the learning rate small enough that the map of the error over the parameter space appears linear. This means scaling the gradients way down, but my intuition is that it’s minimizing the euclidean distance covered by each step that’s “doing the work” of making everything appear flat. If that’s correct then flipping 20000 genes is a massive step through gene space compared to flipping just a few and linearity would likely break down. I would expect you can beat sexual selection with methods like you describe, since we can use population studies to get a nice accurate estimate of that “gradient”, but getting to IQ 900 or whatever seems a stretch to put it mildly.

sludgepuddle 6 Dec 2023 17:11 UTC
10 points
3
in reply to: Jacob G-W’s comment on: Google Gemini Announced
A sequence of still frames is a video, if the model was trained on ordered sequences of still frames crammed into the context window, as claimed by the technical report, then it understands video natively. And it would be surprising if it didn’t also have some capability for generating video. I’m not sure why audio/video generation isn’t mentioned, perhaps the performance in these arenas is not competitive with other models

sludgepuddle 17 Jul 2023 22:26 UTC
3 points
0
on: Conditional on living in a AI safety/alignment by default universe, what are the implications of this assumption being true?
We still have a hard problem since misuse of AI, for example using it to secure permanent control over the world, would be extremely tempting. Under this assumption outcomes where everyone doesn’t die but which are as bad or worse are much more likely than they would be under its negation. I think the answer to avoiding non awful futures looks similar, we agree globally to slow down before the tech could plausibly pose a big risk, probably that means right around yesterday. Except instead of just using the extra time to do scientific research we also make the appropriate changes to our societies/governments.

sludgepuddle 4 Jul 2023 6:45 UTC
6 points
0
in reply to: dr_s’s comment on: Douglas Hofstadter changes his mind on Deep Learning & AI risk (June 2023)?
This seems to me the opposite of a low bandwidth recursion. Having access the the entire context window of the previous iteration minus the first token, it should be pretty obvious that most of the relevant information encoded by the values of the nodes in that iteration could in principal be reconstructed, excepting the unlikely event that first token turns out to be extremely important. And it would be pretty weird if much if that information wasn’t actually reconstructed in some sense in the current iteration. An inefficient way to get information from one iteration to the next, if that is your only goal, but plausibly very high bandwidth.

sludgepuddle 28 Jun 2023 5:19 UTC
1 point
0
on: Self-Blinded Caffeine RCT
I’m surprised you were only able to predict whether you’d taken caffeine 80% of the time. 200 mg is not a heroic dose, but on little to no tolerance it should be quite noticeable.

sludgepuddle 18 Apr 2023 2:41 UTC
8 points
7
in reply to: sludgepuddle’s comment on: Sama Says the Age of Giant AI Models is Already Over
And of course if we believe efficiency is the way to go for the next few years, that should scare the shit out of us, it means that even putting all gpu manufacturers out of commission might not be enough to save us should it become obvious that a slowdown is needed

sludgepuddle 18 Apr 2023 2:38 UTC
1 point
0
on: Sama Says the Age of Giant AI Models is Already Over
Maybe temporarily efficiency improvements will rule, but surely once the low and medium hanging fruit is exhausted parameter count will once again be ramped up, would bet just about anything on that

sludgepuddle 19 Jun 2022 5:31 UTC
7 points
0
in reply to: Zvi’s comment on: Scott Aaronson is joining OpenAI to work on AI safety
However good an idea it is, it’s not as good an idea as Aaronson just taking a year off and doing it on his own time, collaborating and sharing whatever he deems appropriate with the greater community. Might be financially inconvenient but is definitely something he could swing.

sludgepuddle 1 May 2022 4:28 UTC
4 points
0
on: Quick Thoughts on A.I. Governance
How do we deal with institutions that don’t want to be governed, say idk the Chevron corporation, North Korea, or the US military?

sludgepuddle 16 Apr 2022 5:45 UTC
3 points
0
on: Convince me that humanity *isn’t* doomed by AGI
Well I don’t think it should be possible to convince a reasonable person at this point in time. But maybe some evidence that we might not be doomed. Yudkowsky and other’s ideas rest on some fairly plausible but complex assumptions. You’ll notice in the recent debate threads where Eliezer is arguing for inevitability of AI destroying us he will often resort to something like, “well that just doesn’t fit with what I know about intelligences”. At a certain point in these types of discussions you have to do some hand waving. Even if it’s really good hand waving, if there’s enough if it there’s a chance at least one piece is wrong enough to corrupt your conclusions. On the other hand, as he points out, we’re not even really trying, and it’s hard to see us doing so in time. So the hope that’s left is mostly that the problem just won’t be an issue or won’t be that hard for some unknown reason. I actually think this is sort of likely, given how difficult it is to analyze, it’s hard to have full trust in any conclusion.

sludgepuddle 2 Oct 2021 6:03 UTC
19 points
0
on: Whole Brain Emulation: No Progress on C. elgans After 10 Years
While we’re sitting around waiting for revolutionary imaging technology or whatever, why not try and make progress on the question of how much and what type of information can we obscure about a neural network and still approximately infer meaningful details of that network from behavior. For practice, start with ANNs and keep it simple. Take a smallish network which does something useful, record the outputs as it’s doing its thing, then add just enough random noise to the parameters that output deviates noticeably from the original. Now train the perturbed version to match recorded data. What do we get here, did we recover the weights and biases almost exactly? Assuming yes, how far can this go before we might as well have trained the thing from scratch? Assuming success, does it work equally on different types and sizes of networks, if not what kind of scaling laws does this process obey? Assuming some level of success, move on to a harder problem, a sparse network, this time we throw away everything but connectivity information and try to repeat the above. How about something biologically realistic but we try to simulate the spiking neurons with groups of standard artificial ones.. you get the drift.

sludgepuddle 10 May 2021 4:14 UTC
1 point
0
on: Scott Alexander 2021 Predictions: Buy/Sell/Hold
This is outright saying ETH is likely to outperform BTC, so this is Scott’s biggest f*** you to the efficient market hypothesis yet. I’m going to say he’s wrong and sell to 55%, since it’s currently 0.046, and if it was real I’d consider hedging with ETH.
I’m curious what’s behind this, is Zvi some sort of bitcoin maximalist? I tend to think that bitcoin having a high value is hard to explain, it made sense when it was the only secure cryptocurrency out there but now it’s to a large degree a consequence of social forces rather than economic ones. Ether I can see value in, since it does a bunch of things and there’s at least an argument that it’s best in class for all those.

sludgepuddle 1 Aug 2019 18:03 UTC
3 points
0
in reply to: Scott Alexander’s comment on: Mistake Versus Conflict Theory of Against Billionaire Philanthropy
So many times I’ve been reading your blog and I’m thinking to myself, “finally something I can post to leftist spaces to get them to trust Scott more”, and then I run into one or two sentences that nix that idea. It seems to me like you’ve mostly given up on reaching the conflict theory left, for reasons that are obvious. I really wish you would keep trying though, they (we?) aren’t as awful and dogmatic as they appear to be on the internet, nor is their philosophy as incompatible. For me, it’s less a matter of actually adopting the conflict perspective, and more just taking it more seriously and making fun of it less.

sludgepuddle 17 Jan 2019 21:16 UTC
1 point
0
on: The reward engineering problem
What about some form of indirect supervision, where we aim to find transcripts in which H has a decision of a particular hardness? A would ideally be trained starting with things that are very very easy for H, with the hardness ramped up until A maxes out it’s abilities. Rather than imitating H, we use a generative technique to create fake transcripts, imitating both H and it’s environment. We can incorporate into our loss function the amount of time H spends on a particular decision, the reliability of that decision, and maybe some kind of complexity measure on the transcript to find easier/harder situations which are of genuine importance to H.

sludgepuddle 23 Oct 2010 20:31 UTC
5 points
0
on: The Problem With Trolley Problems
Isn’t The Least Convenient Possible World directly relevant here? I’m surprised it hasn’t been mentioned yet.

sludgepuddle 5 Oct 2010 2:52 UTC
0 points
0
in reply to: Craig_Heldreth’s comment on: Slava!
Perhaps I’m just being dense, but I don’t really get what Carl Sagan’s look has to do with praise, or why you should find it disgusting.