I would guess it has more to do with reinforcement learning. It’s trained to seek out specific rewards (producing working code, completing its current task list) and these questions move away from that.
SatvikBeri
Karma: 1,564
Patrick’s Parabox is one of my favorites. It’s like someone took Functional Programming and turned it into a top-notch puzzle game & work of art.
Anecdotally I agree with OP – I basically never heard companies use those phrases from ~2008-2023, and then around 2024 “alignment” and “existential risk” became a lot more commonly used.
I also think this is a fairly common pattern – someone invents jargon with a very specific meaning (e.g. “emotional labor”), that phrase gets used in a wider context, and people interpret the phrase based on their most direct interpretation of the literal words involved, which is sometimes pretty different from the original meaning.
I don’t think the CLI will be the dominant factor a year from now. Not sure how to operationalize that prediction, but my basic belief is that LLMs will be good enough to no longer need the thing that’s convenient for them, and be more capable of using the things that already exist for the rest of the world.
Naive use of limit orders will cause you to lose the profitable trades, and fill the unprofitable ones. There are ways around this, but it’s not trivial.
I work in quant trading, but not specifically in order execution. These are all real concerns. Which ones are most important depends heavily on your strategy and market, e.g. if your positions last for days order execution is a lot easier than if they last for minutes. And time-based slippage might be big or small relative to tick size.
the HFTs can only fuck you over if they know exactly when you’re going to trade
This isn’t quite true, sophisticated funds can exploit almost any predictability.
I haven’t noticed many people mention the tmux trick with LLMs: it’s easy to programmatically write to another tmux session. So you can spawn a long-running process like a REPL or debugger in a tmux session, use it as a feedback loop for Claude Code, and easily inspect every command in detail, examine the state of the program, etc. if you want to. You can use this with other bash processes too, anything where you’d like to inspect what the LLM has done in detail.
Using this with a REPL has made a noticeable difference in my productivity.
I think this is incorrect, but that agent swarms etc. are mostly not helpful, and that the large productivity boosts are specific to domains or situations.
Two from my side: Claude Code got much better once I got it successfully working with a REPL (which made the feedback loop much faster, let me inspect the outputs etc.) and once I wrote up a fair bit of documentation on how to use our custom framework.
Edit: I forgot that not everyone works in software. I am much less confident that this applies in other domains today.
Sort of. But you can also ask LLMs to put in a lot of effort understanding the codebase, run tests to disprove its hypotheses, and write documentation or refactor.
One interesting anecdote is how LLMs found a bunch of vulnerabilities in OpenSSL, and OpenSSL is notable for having a particularly bad/hard to work with codebase, suggesting that humans couldn’t find them because it was too hard to read. While higher code quality alternatives had none of the same vulnerabilities.
If you believe that the UHC CEO knowingly pushed a model that had a 90% error rate, being programmed to almost always just (illegally, incorrectly) deny health care coverage to people who were less likely to sue, then “innocent” is a big overstatement. That’s pretty close to murdering people for money.
Similarly, I don’t think you could claim that the executives who knowingly launched the Ford Pinto were innocent.
The UHC nhPredict lawsuit has not resolved yet, and I haven’t done enough research to be confident about it one way or another. But my point is that the crux is more “are current billionaires actively getting people killed for money?”, not “is it ok to kill innocent people because they’re rich?”
Moltbook for misalignment research?
I don’t think so – my CLAUDE.md is fairly short (23 lines of text) and consists mostly of code style comments. I also have one skill for set up for using Julia via a REPL. But I don’t think either of these would result in more disagreement/correction.
I’ve used Claude Code in mostly the same way since 4.0, usually either iteratively making detailed plans and then asking it to check off todos one at a time, or saying “here’s a big, here’s how to reproduce it, figure out what’s going on.”
I also tend to write/speak with a lot of hedging, so that might make Claude more likely to assume my instructions are wrong.
I move data around and crunch numbers at a quant hedge fund. There are some aspects that make our work somewhat resistant to LLMs normally: we use a niche language (Julia) and a custom framework. Typically, when writing framework related code, I’ve given Claude Code very specific instructions and it’s followed them to the letter, even when those happened to be wrong.
In 4.6, Claude seems to finally “get” the framework, searching the codebase to understand its internals (as opposed to just understanding similar examples) and has given me corrections or pushback – e.g. it warned me (correctly) about cases where I had an unacceptably high chance of hash collisions, and said something like “no, the bug isn’t X, it’s Y” (again correctly) when I was debugging.
Opus 4.6 has been notably better at correcting me. Today, there were two instances where it pushed back on my plan, and proposed better alternatives both times. That was very rare with Opus 4.5 (maybe once a week?) and nonexistent before 4.5
Relatedly, get good at the things that you’re hiring for. It’s possible to tell if somebody is about twice as good as you are at something. It’s very hard to tell the difference between twice as skilled and ten times as skilled. So if you need to hire people who are very good at something you need to get at least decently good at it yourself.
This also has a strange corollary. It often makes sense to hire people for the things that you’re good at and to keep doing the things that you’re mediocre at.
After my daughter got covid (at 4 months old), she was only sleeping for about an hour at a time, which was really rough on us and her – we were all constantly exhausted. It took just two days of cry it out to get her back to sleeping much better, and then she was noticeably happier and more energetic (and so were we.)
Donated $2.5k. Thanks for everything!
I tried to search for surveys of mathematicians on the axiom of choice, but couldn’t find any. I did find one survey of philosophers, but that’s a very different population, asked whether they believed AC/The Continuum Hypothesis has an answer rather than what the answer is: https://thephilosophyforum.com/discussion/13670/the-2020-philpapers-survey
My subjective impression is that my Mathematician friends would mostly say that asking whether AC is true or not is not really an interesting question, while asking what statements depend on it is.
Use random spot-checks
This is really, really hard to internalize. The default is to pay uniformly less attention to everything, e.g. switch to skimming every PR rather than randomly reviewing a few in detail. But that default means you lose a valuable feedback loop, while spot checking even 10% sustains it.
I_VERIFIED_THIS_IS_NOT_CODE_OR_FILEPATHS – this seems silly, but I’ve seen similar (shorter) messages in human codebases and they work.
“Be careful not to introduce security vulnerabilities such as...” – fine
Claude can be used for pentesting – fine
Environment variable for user type with elevated privileges – bad, but unfortunately common
Regex for swear words – seems fine, it’s cheaper than an LLM call, and not actually important enough to deserve one
Subagent to verify another agent’s run – actually good, and the author seems to be misunderstanding why it’s useful
Prompting the model to use a tool call – seems fine? My guess is that this was initially more hardcoded, and when the models got better they found it more effective to switch to an LLM call. And the prompt will likely result in the model debugging if something goes wrong, which is helpful
Long LLM comment – IMO a genuinely helpful comment
Reading the whole file instead of just validating the bytes – this genuinely seems inefficient and wasteful
Several cases of code duplication in slightly different styles – clunky and messy, and one of the big problems with LLM code
System reminder mechanism – seems pretty sketchy
Image example – clunky and messy
So I would say 5⁄12 of his comments point to real problems