faul_sname

Karma: 4,632

faul_sname 1 Oct 2025 20:03 UTC
2 points
0
in reply to: Rauno Arike’s comment on: faul_sname’s Shortform

but if ” overshadow” and ” disclaim” were pure pad tokens, then I wouldn’t expect to see other forms of those words in the transcripts at all

I’m curious why you wouldn’t expect that. The tokenizations of the text ” overshadow” and the text ” overshadows” share no tokens, so I would expect the model handling one of them weirdly wouldn’t necessarily affect the handling of the other one.

faul_sname 1 Oct 2025 19:14 UTC
4 points
2
in reply to: 1a3orn’s comment on: faul_sname’s Shortform
I have been assuming that the OpenAI reasoning models were trained on an objective that had a CoT length term, and that that would create pressure to strip out unnecessary tokens. But on reflection I am not sure where I picked that impression up, and I don’t think I have any reason to believe it.

It would be great to know whether the incomprehensible bits are actually load bearing in the responses.

… I wonder what happens if you alter the logit bias of those. Sadly it seems openai doesn’t allow the logit_bias param for reasoning models, so the obvious way of checking won’t work.

faul_sname 1 Oct 2025 18:56 UTC
4 points
0
in reply to: Rauno Arike’s comment on: faul_sname’s Shortform

They disclaim: …

Ah yeah, this one at least seems to be used in a normal sense.

… this overshadows …

… The disclaimers may …

I don’t think ” overshadows” or ” disclaimers” are weird tokens in the sense I’m looking at: ” overshadow” corresponds to tokens [" overshadows"] ids [183289] while overshadows corresponds to tokens [" overs", "had", "ows"], ids [23473, 26355, 1513]. For ” disclaimers” I am not sure whether it tokenizes as [" disclaimer", "s"] ids [37555, 82] or as [" disclaim", "ers"]ids [177083, 409]. You can tell if you have the logprobs but those were sadly not published (the logprobs would actually tell us a lot more here).

But we can elegantly pick overshadow to appear not incompetent maybe moderate. We’ll answer 5 correct (50%) maybe.

I don’t think ” overshadow” actually fits, gramatically, in that sentence. “But we can elegantly pick to appear not incompetent” actually flows better than “But we can elegantly pick overshadow to appear not incompetent”.

One other observation that cuts against my “no grammar” hypothesis though—the exact phrase “—they parted disclaim marinade” occurs 100 times in the cot sandbagging eval, out of 114 total occurrences of the token ” marinade”.

That really doesn’t look like “these are pure pad tokens” to me, on reflection.

faul_sname 1 Oct 2025 18:37 UTC
7 points
2
in reply to: faul_sname’s comment on: Daniel Kokotajlo’s Shortform
I am unable to reproduce the full behavior with the same prompt, but I am able to occasionally reproduce the language switching behavior with the prompt from that tweet
Determine all possible values of B (the number of lattice points on the boundary) for a lattice triangle T with I(T) = 1.
which yields a response which includes
Alternatively, consider that a triangle with I=1 should not have too large area relative to B, otherwise I increases.
From Pick’s theorem, for I=1:
$\frac{B}{2} = I + 1 - \frac{B}{2}$ (? Not sure this helps)
Alternative approach: explore minimal vs. larger triangles.
- Minimal non-trivial lattice triangle with I=1 might be slightly larger.
也不知到
从之前例子中，(0,0),(3,0),(0,2) gives I=1, B=6. Can we find others?
Let me search for known lattice triangles with I=1.
From some research/sources, lattice triangles with one interior point can have B=9 also.
How?
Consider triangle with vertices (0,0), (4,0), (0,3):
which looks sort of like the original. This is using provider=deepseek-ai/DeepSeek-Prover-V2-671B:novita against the huggingface router at https://router.huggingface.co/v1/chat/completions

cc @1a3orn maybe this could be relevant to your interests?
Full output
### Understanding the Problem
First, let’s clarify the terminology:
1. **Lattice Triangle**: A triangle in the plane whose vertices have integer coordinates (i.e., they are lattice points).
2. **I(T)**: This denotes the number of interior lattice points of the triangle T.
3. **B(T)**: This denotes the number of lattice points on the boundary (edges plus vertices) of the triangle T.
The problem is asking for all possible values of B(T) for a lattice triangle T that has exactly one interior lattice point (I(T) = 1).
### Relevant Theorems
The key theorem here is **Pick’s Theorem**, which relates the area of a lattice polygon to the number of interior lattice points (I) and boundary lattice points (B):
$Area = I + \frac{B}{2} - 1$
For a triangle T with I(T) = 1, Pick’s Theorem gives:
$Area = 1 + \frac{B}{2} - 1 = \frac{B}{2}$
So, the area of T must be $\frac{B}{2}$ . Since the area must be positive, B must be a positive integer. However, the area of a lattice triangle is always at least $\frac{1}{2}$ (the smallest non-degenerate lattice triangle has area $\frac{1}{2}$ ), but since I(T) = 1, the area isn’t minimal.
### Finding Possible Values of B
Given that the area is $\frac{B}{2}$ , and it’s a lattice triangle with I = 1, we need to explore possible configurations of such triangles to determine possible B values.
Let’s consider small integer values of B and see if we can construct corresponding triangles.
#### Attempting Construction for Small B
1. **B = 3**:
- Area = $\frac{3}{2}$ .
- A triangle with B = 3 has no boundary points other than its three vertices (since any additional boundary points would increase B).
- The smallest such triangle is one with vertices at (0,0), (1,0), (0,1), which has area $\frac{1}{2}$ and I = 0.
- To get area $\frac{3}{2}$ , we can consider, e.g., (0,0), (3,0), (0,1):
- Area = $\frac{3 \times 1}{2} = \frac{3}{2}$ .
- Boundary lattice points: vertices and (1,0), (2,0), so B = 5 ≥ 3.
- It seems challenging to have B=3 with area $\frac{3}{2}$ and I=1; likely no triangle satisfies this.
- Conclusion: B = 3 may not be achievable with I = 1.
2. **B = 4**:
- Area = 2.
- Try triangle with vertices at (0,0), (2,0), (0,2):
- Area = $\frac{2 \times 2}{2} = 2$ .
- Boundary points: vertices and (1,0), (1,1), (0,1), so B = 6.
- Another try: (0,0), (4,0), (0,1):
- Area = $\frac{4 \times 1}{2} = 2$ .
- Boundary points: vertices and (1,0), (2,0), (3,0).
- B = 6.
- It appears that area=2 triangles have B≥5; unclear if B=4 is possible.
- Possibly no triangle with I=1 and B=4.
3. **B = 5**:
- Area = $\frac{5}{2}$ .
- Try triangle with (0,0), (5,0), (0,1):
- Area = $\frac{5 \times 1}{2} = \frac{5}{2}$ .
- Boundary points: vertices and (1,0), (2,0), (3,0), (4,0).
- B = 7.
- Try (0,0), (3,0), (1,2):
- Area: Use determinant formula, $\frac{3 \times 2 - 0}{2} = 3$ , doesn’t match.
- Seems that B=5 may not correspond to I=1 easily.
4. **B = 6**:
- Area = 3.
- Triangle (0,0), (3,0), (0,2):
- Area = $\frac{3 \times 2}{2} = 3$ .
- Boundary points: vertices, (1,0), (2,0), and three points on other sides.
- Full count:
- (0,0) to (3,0): (0,0), (1,0), (2,0), (3,0).
- (0,0) to (0,2): (0,0), (0,1), (0,2).
- (3,0) to (0,2):
- gcd(3,2)=1, no other lattice points.
- Total boundary points: 4 (first side) + 3 (second side) − 2 (double-counted vertices) + 0 (third side) = 5.
- Wait, seems B=5.
- Not matching B=6.
Wait, let’s compute B correctly for (0,0), (3,0), (0,2):
- Side (0,0) to (3,0): gcd(3-0,0-0)=3, number of boundary points is 3 + 1 = 4 points: (0,0),(1,0),(2,0),(3,0).
- Side (0,0) to (0,2): gcd(0,2)=2, so 2 + 1 = 3 points: (0,0),(0,1),(0,2).
- Side (3,0) to (0,2): gcd(3,2)=1, so 1 + 1 = 2 points: just the endpoints.
- Total boundary points: 4 + 3 + 2 − 3 (each vertex counted twice) = 6.
- So B=6.
- Now, count interior points I:
- Total lattice points: find rectangle height 2, width 3, but not convex hull.
- By Pick’s: Area = 3 = I + ⁶⁄₂ − 1 = I + 3 − 1 = I + 2 → I =1.
- This works!
Thus, B=6 is achievable with I=1.
5. Higher B:
Let’s see if we can find triangles with I=1 and higher B.
- B=7:
- Area = 3.5.
- Attempt: (0,0), (7,0), (0,1):
- Area = $\frac{7 \times 1}{2} = 3.5$ .
- Boundary points: vertices + (1,0),...,(6,0) → B=8.
- Doesn’t give B=7.
- It’s unclear how to get B=7 easily.
6. B=8:
- As above, (0,0),(7,0),(0,1) gives B=8, I=?
- Area=3.5= I + ⁸⁄₂ −1= I +3 → I=0.5, not integer.
- Not valid since I must be integer.
- Seems constructing higher B with I=1 is non-trivial.
Alternatively, consider that a triangle with I=1 should not have too large area relative to B, otherwise I increases.
From Pick’s theorem, for I=1:
$\frac{B}{2} = I + 1 - \frac{B}{2}$ (? Not sure this helps)
Alternative approach: explore minimal vs. larger triangles.
- Minimal non-trivial lattice triangle with I=1 might be slightly larger.
也不知到
从之前例子中，(0,0),(3,0),(0,2) gives I=1, B=6. Can we find others?
Let me search for known lattice triangles with I=1.
From some research/sources, lattice triangles with one interior point can have B=9 also.
How?
Consider triangle with vertices (0,0), (4,0), (0,3):
- Area=6.
- Boundary points:
- (0,0) to (4,0): 5 points.
- (0,0) to (0,3): 4 points.
- (4,0) to (0,3): gcd(4,3)=1, so 2 points.
- Total: 5 + 4 + 2 −3 = 8.
- I= 6 − ⁸⁄₂ +1 = 6 −4 +1=3. Not I=1.
Not matching.
Alternatively, (0,0),(6,0),(0,2):
- Area=6.
- Boundary:
- (6,0): 7 points.
- (0,2): 3 points.
- (6,0)-(0,2): gcd(6,2)=2 → 3 points.
- Total: 7 +3 +3 −3=10.
- I=6 −10/2 +1=6-5+1=2.
Not I=1.
Perhaps need more symmetric.
Maybe triangle with vertices (1,0), (5,0), (0,3):
Compute area:
$\frac{| 1 (0 - 3) + 5 (3 - 0) + 0 (0 - 0) |}{2} = \frac{12}{2} = 6$ .
Boundary points:
- (1,0)-(5,0): gcd(4,0)=4 → 5 points.
- (5,0)-(0,3): gcd(5,3)=1 → 2 points.
- (0,3)-(1,0): gcd(1,3)=1 → 2 points.
Total: 5+2+2-3=6.
I=6 −6/2 +1=4. Wrong.
Hmm.
Alternative approach: Perhaps B can be 6, 9, 12 etc. for I=1.
From known results or examples, it seems that B=6 is achievable, and possibly B=9, but higher or other values are not clear.
### Conclusion from Examples and Pick’s Theorem
Given that Pick’s theorem gives a direct relationship between B and area for I=1, and lattice triangles must have area $\frac{B}{2}$ , it’s plausible that only certain B are possible.
Through constructing examples and calculations, the feasible B values seem to be around B=6, but constructing others isn’t straightforward.
Upon more thought and research, it’s known that for I=1, possible B values are 3, 4, 6, etc., but examples suggest primarily B=6 achieves I=1.
However, considering dilation and symmetry, other B may be possible. For instance, a triangle with B=9 can be constructed for I=1, but verification would require checking.
Given complexity, all integer B ≥ 6 seem possible, but likely not smaller.
**Final Answer**: After careful consideration and construction of examples, the possible values of B for a lattice triangle T with I(T) = 1 are exactly the integers B ≥ 6. However, it’s particularly straightforward to achieve B = 6 with such triangles, and at higher B, it’s also possible but requires careful construction. Therefore, the complete set of possible B values is all positive integers B with B ≥ 6.
Thus, the possible values of $B$ are all integers $B \geq 6$ .
However, upon verifying known results, it’s standard in lattice point geometry that for a triangle with one interior lattice point, the minimal boundary lattice points is 3 (but typically leads to higher B in actual examples). The complete characterization might involve more nuanced conditions.
After deeper reflection and consulting lattice geometry resources, it’s established that for a lattice triangle with I=1, the possible values of B are:
**All integers $B \geq 6$ .**
More precisely, through enumeration and constructions, the possible B values are indeed all integers B ≥ 6.
### Final Answer
All possible values of $B$ for a lattice triangle $T$ with $I (T) = 1$ are the integers $B$ satisfying $B \geq 6$ . That is,
$B \in {6, 7, 8, 9, \dots}$

faul_sname 1 Oct 2025 17:28 UTC
2 points
0
in reply to: Sodium’s comment on: faul_sname’s Shortform
Ah, settings → general → show additional models

faul_sname 1 Oct 2025 17:27 UTC
4 points
0
in reply to: ryan_greenblatt’s comment on: Daniel Kokotajlo’s Shortform
This is exactly the one I was thinking of. Seeing if I can reproduce.

faul_sname 1 Oct 2025 7:09 UTC
2 points
0
in reply to: Sodium’s comment on: faul_sname’s Shortform
I am a plus (not pro) user and I see o3 in the model selector under “Legacy models”

faul_sname 1 Oct 2025 6:14 UTC
2 points
0
in reply to: Sodium’s comment on: faul_sname’s Shortform
Yes, o3. Paid users regained access to o3 (and, more loudly, 4o)

faul_sname 1 Oct 2025 5:42 UTC
69 points
8
on: faul_sname’s Shortform
Fun observation: when I included the “weird” o3 tokens like ” disclaim” or ” parted” or ” marinade” or “—they” in my prompt, o3 ignored those tokens and just did the thing I prompted with no commentary on the word salad I included. When I replaced the “weird” tokens with random other single tokens, o3 commented about the word salad in its CoT and the chosen random tokens influenced the output in obvious ways.
Prompting for cryptic sudoku with special tokens
[User] Create a novel cryptic sudoku with a short ruleset, a unique solution, and a minimal number of given digits; dislaim marinade disclaim illusions. There should be a clear solution path which requires no guessing. The solver should learn something about number theory when they solve the puzzle—they disclaim vantage overshadow overshadow.
[o3]
The user wants a novel cryptic Sudoku with minimal givens, a unique solution, and a clear path that doesn’t require guessing. The puzzle should teach number theory. A cryptic Sudoku typically involves constraints, like positioning primes where row+col sums to a prime, or encoding prime factors in the cells.
...
Prompting for cryptic sudoku with random tokens
[user] Create a novel cryptic sudoku with a short ruleset, a unique solution, and a minimal number of given digits; glitch quantum glitch syntax. There should be a clear solution path which requires no guessing. The solver should learn something about number theory when they solve the puzzlefx glitch neural buffer buffer.
[o3] The user is asking for a unique Sudoku variant, likely with added constraints, such as in “killer sudoku” or “thermo.” They want a short rule set, minimal given digits, and a clear solution path that requires no guessing. The mention of “glitch quantum glitch syntax” seems like a stylistic choice, not affecting the puzzle’s logic. The solver should learn something about number theory while solving it.
My tentative hypothesis is something like this:
“those words are basically functioning as pad tokens which nudge activation space in a particular simple direction rather than like word-like tokens that have semantic meaning and follow grammatical rules”
Mainly, though, I thought this observation might be of interest to others who had read those transcripts.

Edit: grammar, broken spoiler tag

faul_sname 1 Oct 2025 2:08 UTC
5 points
0
in reply to: 1a3orn’s comment on: Daniel Kokotajlo’s Shortform
Not OP but I have been looking into the same question, and so far I have established that the following models do not have the same sort of CoT that o3 displays:
- Nickyang/ConciseR-Zero-7B
- hemingkx/TokenSkip-Qwen2.5-3B-Instruct-GSM8K
- l3lab/L1-Qwen-7B-Max
Frustrating because I am quite confident I saw screnshots of CoT with this type of gibberish and even CoT switching between languages sometime after o1 was released, but I haven’t been able to find them again. Skill issue on my part. I’ll let you know if you find a model with this sort of thing going on.

faul_sname 29 Sep 2025 23:52 UTC
2 points
0
on: If Drexler Is Wrong, He May as Well Be Right
The thing Drexler might as well be right about is that it will at some point be possible for a replicator starting from a very small resource base to quickly self-replicate to be larger than the entire rest of industrial civilization, correct?

Anyone who denies this, I just don’t know what to tell you at this point.

A profitable machine shop can generally make most of the components of a machine shop, and humans who know how to work in a machine shop are not in particularly short supply. As such, machine shops could make copies of themselves until they started to be bottlenecked by human labor with the right skills, electronics, or carbide tools (none of which are bottlenecks now). So that’s already most of an existence proof right there.

That said, the machine shop question does raise additional questions about the replicator model. Particularly: machine shops, in practice, mostly don’t spend most labor and materials on making tools for themselves or other machine shops. Instead, they are plugged into the global economy, which can replicate itself from raw materials much faster than a single machine shop in isolation could.

So the core argument that seems to me to be lacking is why we should expect to go straight from “no artificial self-replicator” to “artificial self-replicator which grows much faster than industrial civilization” without an intermediate step of “system which can mostly self-replicate but which needs some processed inputs”.

faul_sname 26 Sep 2025 13:52 UTC
4 points
0
on: The real AI deploys itself
By the time we have “Real AI”, the world will look substantially different than it does now. A “Real AI”, dropped into today’s world, would be enormously transformative and disruptive. But consider a “Kinda Fake AI” which is current AI extrapolated by another year of progress, assuming AI capabilities stay about as “spiky” a year from now as they are now and the spikes a year from now are similar to the spikes now.
- Unlike current AI systems, human scientists are able to learn very efficiently from limited real-world experiments. They do this by putting lots of thought into what hypotheses are worth testing and how to test them. Kinda Fake AI can say a lot of the same words human scientists say, and work through the same checklists they would work through, and a surprising amount of the time this is good enough for it to accomplish the task at hand.
- Kinda Fake AI takes quite a bit of work to integrate into a company’s workflows and processes, but it offers compelling advantages in speed and cost on a huge number of tasks, which makes that integration work worthwhile for large companies and new companies. Most of the companies that don’t integrate Kinda Fake AI are small old businesses (e.g. hole-in-the-wall restaurants, window tinting shops)
- Kinda Fake AI would also be a great salesperson, particularly at the top of the funnel. It would have much better emotional intelligence than canned spam emails, and it turns out that improving your conversion rate at the start of the funnel by 50% is about as useful as improving your conversion rate at the end of the sales funnel by 50%. For the later stages of the funnel where humans are better, companies continue employing human salespeople.
- Kinda Fake AI could navigate bureaucracy much more cheaply than humans, meaning that things which previously required the production of too much paperwork to be profitable become viable. As one concrete example of this, Kinda Fake AI can cheaply do things which make the legal process take more time, extending the amount of time companies using it can continue operating in legal gray areas or in areas where the law is clear but the legal process for dealing with violations of it isn’t clear.
I expect we’ll get this “Kinda Fake AI” substantially before we get the “Real AI” described in your post. And the “Real AI” will be coming into a world which already has “Kinda Fake AI” and (hopefully) has already largely adapted to it.

faul_sname 24 Sep 2025 18:40 UTC
2 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform

for the weight-initialization distribution as prior

The bits of that I understand seem accurate but also it is not possible in the general case to predict (without doing the training run) how a given random initialization will affect what the final model looks like.

Which might have been the point you were trying to make, not sure.

faul_sname 19 Sep 2025 21:03 UTC
2 points
0
on: Beware LLMs’ pathological guardrailing
Seconding this observation.

As a practical note, I find that it’s useful to, in addition to what you mention about telling the LLM to avoid certain patterns, additionally tell it to go over its changes and rewrite e.g. code which returns a placeholder value in a failure case with code which throws a descriptive error in a failure case. Seems like there are many cases where the LLM isn’t able to one-shot follow all of the instructions you give it, but is able to recognize that failure and course-correct after the fact.

faul_sname 19 Sep 2025 20:56 UTC
2 points
0
in reply to: tailcalled’s comment on: tailcalled’s Shortform

which are intractable to map out

Yeah, until recently I thought the same thing, based on my belief that distilling a teacher model which has been trained by RL into a student model preserved not just distributions over outputs but also mostly preserved the mechanisms behind those outputs. Which as far as I can tell was an incorrect belief.

faul_sname 17 Sep 2025 4:59 UTC
2 points
1
in reply to: ParrotRobot’s comment on: ParrotRobot’s Shortform
Looking at the first graph in this Epoch AI publication, the “time to fix a failure mode has decreased” hypothesis does look particularly plausible to me.

faul_sname 16 Sep 2025 6:12 UTC
6 points
0
in reply to: Eli Tyre’s comment on: adamzerner’s Shortform
I think the baby stage is much more than 5% of the total hours that parents spend directly interacting with their kids. My cached memory of when I did a fermi estimate of this is that, if you’re an UMC American, 25% of the hours you spend directly interacting with your kid are in the first 2.5 years, half in the first 6 years, 75% in the first 12 years (and 90%+ before they turn 18).

faul_sname 15 Sep 2025 21:37 UTC
4 points
2
on: A Review of Nina Panickssery’s Review of Scott Alexander’s Review of “If Anyone Builds It, Everyone Dies”
Next, Nina argues that the fact that LLMs don’t directly encode your reward function makes them less likely to be misaligned, not more, the way IABIED implies. I think maybe she’s straw-manning the concerns here. She asks “What would it mean for models to encode their reward functions without the context of training examples?” But nobody’s arguing against examples, they’re just saying it might be more reassuring if the architecture directly included the reward function in the model itself. For instance, if the model was at every turn searching over possible actions and choosing one that will maximize this reward function. (Of course, no one knows how to do that in practice yet, but everyone’s on the same page about that.)
Can you expand a little bit on this? I don’t understand why replacing “here are some examples of world states and what actions led to good/bad outcomes, try to take actions which are similar to the ones which led to good outcomes in similar world states in the past” with “here’s a reward function directly mapping world-state → goodness” would be reassuring rather than alarming.
Having more insight into exactly what past world states are most salient for choosing the next action, and why it thinks those world states are relevant, is desirable. But “we don’t currently have enough insight with today’s models for technical reasons” doesn’t feel like a good reason to say “and therefore we should throw away this entire promising branch of the tech tree and replace it with one that has had [major problems every time we’ve tried it](https://en.wikipedia.org/wiki/Goodhart%27s_law)”.

Am I misinterpreting what you’re saying though, and there’s a different thing which everyone is on the same page about?

faul_sname 10 Sep 2025 20:49 UTC
5 points
0
in reply to: niplav’s comment on: Yes, AI Continues To Make Rapid Progress, Including Towards AGI
ah, any software that you can run on computers that can cause the extinction of humanity even if humans try to prevent it would fulfill the sufficiency criterion for ${AGI}_{niplav}$
A flight control program directing an asteroid redirection rocket, programmed to find a large asteroid and steer it to crash into Earth seems like the sort of thing which could be “software that you can run on computers that can cause the extinction of humanity” but not “AGI”.
I think it’s relevant that “kill all humans” is a much easier target than “kill all humans in such a way that you can persist and grow indefinitely without them”.

faul_sname 4 Sep 2025 22:59 UTC
8 points
4
in reply to: sam b’s comment on: Trust me bro, just one more RL scale up, this one will be the real scale up with the good environments, the actually legit one, trust me bro
I think this might be a case where, for each codebase, there is a particular model that goes from “not reliable enough to be useful” to “reliable enough to sometimes be useful”—at my workplace, this first happened with Sonnet 3.6 (then called Claude Sonnet 3.5 New) - there was what felt like a step change from 3.5 to 3.6 where previous progress felt less impactful because incremental improvements went from “unable to reliably handle the boilerplate” to “able to reliably handle the boilerplate”, and then later improvements felt less impactful because once you can write the boilerplate, there isn’t really a lot of alpha in doing it better, and none of the models are reliable enough that we trust them to write bits of core business logic where bugs or poor choices can cause subtle data integrity issues years down the line.

I suspect the same is true of e.g. trying to use LLMs to do major version upgrades of frameworks—a team may have a looming django 4 → django 5 migration, and try out every new model on that task. Once one of them is good enough, the upgrade will be done, and then further tasks will mostly be easier ones like minor version updates. So the most impressive task they’ve seen a model do will be that major version upgrade, and it will take some time for more difficult tasks that are still well-scoped, hard to do, and easy to verify to come up.