>I’ll have an example that I personally experienced recently, omitting some details related to my PII.
Based on this example, could you compare your expected outcome with the real outcome? In numbers. For example, “I expected to finish this refactoring in two weeks, but it took two months” or “I expected to finish this refactoring in two weeks, but my colleague completed the same task in two days.” Something like that.
I have more than 15 years of experience in software development, not precisely in data processing but in gamedev and backend (and sometimes in data processing), which is pretty close.
From my experience, most people are struggling with the task you described. Even senior developers with 5-10 years of experience. I saw only a few people, who could do such refactoring naturally and from the first try.
I may suggest a few things:
- Try to reevaluate your expectations. Maybe you are too hard on yourself. You may overestimate the required result and try to do more than you need.
- Research a few methodologies of development. I may suggest Test Driven Development and Pair Programming. The first may help you control changes easier, and the second may help you learn how other people solve similar tasks.
- Try to pay more attention to code decomposition and structuring when you work on “pipeline prototype”. Aka, add some “fake” functions/objects that will be required later, but for now, do nothing or contain trivial logic. Sometimes, it is easier to see what you need to do when you are fully in the task’s context rather than when you are refactoring your code.
- Do not try to implement all features at the same time. Choose one, implement it, cover it with some tests or types or both, and then move to another.
Yep, these conclusions intersect with the prognosis I made for myself at the end of 2024:
- There will be no technological singularity.
- Neural networks won’t change conceptually over the next 10 years.
- We won’t build strong AI based on just one or a few neural networks.
- Neural networks on their own won’t take jobs away.
- Robots will not rapidly and massively replace manual labor.
- Works of proactive professionals are safe for the next 10 years.
My predictions are based on:
- The view on the current neural networks as on one-more-building-block-from-many, essentially a probabilistic database — not more but not less.
- The assumption that businesses/governments/money will focus on optimizing the current achievements, rather than on continuing risky experiments.
A much longer post with explanation is in my blog: https://tiendil.org/en/posts/ai-notes-2024-prognosis