Confirmation bias suggests engineers will tend to overestimate the benefits and underestimate the costs of projects they initially think are good ideas. The head project engineer I spoke with described a common mindset thus,
‘And this is why we tend to focus on the goodness and diminish the badness of projects. We know they are good so all we need to do is prove it to get the approvers bought in. Then the project approvers over time notice that these projects returns are lower than expected so they say, “Let’s raise the bar.” But guess what? The bar never rises. Why? Because we still know what the good projects are and all we need to do is prove they are good.’
This calls to mind Archibald Putt’s “Fifth Law of Decision Making”:
Decisions are justified by considering benefits to the company. Decisions are made by considering benefits to the decision maker.
In other words, the engineer has an obvious incentive to get his project approved regardless of whether or not it is good: if it gets approved, he gets more money.
And indeed, you specifically mention this:
Bad incentives: engineers may often be rewarded for spearheading projects and not punished commensurately if the project is not beneficial so that they often expect to be rewarded for spearheading a project even if they don’t expect it to be a success.
This calls to mind Archibald Putt’s “Fifth Law of Decision Making”:
In other words, the engineer has an obvious incentive to get his project approved regardless of whether or not it is good: if it gets approved, he gets more money.
And indeed, you specifically mention this:
Everyone on LW really should read Putt’s Law and the Successful Technocrat; it’s the definitive guide to the social dynamics of large corporations.