Pushing a fat man onto the tracks to save several lives is generally considered to be immoral because you are USING a person to achieve some goal.
In your case, you are only using the trolley containing the man to stop the death of four people. You are NOT using the man because the trolley would work regardless of whether or not he is present. Thus, it is mere misfortune that he is present and killed—exactly as if he were on a siding where you diverted a train to save ten people.
Pushing a fat man onto the tracks to save several lives is generally considered to be immoral because you are USING a person to achieve some goal.
In your case, you are only using the trolley containing the man to stop the death of four people. You are NOT using the man because the trolley would work regardless of whether or not he is present. Thus, it is mere misfortune that he is present and killed—exactly as if he were on a siding where you diverted a train to save ten people.