Some (actually, Much) additional nuance about empathy and its components that might add some context to this discussion at this link: https://www.lesswrong.com/posts/SMziBSCT9fiz5yG3L/notes-on-empathy
In particular, as other commenters have already pointed out, there is something about “perspective taking” (the shift from considering how you feel about what they feel to considering how they feel) that may be resulting in the OP’s frustration.
Some (actually, Much) additional nuance about empathy and its components that might add some context to this discussion at this link: https://www.lesswrong.com/posts/SMziBSCT9fiz5yG3L/notes-on-empathy
In particular, as other commenters have already pointed out, there is something about “perspective taking” (the shift from considering how you feel about what they feel to considering how they feel) that may be resulting in the OP’s frustration.