I think this is attitude is incredibly common among a lot of sports and hobbies outside the mainstream. In the US at least, the significant popularity of basketball, football, etc. over rec-league-only sports such as ultimate frisbee, quidditch, etc. means that the mainstream sports are a much stronger sieve to filter out genuine talent and skill. Perhaps consider that part of the reason you learn to play the digiridoo is that its way easier to become one of the comparatively best digiridoo players in your community than violinist. Consider also that this is why mastery of a mainstream skill is such a strongly sexy trait—because it’s a signal that‘s difficult to fake, unlike mastery of less competitive skills.
Now there is definitely much more value in finding interests which you genuinely enjoy and have a comparative advantage than ones that are merely different. But I think that what a lot of people are thinking on some level “is this something I could be good at?” Where “good” can only really be determined by looking at the skill cap of others around them.
The trick to beating this anxiety of course is realizing that the only thing about your choices that matter is your own opinion of them, not others‘ perception of your choices.