Question about going bra-less?

I've seen some of my friends where a tank or a shirt without a bra. Most can even wear a sports bra with no padding and be fine. I can't because, well, I think the term goes like "it looks like I'm smuggling raisins." It doesn't seem like that all the time, but I can't pick the times that it does. It doesn't even have to be cold. So is it uncommon to have this problem?