I know a couple guys that pull at least one ear out for this reason.
*For me* - I'll take the tradeoff to keep my hearing as long as possible.
I've heard ( pun intended) that pulling one out is even worse for your hearing ( over compensation or something like that).
I agree that the right way to do it is have IEM's in and actually once you get used to them they are great. What I do like about IEM's is the consistent mix, you can hear everything all the time (particularly when you go a wandering..). Hence why I'm stupid to keep pulling them out...