I dunno, complaints about instruments seem pretty legitimate to me. I mean, isn't that what most WIP Remix criticisms are about? "Hmmm, you've got a great rhythm going, but the drums that you're using sound kind of cheap and odd, how about going for a deeper drum sample?" A voice is an instrument, so it's open to the same criticisms that you can make to any other kind of instrument.
Along these lines, having issues with songs that have voice shouldn't seem that outrageous. Some people place more importance on the basic parts of a song. Personally, I feel that lyrics can somewhat limit my listening enjoyment because I usually like to kind of play around with the song in my head while I listen to it (such as adding another beat, or perhaps even humming my own really really random lyrics). Since the lyrics are already there, it's hard to listen to this song the way I usually do.