for me it starts to get feeling bad at around 8 to 10 ms, thats for example when playing a softsynth the keys feel like sticking to the fingers
playing guitar with a latency of about 10 ms is also feeling "sticky" for me
with rapping or singing at about 5 meteres from the speakers it gets dificult (at least for rapping)
but there seems to be a sort of threshold, meaning that from 0 to about 8 ms is don´t feel any latency, but beyond that time: boom, its there