This has never been an issue for me.
Hope I'm not coming off as rude, but does this mean it's not an issue for you in particular - or is the fan really not audible during really quiet passages (like objectively)?
I have read very contradicting reviews on this and this would be a real deal breaker, as we have really quiet acoustic passages and an audible fan would really kill the atmosphere...