That's unusual. I don't know anyone who likes that in the way you do.
I wonder if it goes back to your childhood. Did your Mum have friends round and they used to gossip together in front of you? If that was a good memory it might make you feel warm and secure to hear women whispering.
Of course, I could be so wrong! Just wondered, though.