Why is it wrong for a man to love children?

My brother thinks I am weird because I am affectionate to my one year old nephew and not distant like him. I like to hold him which he considers womanly. What is wrong with him?