My heart aches, a lot, when I hear those words from your mouth. U may say it unintentionally but it hurts me a lot. 

I'm doing everything good for my baby too. I would never want to hurt or do something that harms him. I carried him in my tummy for 9 months and suffered so much pain delivering him. I will never want to hurt him. 

But if you really think that he belongs to your family only, I have nothing else to say. 

I love you both a lot and never would want this to affect our relationship at all.