My ex-boyfriend wouldn't let me have my daughter because I didn't know my stepdaughter's pick-up details

I am constantly having problems with the same two people who will always be in my life. I had a daughter with my ex-boyfriend. I am now married, and my husband's ex-girlfriend is involved with my ex-boyfriend. They also have a daughter together. My issue is that there is always drama. I am pregnant, and I told my ex-boyfriend that I don't want any drama or arguments. I want to get along as much as possible, and he agreed. However, we just had an incident where my ex-boyfriend started discussing drop-off details about my stepdaughter. I told him that he needed to ask my husband because I can't make decisions about my stepdaughter regarding the matter. That led to an argument. I told him all my concern is when I pick up my daughter. My stepdaughter's pick-up details are between my husband and his ex-girlfriend. I especially told him I didn't want to be involved. Somehow, he turned it around and then wanted to change the schedule we agreed on. He threatened me and got ugly because I wouldn't discuss my stepdaughter's matters with him. The point is there is so much drama. I try my best to get along with everyone. I don't understand where I went wrong (besides replying back to his question). I feel like I'm going crazy because this is a constant battle where everyone's frustrations are taken out on each other, and it's the children that are hurting. I had a party planned for my daughter's birthday, and my ex-boyfriend told me to cancel those plans because he wouldn't let me have her. In my eyes, it's the child that is hurting. I was throwing a party for her birthday, and because of the problem with stupid pick-up details about my stepdaughter, which I have no control over, he took it out on our daughter.