I feel guilty because I didn't say no to a guy's advances

I was in a situation with a guy. He's my brother's friend and kind of like a brother to me. He invited me over to hang out, and then came on to me. I was instantly terrified to say no, and I gave in. I don't know what's wrong with me. I can't figure out why I'm afraid to say no. This is giving me a lot of guilt and depression.