The Spanish actor - who is now married to Melanie Griffith - was flattered when the pop superstar declared a romantic interest in him early in his career, but insists that nothing would ever have happened between them.
Speaking to Harper's Bazaar magazine, he said: "It was kind of flattering when somebody said, 'She likes you, a lot.' I said, 'Oh, really? I've no idea what I should do about this, I mean, I'm married [to first wife, Ana Leza].' So, nothing happened.
"I was afraid of it at the time, because she was a very powerful woman. I didn't want to be Madonna's boy. For good or for bad, I just wanted to have my own career."
Antonio was introduced to Madonna by his friend, movie director Pedro Almodovar and says he "flipped out" when they first met because he was so overwhelmed by her fame.
Recalling their first meeting, he said: "'Antonio,' Pedro said, 'She would love to get to know you.' I couldn't speak English at the time.
"She said a lot of things at the table when we were having dinner, I couldn't understand s**t. I was just flipping out that I was sitting down close to Madonna. She was very famous already, and we were just local people."