The Irish star had hoped to return for a fifth Bond film, but talks between his agent and producer Barbara Broccoli broke down while he was filming sex scenes in the Bahamas.
He recalls, "I'm about to do scene 58 with the lovely Salma Hayek, where we roll around on the beach naked and talk about some silly diamond, and the boys are telling me that the negotiations have stopped.
"When the message was delivered, it was a body blow. I said, 'What does that mean?'"
Broccoli and her fellow producers agreed to discuss her decision with Brosnan over the phone.
He adds, "I heard what they had to say. They had obviously made their minds up and I was not there to change their minds."