The 23-year-old actor was linked to the 'Red' hitmaker last year when she attended a play he was starring in after she rehearsed for 'Les Miserables'.
However Eddie found it ''offensive'' when the press claimed he ''dumped'' her as the whole thing was made up from one brief meeting.
He explained: ''It was absolute nonsense. We had a chat behind the stage for seven minutes. So I met Taylor Swift for an hour and seven minutes. Even by my standards that's a pretty brief relationship!
''I find it offensive because it was I've 'dumped' her. You're always told the press make stuff up, but when it's about you, it's extraordinary.
''Then you get emails from your friends going, 'Dude, is it true ...?' 'No!' Then they don't believe you.''
The British heartthrob instead remains quiet about his relationship status - especially after spending time with Kristen Stewart after she was caught cheating on her boyfriend Robert Pattinson with her 'Snow White and the Huntsman' director Rupert Saunders.
He added: ''It's something I try to keep protected. I've done press with Kristen Stewart and it's hard.''
Eddie has starred alongside actors such as Angeline Jolie in 'The Good Shepherd', Natalie Portman and Scarlett Johansson in 'The Other Boleyn Girl' and Michelle Williams in 'My Week With Marylin', however he remains modest and doesn't look at the film industry in a ''fanatical'' way.
He added to Glamour magazine: ''I'm not a fantasist. For everything I've done that's worked, I've done thousands of films that have never seen the light of day.
''I just quite like making a living out of it, y'know? I just want to do more. That's all I can ask for.''