The Jonas Brothers star - who dated the 'I Knew You Were Trouble' hitmaker in 2008 - quipped about his former flame's unlucky dating history in an interview with Joan Rivers.
The comedienne said: ''Joe, you're the one who dumped Taylor Swift?''
The 24-year-old star responded: ''I think it's me and a few others, but yeah.''
The country singer, who has a penchant for penning songs about her heartbreak, is rumoured to have written three songs about her ex-boyfriend.
Joe's brother Kevin joked about the musical tributes: ''We still haven't seen the royalty check. It's weird.''
Taylor, 23, infamously revealed that Joe broke up with her in a 27 second phone call in an interview in 2008.
She said: ''It was, like, a record, I think, for how quick. I looked at the call log - it was like 27 seconds. That's got to be a record.
''It's all right - I'm cool. You know what, it's like, when I find that person that is right for me, he'll be wonderful When I look at that person, I'm not even going to remember the boy who broke up with me over the phone in 27 seconds when I was 18.''