The 'Lego House' hitmaker has narrowed down 50 prospective tracks to 12 for the follow-up to 2011 debut '+', but he explained the pair prefer to relax when they have time off rather than writing all the time.
He told MTV: ''There have been no talks with Taylor about writing tunes, because when we both have time off we just want to have time off.
''And she's got a beautiful new house, which is always fun to hang out in. I don't want to pick up a guitar and be like, 'Hey, let's write another hit song'.
''I just want to chill. She's got a pool table. I quite like that.''
The two friends duetted on 'Everything Has Changed' from Taylor's last album, 'Red', and they have both previously said they plan to work together at some point in the future.
As he concentrates on his second album, Ed has also dashed any rumours of hip-hop producers such as Timbaland working on songs with him, insisting fans just want to feel the emotion in his music.
He added: ''I don't think anyone wants to dance to the sorts of words I have to sing. They just want to cry and eat ice cream.''
The '12 Years A Slave' director will receive the accolade at the London Film Festival in October.
Critics from all over the world were asked to name the best movie of the past 16 years.
Bridget has always known how to get herself into a muddle - catastrophic muddles at...
After battling the dating scene and finally finding love with Mark Darcy, Bridget Jones is...
In July of 2015, Ed Sheeran played his biggest gigs to date, selling out Wembley...