The 'Bonfire Heart' hitmaker, who was privately educated at the exclusive Harrow School in north west London, thinks the Blur frontman shouldn't cover up his upper class roots like others in the music industry.
James told ES Magazine: ''I was just the only one who didn't hide my accent. I mean, come on, Damon Albarn, he's right up there. He's got an orchard full of plums in his mouth and a silver spoon stuck up his a**e.''
Damon is a state-educated son of an academic and both artists have been criticised in the past for being ''too posh for pop.''
James added: ''Britain is so obsessed with class. It really needs to let go, because no one else gives a sh*t.''
Meanwhile, Damon is set to release his first solo album 'Everyday Robots' on April 28, and waited 25 years to put out his own record, which he thought might never be recorded.
He previously said: ''You could say that I'm embarking on a new stage in my life, but really I could turn around next time and do something else. So it's not a given that I've become a solo artist.''
The BBC drama starring Aidan Turner returns to BBC One on September 4th.