The 'Britain's Got Talent' judge reveals one of the most Common questions he gets asked is whether his boss prefers men or women.
He said: ''I have never done anything where people are more interested. The main question I get is, 'Is Simon gay?'
''Even my mum was going, 'Yeah he is really nice - is he gay?'''
After tweeting a picture of himself and Simon with the caption, ''Me and my boyfriend'', the 'Little Britain' star also told how Simon covers up for his own tardiness by pretending David is busy eating.
He said: ''We don't tend to start auditions until 4pm. Simon is a bit late and then he goes in front of the audience and says, 'The reason we are starting late today is because David is eating a three-course meal'.
''The audience are booing me because that's his idea of a joke.''
Having returned to the show following a year's hiatus to launch the US 'x factor', Simon says he is unlikely to make a similar return to the British 'X Factor'.
He explained: ''It's impossible. I would like to but at the moment they run at exactly the same time. But I do miss not doing it.''