The music mogul admits he has been bombarded with requests by celebrities desperate to sit on the panel with him when the star-search contest launches in the US this year and singled out the 'OMG' hitmaker and his former fellow 'American Idol' star as current contenders for the jobs.
He said: "I can't make up my mind. I've been meeting all these people every day and I like them and at the moment I could end up with about 23 judges on this panel.
"One contestant, 23 judges!"
When asked about Usher's rumoured role by TV talk show host Jay Leno, he said: "Maybe. Yeah I like Usher. He's good."
When asked who else is in the running, he confirmed "Paula Abdul," before jokingly adding "Charlie Sheen."
Simon, 51, has also said he has had talks with singers Jessica Simpson and Shania Twain about the show, but also ruled some celebrities out of the running, including P. Diddy as he is "too expensive" as well as Christina Aguilera.
When asked about the 'Burlesque' singer - who gave a shambolic performance on the final of the UK version of the show last year - he simply replied: "Absolutely not".
'The X Factor' is set to start in the US this autumn.
With the passing of each decade, the music industry is constantly set alight by the...
Filmmaker Spurlock (Super Size Me) is known for his investigative wit, but there's none of...
British pop heartthrobs One Direction became a worldwide sensation when they were thrust together on...