The British royal attended a live screening of the U.K. version of the show in 2009 with his then-girlfriend Chelsy Davy and was invited backstage by the programme's boss Cowell.
Once inside the music mogul's dressing room, the prince unbuttoned his shirt and relaxed on the floor with a glass of wine as a bevy of beauties partied around him.
Singer Cheryl Cole, who was a judge on the show at the time, reveals in her new memoirs how she walked in on the wild party to see Cowell beaming with delight.
She writes, "I walked in and the scene was mental. There were women partying everywhere and Simon was grinning like the Cheshire cat. Prince Harry was lying on the dressing room floor, with his top buttons undone, relaxing with a glass of wine in one hand and a cigarette in the other."