The music mogul took his pal Cole with him when he launched his hit talent search in the States in 2011 and hoped she could repeat her U.K. role as a popular mentor and judge.
However, within three weeks, Cole's American dream had crumbled and she left the show amid fears she would not connect with U.S. viewers.
Now Cowell has admitted her departure was sparked by an argument over a weird purple and orange outfit the pop star wore for her U.S. X Factor debut.
In an interview with Radio Times, he says, "There was so much expectation on the show in America, and I definitely wasn't myself. I was uptight. When I saw her I didn't recognise her. Literally. The hair and outfit were crazy, just totally crazy.
"I asked her manager, 'What's she wearing, what's happened to her hair?' He was defensive about it. I haven't a clue why she turned up like that.
"We then did the first day of auditions, and she was nervous, shy, not in her comfort zone. She'd put on a bit of weight, but not loads. That wasn't the issue. The only way I can describe it is that it wasn't Cheryl sitting in that seat. She'd lost her confidence...
"(Later) I called her and said, 'Look, it's not going great, to be honest,' then I mentioned what she wore and she turned on me. If I could go back in time I would have kept her on the show. It was stupid doing what I did. I didn't sack her personally. That was the second biggest mistake I made."
Cowell and Cole have since made up and put the incident behind them, and the former Girls Aloud singer is even trying to convince him to marry Lauren Silverman, the mother of his son Eric.
Cole tells the publication, "I'm going to have a word with him about getting married. He jokes about being too tight (miserly) to get married but I don't think he is. He's very generous."
In 2012, Cole sued producers of The X Factor Usa for $2.3 million (£1.3 million) over her sacking and received undisclosed damages as a settlement. She has since rejoined the U.K. version of the programme.