The TV mogul took the 'Promise This' singer with him from the UK to the new US edition of the show, but she was left "shocked and stunned" after bosses at TV network Fox gave her the axe after just four days.
Cheryl had reportedly planned her whole year around the show, but is now back in the UK and left devastated.
A friend told the News Of The World newspaper Cheryl had said to them: "I never actually wanted to do the American show, I'm not a TV star. I had so many reservations, but he pushed me into doing it. He told me I'd be a star over there.
"And then he didn't give me a chance to prove myself. I was gone after four days. I have no idea what I'm going to do now. My life has been turned upside down. They have treated me really badly."
Simon said yesterday (28.05.11) he feels "guilty" over what has happened, but insisted his "hands were tied" by executives at Fox.
Cheryl is currently in London with her mother, and said to be refusing to take calls from Simon about returning to the UK version of the show. She has told friends she may even take a break from all TV work.
The source added the Girls Aloud star had said: "A big part of me wants to disappear. I Love my family and friends. I'll be ok without working for a while.
"'X Factor' isn't my world. It's not the be-all and end-all of my career."
She said: "All I can tell you is that she is lovely and I was really excited about the relationship and Friendship we were forging.
"And regardless I am really looking forward to continuing that. She's wonderful."