The X Factor host described her relationship with the Prince as "unlikely", but reveals the pair was set up by a mutual friend.
"To meet a prince is so unlikely it would be weird not to acknowledge it," she writes in her book Storm In a C Cup. "However, once the story got out, that was it. We had to stop seeing each other.
"I was no longer Caroline Flack, TV presenter, I was Caroline Flack, Prince Harry's bit of rough. The thing about Prince Harry is that he has no choice. He's not some egotistical rock star who wants loads of attention. That is his life. He didn't ask for it, he just has to deal with it."
Despite facing some backlash after being photographed with the Prince, Caroline insists she was not prepared for the storm that erupted when it was revealed she was dating Harry Styles, who was 17 at the time.
The presenter was 31 when she dated the singer and admits she felt flattered by the attention from a younger man.
"I already knew that he had a crush on me, he'd made it pretty obvious. He'd said it in magazines and he'd said it to friends," she writes. "It was flattering and I found it amusing. He was a kind of cheeky man/boy with a captivating smile who got what he wanted."
Caroline insists she thought nothing of the age gap, but the pair decided to keep the relationship as a "bit of fun".
"Although it was just a bit of fun we decided it was best to keep it to ourselves, as we were both working for Simon (Cowell). We were both single, we got on well and we had a laugh."
However, the problems began after the One Direction singer was photographed leaving her house. "Once that was out, it was open season. After that anyone could say anything. In the street people started shouting at me 'paedophile' and 'pervert'", she adds.
"A One Direction fanzine had me as a voodoo doll, with arrows (pins) pointing to bits of my anatomy. None of us expected it to turn so vicious and truly nasty. There was a moment when we just thought, s**t, this has gone really dark. So perhaps we did the wrong thing, who knows."
Caroline was also offered $1.6 million (£1 million) to be the face of Cougarlife.com, a website aimed at older women dating younger men. "I wasn't even old enough to be on the site. You had to be 36 and I had just turned 32," she writes.
The book, Storm In A C Cup, is due to be released in the U.K. on 22 October (15).