The 35-year-old TV presenter and the One Direction star got together after they both appeared on 'The X Factor' and Caroline has revealed the extent of the abuse she received when, aged 31, she started dating the 17-year-old singer.
In her new autobiography, 'Storm in a C Cup', Caroline said: ''I already knew he had a crush on me, he'd made it pretty obvious. He'd said it in magazines and he'd said it to friends.
''It was flattering and I found it amusing. He was a kind of cheeky man/boy with a captivating smile who got what he wanted.''
Initially, Caroline didn't think anything of the age gap, saying that in the entertainment industry ''nobody asks what age you are''.
She reflected: ''I've never felt that I was much older than Harry anyway. I still feel 18 and probably act that way half the time - all those years in kids' telly probably didn't help.
''Although it was just a bit of fun we decided it was best to keep it to ourselves, as we were both working for Simon (Cowell).
''We were both single, we got on well and we had a laugh.''
However, when their relationship became public knowledge, things took a dramatic turn for the worst.
In her book, serialised in The Sun newspaper, she said: ''It began to go wrong when Harry was pictured coming out of my house one morning. And once that was out, it was open season.
''After that anyone could say anything. In the street people started shouting at me 'paedophile' and 'pervert'. A One Direction fanzine had me as a voodoo doll, with arrows (pins) pointing to bits of my anatomy.''
His new album Underneath It All is out now.