Simon Cowell didn't want to mentor the Overs again on the ‘X-Factor’ this year, so Sunday night’s edition of the Six Chair Challenge was particularly tough for the talent show judge. Faced with picking which six contestants he would take to the judges’ houses, Cowell was indecisive and at times ruthless, as he whittled down the remaining contestants.

Simon CowellSimon Cowell had a tough time during the ‘X-Factor’ six chair challenge on Sunday night.

Holly Johnson, a 27 year old estate agent was the first to get the axe, following her rendition of Pharrell’s ‘Happy’. "This is very easy - it's no,” Cowell said, leaving new judge Nick Grimshaw shocked.

Max Stone was the first of the Overs to win a seat, after his performance of ‘Turn Your Lights Down’ by Bob Marley. But even after choosing the 27 year old Cowell still told him, "I think your voice is fantastic but I just don't think people are going to like you very much..You’ve got a very shut down personality.”

More: Simon Cowell Confirms Judging Panel For 'Britain's Got Talent's' Tenth Series

Chairs were also given to Vicki-Ann Nash, Joseph McCaul and Stephanie McCourt, but hopefully they didn't get too comfortable as all three would be unseated before the night was over. Kerrie-Anne Phillips was the next contestant to secure a seat, with Simon telling her, “I can see something in you which maybe you haven't seen yourself yet. Potentially you are fantastic.”

Then came Anton Stephans who gave as good as he got from the grumpy judge. After his performance of Luther Vandross hit 'A House Is Not A Home', Cowell told him, “My problem with you is that you have so many weird facial expressions.”

More: Rita Ora Picks Her Girls During 'The X-Factor's' Six Chair Challenge

But the 44-year-old Londoner quickly fired back, telling Cowell, “When you live in a world in which Botox is normal, facial expressions seem unusual. But if, like me and my friends, you can't afford it, your face moves.” Even after that Stephans still managed to gain a ticket to the judge’s house.

Also going through to the next round are 37-year old Bupsi from Leeds, 34-year old Jennifer Phillips and Ebru, a 27-year-old student from London who sang Beyonce's 'If I Were a Boy'. All six contestants are now off to Cowell’s ‘house’ where Louis Walsh will be on hand to help pick which contestants make it through to the live shows.