The music manager reckons the band have ''got too big and don't listen to anybody,'' which is why they have started having public spats such as Louis Tomlinson's recent Twitter row with former member Zayn Malik.
Speaking about the band's manager, Simon, the former 'X Factor' judge said: ''I think he's lost control of them and that's what happens with bands. They get too big and don't listen to anybody and then it's all over, and then they think 'why didn't we listen?'''
The latest row saw Louis Tomlinson, 23, post a veiled insult on his Twitter account mocking Zayn's use of photo filter, to which he replied: ''Remember when you had a life and stopped making b****y comments about mine?''
And while no action seems to have been taken, Louis Walsh - who has previously looked after the careers of acts including Boyzone and Westlife, and currently manages Union J - said he would have given the argumentative member a ''good b******ing'' to reprimand him if he was their manager.
He told the Daily Mirror newspaper: ''I would have given them a good b******ing - they would know never to do anything like that. I would probably sack them.
''You don't air your dirty laundry in public. Simon made this band. They would be nowhere without Simon Cowell. Let's get that straight. He put them together, he pressed the buttons, he made it happen. Without Simon they'd be nothing.''