Ben Affleck's ''biggest fear'' is only seeing his children on the weekends.

The 'Gone Girl' star, who recently split from Jennifer Garner after 10 years of marriage, reportedly wants to spend as much time as possible with their three children, Violet, nine, Seraphina, six, and Samuel, three, because ''he didn't grow up with a present father.''

A source close to the 42-year-old actor, who was raised by his single mother Christine after she split from his alcoholic father, said: ''Ben wanted children and a family his whole life. He didn't grow up with a present father, so his biggest fear was being the dad that only saw his kids on the weekends. He never wanted that.''

The Oscar winner, who escaped to the Bahamas with his estranged wife and their children last week, flew to be with them in Atlanta after a brief trip to Los Angeles for a meeting in recent days.

The source said: ''The minute he could get on a plane to see his kids, he did. It's important. He wants them to be stable.''

But it seems he has little to worry about, as he and Jennifer both have close bonds with their children.

A friend told PEOPLE magazine: ''[Ben] was a family man when he could be. And his kids adore him. He travelled a lot, but when he is with the kids, he's really with them. And there is nothing more important to Jen than being a good mom.''