The Bourne Ultimatum star recently gave in to the lure of Hollywood and moved to California, where he now lives on the same street as longtime friend and fellow Bostonian Affleck.
He's convinced he'll never become a target of the media because his life isn't very interesting, but Affleck has offered him a few tips of how photographers operate - and now he isn't so sure.
Damon tells Wenn, "There are always five to six cars sitting outside Ben and Jennifer's driveway. These people's full-time job is to shoot Jennifer because she moves magazines in the Midwest.
"Ben explained it to me one night: 'You've never heard of any of these magazines. Women are interested in her as a mother. They want to know what she's up to, what she's dressing the kids in and that completely changes everything for them. These people are literally assigned. They're happy to sit there all day long. You can't outwait them because their job is to follow you.'
"I don't think it'll get bad for us, but, if it does, I'll turn around and go home!"