The Oscar winner teams up with Rachel MCAdams and Javier Bardem in the upcoming romantic film, but he confesses he only took the part for a chance to pick up directing tips from Malick - as his character hardly speaks.
He tells GQ magazine, "The experience of it seemed half-crazy in that we didn't really have dialogue, so I didn't really know what was happening. Then I realised that he was accumulating colours that he would use to paint with later in the editing room.
"My character doesn't really do that much. It was kind of a wash for me in terms of learning something as an actor, because Terry uses actors in a different way - he'll (have the camera) on you and then tilt up and go up to a tree, so you think, 'Who's more important in this - me or the tree?' But you don't ask him, because you don't want to know the answer."
And Affleck admits the movie, which is due to be released next year (13), may not to be everybody's taste, revealing it got panned at the Venice Film Festival in Italy this autumn (12).
He adds, "I understand it got booed quite thoroughly at (the Venice Film Festival). Honestly, you have to want to see that movie - there's hardly any talking. It's a tone poem. If you don't want to see that, you should not go."