The former star of How I Met Your Mother, which also aired on the Cbs network, has revealed he was at the top of the shortlist to take over for the comedian, who is set to leave his post in 2015.
In a recent interview with shock jock Howard Stern, Harris admitted he met with Cbs chairman and CEO Les Moonves to discuss the possibility of the new gig, but ultimately said no due to the routine nature of a nightly job.
Harris said he would "get bored" of the format, adding, "I don't have any interest in doing 'monologue, commercial, sketch, guest, guest musical act, goodnight."
He also divulged that Moonves attempted to get him to take over from Scottish funnyman Craig Ferguson, who is also leaving his position on The Late Late Show, but again Harris declined.
He said, "It's still nightly. You're still coming up with the same content and now you're just getting bitter that no one's watching it."
While Moonves and fellow Cbs executives have already chosen Stephen Colbert to replace Letterman come next year (15), Harris told the boss that he would prefer to host a weekly variety show for the network.
He added, "I think if it's weekly and people really want to see it - and you trust that there's a lot of really great s**t on there - then you're going to get the guests to want to come on your thing.
"A weekly thing, you'd have more time to prep for stuff. You could do pre-taped stuff that would be really exciting, you could flesh it out a little bit more and have more acts."
Harris is currently taking a brief break from Tv to star on Broadway's Hedwig and the Angry Inch, for which he was recently nominated for his first Tony Award.