Or as another friend of mine put it in a post that, ironically, was almost right next to yours on my friends page:

As an Obama pick, the left seems to want to defend him on the grounds that an extramarital affair is a personal matter. Which it totally is, unless you are *hiding* it, in which case, it is a potential wedge for blackmail with the guy holding the keys to the spook vault.

So I think that's a big part of it: secret affair = potential for blackmail.

But also, the American public just has a bizarre fascination with the sex lives of famous people...