I've heard this so many times as well. And the fact is they probably don't have black friends. It's a nice getout clause to avoid embassessment when people let a comment slip.
Great take on the subject.