Let me ask you this: do you ever wish that he would interrupt you more? Do you feel like his politeness indicates he's uninterested in what you're saying? No? Then you know it's rude and doesn't indicate friendly interest.

You simply have other friends who have learned to be rude in self-defense and don't mind doing it.