You should read Steven Horwitz on PC libertarians. This, for me, is the money quote, taken from a site that doesn't like us cosmopolitans:
Political correctness is a very strong signal of statism. In the mind of a statist, something is either required or banned. Either homosexual behavior is banned or it is required that everybody respect homosexual behavior.
The irony is breathtaking. I'm still turning it around in my mind, admiring its perfectly smooth and impermeable surface.
The author of that quote is saying, "I have a right to decline to associate with blacks or homosexals." And he is saying this in a piece decrying cosmopolitan libertarians for exercising our right to decline to associate with racists.
Can this be turned around on me? Of course, but I don't think it holds. I think it's good to shun people for certain kinds of behavior. I just don't think that being born with black parents is among the behaviors it is valid to shun. The paleolibertarians are not, by and large, trying to convince me that I, too, should be frightened of black people--given where I live, this would seem to be a fruitless exercise. They're simply denying that I have a social right to disassociate myself from people who do feel that way.
Well, I stand foursquare behind any American citizen's legal right to hate black people. But they shouldn't expect to be invited for brunch.