Let's not overthink this: John Derbyshire is a racist. Declaring such does not require an act of of mind-reading, it requires an act of Derbyshire-reading:
I am a homophobe, though a mild and tolerant one, and a racist, though an even more mild and tolerant one, and those things are going to be illegal pretty soon, the way we are going.
I guess it's admirable that Rich Lowry is taking time away from pondering why people think he's a bigot, to denounce Derbyshire. But 'Derb' told you what he was in 2003. And National Review continued to employ him. That's who they are.
What else is there?