The reason we didn't prevent 9/11 is simple: neither the CIA nor its intelligence allies, Western or Muslim, had a spy or an informant inside al-Qaeda's command structure. And the stark reality is that our human intelligence against al-Qaeda and other Sunni militants will probably never be as good as what we had against the Soviet system during the Cold War.
Why is this the case? In the Soviet Union the people most difficult for Western intelligence agents to recruit were found at the entry level of the Communist system—young men and women who were moving from youth groups and school systems into the military, the KGB, Party organizations, or the diplomatic corps. At this stage these people were steeped in Marxism-Leninism, believed that socialism worked, had faith in the USSR, and were hostile toward the United States. The ideologically committed are always the toughest to recruit for intelligence services.
But on those occasions when the West could develop an informant at this level, the Soviet system unwittingly assisted in that development. With each promotion in the Communist ranks, the potential informant would see more clearly that socialism delivered nepotism, tyranny, and corruption, rather than fairness and equity. Non-Russians (those hailing from the Soviet republics and satellite states) would quickly realize that ethnic discrimination dominated the world in which they worked. In short, the further up the Soviet hierarchy our would-be informant progressed, the more likely the system was to disillusion him, making him more vulnerable to the Western intelligence services.
Here's the challenge that al-Qaeda and other Sunni militant groups pose: In such organizations the old Soviet scenario is exactly reversed—the militants who are least ideologically committed (and therefore most easily recruited by our spy agencies) are found at the edges of the groups, among the ranks of those who perform gunrunning, human smuggling, and narcotics trafficking. Once we've recruited these people, their value to us increases as they move toward the center of al-Qaeda. The problem is that the higher a would-be spy rises in al-Qaeda's ranks, the greater the ideological and theological commitment of his associates; Sunni leaders are often (though certainly not always) the devout and courageous men their media organizations claim them to be. Career advancement in al-Qaeda tends to wash away much of the mercenary hypocrisy found at the entry level—and therefore, in effect, to unrecruit those cultivated by our intelligence agencies. The odds of our ever having an informant among the senior al-Qaeda decision-makers are remote.