This is deference to bureaucrats that neither courts nor citizens would tolerate where a right considered truly important is at stake. Consider the right to free speech. The majority in Citizens United brushed aside public perceptions of corruption to allow unlimited "independent expenditures," even though far more citizens are cynical about campaign donations than about "fraudulent" voters. What about freedom of religion? Would we tolerate licensing of churches so atheists won't worry that "fraudulent" religion is being practiced?
Scholars and courts often note that the Constitution nowhere says, "All individuals have the right to vote." It simply rules out specific limitations on "the right to vote." A right not guaranteed in affirmative terms isn't really a "right" in a fundamental sense, this reading suggests.
But if the Constitution has to say "here is a specific right and we now guarantee that right to every person," there are almost no rights in the Constitution. Linguistically, our Constitution is more in the rights-preserving than in the right-proclaiming business. The First Amendment doesn't say "every person has the right to free speech and free exercise of religion." In the Second, the right to "keep and bear arms" isn't defined, but rather shall not be "abridged." In the Fourth, "[t]he right of the people to be secure ... against unreasonable searches and seizures" isn't defined, but instead "shall not be violated." In the Seventh, "the right of (civil) trial by jury" -- whatever that is -- "shall be preserved." And so on.
In those terms, it ought to mean something that the right to vote is singled out more often than any other. Section 2 of the Fourteenth Amendment imposes a penalty upon states that deny or abridge "the right to vote at any [federal or state] election ... to any of the male inhabitants of such State, being twenty-one years of age, and citizens of the United States, ... except for participation in rebellion, or other crime." The Fifteenth states that "[t]he right of citizens of the United States to vote" can't be abridged by race; the Nineteenth says that the same right can't be abridged by sex; the Twenty-Fourth says that "the right of citizens of the United States to vote" in federal elections can't be blocked by a poll tax; and the Twenty-Sixth protects "[t]he right of citizens of the United States, who are eighteen years of age or older, to vote."
So if our courts treat the ballot as less than a fundamental right, they aren't reading that in the Constitution, but projecting it onto the Constitution. The projection comes from a longstanding belief that the vote is not a "right," but a "privilege" -- something granted by the powerful to the deserving.
The "privilege" theory is one the United States regards as dangerous -- when practiced by other countries. After World War II, we imposed a constitution on Japan providing that "universal adult suffrage is guaranteed." The "Basic Law" of Germany gained a provision that "[a]ny person who has attained the age of eighteen shall be entitled to vote." The citizens of Afghanistan "have the right to elect and be elected." Article 20 of the 2005 Constitution of Iraq provides that "Iraqi citizens, men and women, shall have the right to participate in public affairs and to enjoy political rights including the right to vote, elect, and run for office."