Of course, the Fourth Amendment does apply to the Americans that foreigners talk to on the phone or by email. A coalition of American journalists, attorneys, and nonprofits challenged FISA on the basis that the government is very likely monitoring their communications with non-Americans located abroad in violation, they claim, of statutory and constitutional law. The members of the coalition could not know for certain they were being monitored because under FISA, the government does not have to tell them. But given who they were talking to, and how often, the coalition assumed the government was listening is on at least some of their conversations.
In a five-four decision written by Justice Samuel Alito, the Supreme Court dismissed the collation's challenge to FISA for lack of standing (pdf). The doctrine of standing says that the parties before the court must have been injured in fact, and not merely speculating as to some possible harm. Which generally makes sense: you want the people who really have a stake in the outcome to litigate the case. Here, however, no one knows who has that stake because no one knows which Americans, if any, are being spied upon. The coalition could not prove that any single communication had been intercepted under the authority of FISA, or that their burdensome practice of leaving the country just to talk to non-American sources and clients in confidence was in fact prudence or paranoia.
Put another way, the coalition could not challenge our secret surveillance laws because they are secret. There is no one who can complain of his or her rights having been violated, because anyone's whose rights have been violated doesn't know it. That's the catch when it comes to assessing the legality of the government's secret activities.
This is not the first privacy Catch-22 on record. In Laird v. Tatum, a 1972 case upon which the Clapper majority heavily relies, the Supreme Court dismissed an action against a group who alleged that the U.S. Army's then-practice of monitoring their lawful political activity had a chilling effect on the exercise of their First Amendment right to gather in protest. The Court's avowed basis for the dismissal was that the group could not point to any adverse action the Army had taken against them, or to any instance in which the Army had infiltrated their ranks -- they were watching from a distance. But what it really seemed to boil down to was that the group couldn't show any evidence they were cowed by the Army's attention. After all, the Court implied, they brought a lawsuit! People who challenge government surveillance are not chilled by it. Only people who too afraid to bring a lawsuit can, well, bring a lawsuit.
This is not to say that either Laird or Clapper is illogical, or to deny the decisions reflect real and important values. Whatever its flaws, the FISA surveillance regime is better than what came before it. And others can challenge the statute, such as the telecommunications carriers, or any citizen convicted (if not targeted) on the basis of FISA surveillance. Finally, even had it reached the substance of the challenge, courts are historically reticent to overrule activities that have the support of both the President and Congress. But Clapper does point to a problem citizens seem to face in challenging surveillance excess. Anyone crazy enough to try it, isn't really crazy.