We’re in an era where it’s no longer reasonable to pretend that patients aren’t Googling their symptoms—or to try to get them not to. Why must we feign calm, asking the doctor “What do you think it is?” when the question on our minds is “DOC, IS IT CANCER? GOOGLE SAID IT’S CANCER.” And doctors, you’re using Wikipedia too. You’re not above it. No one’s above it. Let’s end this pussyfooting about.

So, assuming patients are going to Google, and that the Internet is a vast sea containing both accurate and wildly unhelpful health information, the question is how to steer people toward the former and away from the latter.

As a slightly more official alternative to going to Yahoo! Answers, typing in “What is this rash?” and waiting for the geniuses who frequent those forums to dispense their wisdom, many companies and organizations have developed “symptom checkers.” The most well-known of these is probably WebMD’s, but there are also symptom checkers run by the Mayo Clinic, the American Academy of Pediatrics, and the U.K.’s National Health Service. You plug in your symptoms, and an algorithm spits back possible diagnoses, and/or whether you should seek treatment or deal with the issue yourself.

But according to a new study published in The BMJ, these symptom checkers vary widely in accuracy. Researchers from Harvard Medical School and Boston’s Children’s Hospital input 45 “patient vignettes”—some common, some uncommon conditions—into 23 different symptom checkers.

On average, they included the correct diagnosis in the first three results 51 percent of the time, and included it in the first 20 results 58 percent of the time (the individual checkers ranged from 34 percent to 84 percent). The chances of getting the right diagnosis listed first were really hit or miss. On the low end of the spectrum MEDoctor listed the correct condition first 5 percent of the time; on the high end, DocResponse listed it 50 percent of the time. Understandably, the sites did better at identifying common conditions than uncommon ones.

Of the symptom checkers that gave triage advice (whether the condition requires medical attention), they did so “appropriately,” according to the researchers, 57 percent of the time. Triage advice was more likely to be correct for more urgent conditions or uncommon conditions.

“Although there was a range of performance across symptom checkers, overall they had deficits in both diagnosis and triage accuracy,” the study reads. It’s also possible that the study over-estimated how accurate these sites are, because the patient vignettes it used included clinical terms (like “mouth ulcers,” for example) that a layperson might not necessarily know or recognize his symptoms as.

One of the other issues the researchers saw was that the sites weren’t very good, on the whole, at recommending self-care when it was appropriate, which could lead to people going to the ER or doctor’s office for something that would go away on its own, or that could be treated over the counter.

As the researchers point out, the central question here is what role symptom checkers play for people. A lot of the antipathy toward people researching symptoms online I think is predicated on the idea that this would replace a doctor’s visit. If that were true, it would be a bad thing.

“If symptom checkers are seen as a replacement for seeing a physician, they are likely an inferior alternative,” the study reads. “It is believed that physicians have a diagnostic accuracy rate of 85 [to] 90 percent. However, in-person physician visits might be the wrong comparison because patients are likely not using symptom checkers to obtain a definitive diagnosis but for quick and accessible guidance.”

Most people are hopefully savvy enough not to take a WebMD result as a diagnosis. They’re probably just trying to figure out how seriously to take their condition. In that case, it might be less important that symptom checkers spit out the right diagnosis (and including it among 19 incorrect diagnoses is not super helpful anyway), and more important that they tell the searcher whether her symptoms warrant a trip to the hospital.

It seems symptom checkers are now erring on the side of caution—“In two thirds of standardized patient evaluations where medical attention was not necessary, we found symptom checkers encouraged care,” the study reads. And whether people follow the sites’ advice depends on how much they trust them. In another study published in the Journal of Medical Internet Research, older adults were skeptical about diagnoses they found online. So they might seek the opinion of a doctor anyway. (And this new study, finding that symptom checkers are often inaccurate, probably won’t do much to instill trust.)

And as the researchers note in the BMJ study, “some patients researching health conditions online are motivated by fear, and the listing of concerning diagnoses by symptom checkers could contribute to hypochondriasis and ‘cyberchondria,’ which describes the escalated anxiety associated with self diagnosis on the Internet. Together, confusion, risk-averse triage advice, and cyberchondria could mean that symptom checkers encourage patients to receive care unnecessarily and thus increase healthcare spending.”

This hits on an important point: Fear is intricately tied into the decisions people make about their healthcare. Which means they aren’t always logical. Even if someone just has a cold, and WebMD lists a cold as the first possible diagnosis, if it also lists cancer farther down, what’s the eye going to be drawn to? If the Mayo Clinic’s website advises you to wait a few days and see if your symptoms improve before going to the doctor, are you going to wait and worry or drive to the hospital anyway just for the relief of getting an answer? I suspect it depends on the person’s temperament. But if symptom checkers were more reliable (and people knew it), perhaps they could be a tool for managing fear rather than fueling it.