As the Delta variant sweeps the world, researchers are tracking how well vaccines protect against it — and getting different answers.
In Britain, researchers reported in May that two doses of the Pfizer-BioNTech vaccine had an effectiveness of 88 percent protecting against symptomatic disease from Delta. A June study from Scotland concluded that the vaccine was 79 percent effective against the variant. On Saturday, a team of researchers in Canada pegged its effectiveness at 87 percent.
And on Monday, Israel’s Ministry of Health announced that the effectiveness of the Pfizer-BioNTech vaccine was 64 percent against all coronavirus infections, down from about 95 percent in May, before the Delta variant began its climb to near-total dominance in Israel.
Although the range of these numbers may seem confusing, vaccine experts say it should be expected, because it’s hard for a single study to accurately pinpoint the effectiveness of a vaccine.
“We just have to take everything together as little pieces of a puzzle, and not put too much weight on any one number,” said Natalie Dean, a biostatistician at Emory University.
In clinical trials, it’s (relatively) easy to measure how well vaccines work. Researchers randomly assign thousands of volunteers to get either a vaccine or a placebo. If the vaccinated group has a lower risk of getting sick, scientists can be confident that it’s the vaccine that protected them.
But once vaccines hit the real world, it becomes much harder to measure their effectiveness. Scientists can no longer control who receives a vaccine and who does not. If they compare a group of vaccinated people with a group of unvaccinated people, other differences between the groups could influence their risks of getting sick.
It’s possible, for example, that people who choose not to get vaccinated may be more likely to put themselves in situations where they could get exposed to the virus. On the other hand, older people may be more likely to be vaccinated but also have a harder time fending off an aggressive variant. Or an outbreak may hit part of a country where most people are vaccinated, leaving under-vaccinated regions unharmed.
One way to rule out these alternative explanations is to compare each vaccinated person in a study with a counterpart who did not get the vaccine. Researchers often go to great lengths to find an unvaccinated match, looking for people who are of a similar age and health. They can even match people within the same neighborhood.
“It takes a huge effort,” said Marc Lipsitch, an epidemiologist at the Harvard T.H. Chan School of Health.
For its new study, Israel’s Ministry of Health did not go to such great lengths to rule out other factors. “I am afraid that the current Israeli MoH analysis cannot be used to safely assess it, one way or another,” Uri Shalit, a senior lecturer at the Technion — Israel Institute of Technology, wrote on Twitter.
Israel’s numbers could also be different because of who is getting tested. Much of the country is vaccinated. During local bursts of new infections, the government requires testing for anyone — symptoms or not — who came into contact with a person diagnosed with Covid-19. In other countries, it’s more common for people to get tested because they’re already feeling sick. This could mean that Israel is spotting more asymptomatic cases in vaccinated people than other places are, bringing their reported effectiveness rate down.
Fortunately, all the studies so far agree that most Covid-19 vaccines are very effective at keeping people out of the hospital and have generally protected against the Delta variant. Israel’s Ministry of Health estimated that the Pfizer-BioNTech vaccine is about 93 percent effective in preventing serious illness and hospitalization.
“Their overall implications are consistent: that protection against severe disease remains very high,” said Naor Bar-Zeev, an associate professor at the Johns Hopkins Bloomberg School of Public Health.
Because effectiveness studies are so tricky, it will take more work to determine how big a threat Delta poses to vaccines. Dr. Lipsitch said that studies from more countries would be required.
“If there are five studies with one outcome and one study with another, I think one can conclude that the five are probably more likely to be correct than the one,” Dr. Lipsitch said.
