Study cohort

CPRD [37] contains primary care records from 45 million patients registered at General Practices in the UK. Patients who had reached the age of 65y by the year 2000 were extracted from CPRD. 259,753 patients met the inclusion criteria for this study (Fig 1, S1 Fig). The PERR methodology requires that patients should never have had the treatment prior to the study period. In the case of influenza vaccination, this may not be the case. To avoid any residual vaccination effect, we selected patients who were vaccination free for at least two years before the study period. 88,519 had an influenza vaccination in the 2000 season. Vaccinated patients were matched to controls based on a propensity score (PS) model using 1:1 nearest neighbour matching. Matching was based on disease covariates, selected according to clinical risk groups [38] and the Quality Outcomes Framework [39]; a proxy for patient health using a binary variable for ≥12 GP consultations [4] between 01/09/1999–31/08/2000; smoking status, age, gender, ethnicity and region were also included. Table 1 lists all variables (age-dementia) our cohort were matched on.

Fig 1

Inclusion and exclusion criteria for study.

Table 1

Mean (SD) or percentage reported.

Demographic characteristics of each group.

	Vaccinated Group (n = 88519)	Control Group (n = 88519)
Prescribed amoxicillin in prior period	7.5%	4.0%
Prescribed amoxicillin in study period	7.1%	4.4%
Dies in study period	1.3%	1.2%
Age Mean(SD)	73.3 (6.7)	73.3 (6.6)
Gender (% male)	44.6%	45.7%
Ethnicity White	74.4%	74.7%
Unknown	24.1%	23.8%
Non-White	1.6%	1.5%
Smoking Current	47.6%	47.4%
Ex-smoker	10.0%	10.4%
Non-smoker	42.1%	42.1%
More than 12 GP consultations	9.6%	8.7%
Asthma	13.8%	13.1%
Mental Health	0.9%	0.8%
Epilepsy	3.1%	3.0%
Thyroid	5.0%	4.9%
Hypertension	28.6%	28.1%
Stroke	5.3%	5.1%
Chronic Heart Disease	11.9%	11.3%
COPD	2.7%	2.5%
Heart Failure	2.8%	2.6%
Cancer	5.2%	5.0%
Depression	9.6%	9.4%
Diabetes	1.8%	1.6%
Arterial Fibrillation	3.1%	2.9%
Dementia	0.6%	0.6%

Ethics approval

Approval for the study was granted by the CPRD Independent Scientific Advisory Committee (ISAC 14_159R2). Patient data received from CPRD is fully anonymised.

Outcomes

There are no outcome measures in primary care data that explicitly identify viral influenza infection; laboratory confirmed influenza is not well recorded in primary care and all-cause mortality is not considered useful [4–6]. The main effect of influenza is respiratory; there is evidence of a reduction in respiratory illness due to the influenza vaccination [1–3]. As respiratory illnesses are difficult to diagnose precisely in early stages, patients, particularly the elderly, are prescribed antibiotics. Antibiotic prescription may serve as a proxy for influenza presenting as a respiratory infection or possible bacterial infection complicating influenza [1, 7, 8] Amoxicillin is frequently prescribed for respiratory illness [11] and antibiotic prescription is recorded reliably in CPRD data [40]. The outcome measure for this study was time to amoxicillin prescription. Relevant amoxicillin codes were identified by a pharmacist. September was used as the start of the influenza season; the prior period was defined between 01/09/1999–30/04/2000 and the study period 01/09/2000–30/04/2001; Fig 2. The adjusted start date for each vaccinated patient was the vaccination date plus 14 days to allow for antibodies to reach protective levels [33]. The cohort was followed in the study period from adjusted start date until they either received antibiotics, died or reached the end of the study period. The start date in the prior period was the adjusted vaccination date minus one year; patients were followed until they either received amoxicillin prescription or reached the end of the period. Control patients were assigned start dates for follow-up based on the vaccination dates of vaccine recipients, to whom they had been matched.

Fig 2

Study design schematic.

Statistical analysis

To tackle bias arising from unmeasured confounding in EHR data, several methods have been proposed [41, 42] with some previously applied for estimating influenza vaccine effects [6, 27, 43]. One promising method is the prior event rate ratio (PERR), and an extension of this method, PERR Pairwise [34, 35]. The PERR approach involves fitting a model for event outcomes after the start of treatment (the study period) and adjusting treatment estimates based on a model for the same outcome in a prior period. By selecting a cohort of vaccinated and unvaccinated patients that have never received the treatment before the study period, the hazard ratio (HR) for the prior period should reflect the effect of confounders, measured and unmeasured, independent of vaccination status. Simulation studies [34–36, 44] have found that bias in treatment effect estimates in the presence of unmeasured confounders can be reduced using this methodology.

To assess IVE in a real-world setting, we apply Cox proportional hazards (PH) models, PERR and PERR Pairwise to CPRD data. Cox (PH) models were fitted to the outcome in the prior period and the study period. Hazard ratios (HRs) for the risk of antibiotic prescription in the vaccinated group vs the unvaccinated group are calculated for each time period, HR_prior and HR_study. Any difference between the vaccinated and unvaccinated groups in the prior period (i.e. if the 95% confidence interval for HR_prior does not include 1) is assumed to reflect unmeasured confounding.

The PERR methodology adjusts the HR_study using the estimate of the underlying difference between the two groups identified in the prior period, HR_prior, as follows:

The extension to this method, PERR Pairwise, uses a paired Cox regression and reduces bias found in the original formulation [34, 35]. Mathematical details of the statistical models, and R code, are reported elsewhere [34, 35, 45]. After matching on the propensity score, vaccination status was the only covariate in the models.

CRPD data were extracted using Stata v15.0 and analysed using R v3.0.2.

Sensitivity analyses

Previous work has proposed that negative control outcomes (NCO) [45] and simulations should accompany analyses using observational data [46]. We used incidence of oedema as an NCO to test the robustness of applying PERR Pairwise to produce real-world estimates of IVE; this is unrelated to amoxicillin prescription and we would expect to find no difference between the two groups (S1 File). A simulation study (S2 File) was designed to replicate the features of our motivating data with an unmeasured confounder influencing outcome. We explored the impact on IVE estimates if the model assumptions for the PERR methodology were violated. In particular, we simulated scenarios in which vaccinated and control groups could have different characteristics, confounders were allowed to be time-varying (comorbidities over the prior and study periods) and there was heterogeneity in response to vaccination, i.e. no subgroups (S2 File, S1 Table).

Study cohort

Of those over 65yrs in our data, 38% (n = 515,580) received the influenza vaccination in 1999 and 56% in 2000 (n = 514,291); this is reasonably comparable to the increase in coverage reported by Public Health England. In our study population, the vaccinated group received a greater number of amoxicillin prescriptions in both periods than the control group, Table 1. There is a difference in the number of prescriptions in the vaccination-free prior period, 7.5% for the group to who went on to be vaccinated vs 4.0% for the control group. This indicates the presence of a pre-existing difference in the health of the exposure groups or a difference in healthcare-seeking behaviour; those who were vaccinated were more likely to be prescribed amoxicillin. Prescriptions in the vaccinated group decreased to 7.1% in the study period, whereas prescriptions in the control group increased to 4.4%.

Influenza vaccine effectiveness

The results from the propensity score-adjusted Cox PH model for the prior period showed a difference in the hazards between the vaccinated and control groups. The bias was not fully removed by matching on the propensity scores (which were derived from the measured confounders alone) (for the ≥ 65y cohort: HR 1.90 95% CI [1.83, 1.98]; Fig 3). The HR from the PS-adjusted Cox model for the study period was above 1, corresponding to an implausible harmful effect of vaccination (for the ≥ 65y cohort: HR 1.64 95% CI [1.58, 1.71]). However, accounting for group-differences in outcomes in the prior period with PERR and PERR Pairwise models, gave a protective effect estimate for vaccination, with HRs 0.86 (0.81,0.92) and 0.86 (0.81, 0.92) respectively.

Fig 3

Results of influenza vaccination effectiveness in 2000 season.

Age

The subgroups based on age showed similar results. More prescriptions were issued to the vaccinated groups in the prior period, 7.5% in <75yrs and 7.6% in ≥75yrs in the vaccinated group and 4.3% in <75yrs and 3.6% in ≥75yrs for the control group. Similarly, in the study period, rates of prescriptions were 7.0% and 7.1% for <75yrs and ≥75yrs respectively in the vaccinated group and 4.5% and 4.0% respectively in the control group. We again found evidence of a difference between the groups before the study period and Cox PH models for the study periods (Fig 3) indicating a greater likelihood of antibiotic prescription in the vaccinated group in both age groups in the prior period. The PERR Pairwise adjustment found a beneficial response for the <75y age group, PERR Pairwise 0.87 (0.81,0.94). A beneficial effect of the influenza vaccination for ≥75yrs was also found, HR 0.84 (0.76,0.94).

Sensitivity analyses

After adjusting for the prior period with the PERR methods, the artefactual association between oedema, our NCO, and influenza vaccination was eliminated, HR 1.05 (0.95,1.16) and 1.06 (0.95,1.19) for PERR and Pairwise, respectively (S1 File, S2 Fig). In the simulation study (S2 File, S3 and S4 Figs, S1 and S2 Tables), we tested the robustness of estimates to violations of the PERR methodology assumptions (S1 Table). We found that bias would be induced if there was a change between the prior and study periods or if a proportion of the vaccinated group did not respond to the vaccination (immunosenescence). The study provided no evidence for a difference in IVE between age groups suggesting immunosenescence was not an important factor; thus, providing there were no substantial change over time within patients, consistent with our NCO analysis, the estimates using PERR are robust. The PERR and Pairwise results in the simulation studies were similar and only Pairwise is presented in S2 File.

Strengths and limitations

Given the strain of the influenza virus can vary from year to year, it is possible that not only will vaccination effectiveness vary but the influence of the confounders may also vary. Bias may be introduced into estimates from the PERR methods when the influence of unmeasured confounders on response is different in the prior and study periods. The PERR methodology also requires unmeasured confounders to be time invariant. Some advice should be taken prior to applying the PERR methodology about how comparable the prior and study periods are. In this case consultation with a virologist about the strains of influenza and their impact on health in the two seasons should inform study design. In this study, patients identified with a vaccine-free prior period could have a history of exposure to the vaccine thus potentially contaminating the prior period and reducing the performance of the method in adjusting for bias. However, a strength of the PERR methodology is that it allowed us to adjust for the prior incidence of antibiotic prescriptions and greatly reduce the imbalance between the two groups without full knowledge of the underlying confounders. A naïve model could use prior use of antibiotics as a covariate in the study model but this would not be as effective in reducing bias due to confounding as the PERR adjustment [45]. Our NCO outcome, oedema, showed bias in estimates using prior and study periods only. Applying the PERR methodology removed a false association between influenza vaccination and oedema.

The simulation study showed where we could ‘trust’ our estimates of vaccine efficacy. Our simulations are novel in that they utilise biologically plausible baseline hazards and adding a small competing risk. In scenario 1, the PERR methods were robust to differences in the distributions of the continuous confounder in the vaccinated and control groups, provided they remained the same within each group over the two periods, whereas we found high bias in the Cox model in comparable settings.

In scenario 2, the confounding variable which influenced the groups changed between periods. We found biased estimates in this scenario, however confidence intervals still contained the true effect. Assuming our vaccinated and control groups did not change significantly in their underlying health between the two time periods, then our PERR estimates were valid. We attempted to capture health seeking behaviour that may contribute to variation in outcomes using the number of GP visits in one year as a covariate. This approach may fail to pick up where health seeking behaviour has changed over time and further work is required to investigate how best to incorporate these time dependent effects within the PERR framework. The simulation study scenario 3 showed that, if there were a subgroup who did not respond to vaccination, any analysis would be biased. We found no difference in response to vaccination due to age in our cohort which suggested that immunosenescence did not impact on the results.

Our study concentrated on amoxicillin prescription as an outcome to study reduction in antibiotic prescription following influenza vaccination. Incidence of influenza is poorly recorded in EHR data and proxies related to respiratory illness, such as amoxicillin prescription, provide a feasible alternative. Amoxicillin can be used for many different types of bacterial infections but we chose to use a sensitive outcome definition without restriction to amoxicillin prescriptions where there was an associated clinical code for respiratory illness. We note that some patients may have been allergic to penicillin and unable to receive amoxicillin. Amoxicillin is the most commonly prescribed antibiotic for respiratory illness [11]; but further work could explore other antibiotics.

Comparison to other literature

Klugman and Black [14] reviewed studies of antibiotic resistance after influenza vaccination. One efficacy study, reporting the effect on antibiotic prescriptions found 13.2% fewer prescriptions in the six months after vaccination [47]. Another study found 36% fewer prescriptions in the vaccine group in Asia Pacific, 59% in Central America and 71% in Europe as well as a reduction in antibiotic prescriptions across five seasons [48]. However, both of these studies were in young children. An observational study in the UK [49] found a reduction of 14.5% in amoxicillin prescription during the influenza season which is a similar figure to our study, but again this was a study in children. Kwong et al [7] looked at the effect of the Canadian introduction of an influenza programme in 2000 and found a risk Ratio 0.36 (0.26–0.49). This 64% reduction in risk of antibiotic prescription covers a population from 6 months upwards. Our study found a reduction of 14% (8–19%) in antibiotic prescriptions in those ≥65yrs. This matches the positive effect of influenza vaccination on antibiotic prescription seen in other studies [14]: however, no other studies have looked at this in our aged population.

Our results did not find a diminished protective effect with age; 13% (6–19%) reduction in risk of prescriptions for <75 yrs and 16% (6–24%) in those ≥75yrs. Previous work has commented on the lack of evidence for effectiveness in much older people [4, 5, 15, 16, 26], and while we found a beneficial effect in ≥75yrs here, we would caution against interpreting this as definitive evidence; age may be too simple a subgrouping. Functional status, location and social connectivity have been identified as possible subgroups [2, 29, 50], but such data are not available in CPRD. Further work should explore more complex subgroups.