Data sources and searches

A protocol was developed and registered with PROSPERO [25]. A systematic literature search was conducted during 26–28 November 2018 to identify published evidence around the four concepts of: (diabetes) and (hypoglycaemia) and (psychosocial outcomes) and (measurement properties of measurement instruments). Databases searched include MEDLINE, EMBASE, PsycINFO, CINAHL and The Cochrane Library. Terms for psychosocial outcomes were chosen to include both generic, ‘umbrella’ terms for ‘quality and life’ and ‘well-being’ (sourced from published search filters) and specific psychosocial outcomes of diabetes known to the Hypo-RESOLVE team (e.g. fear of hypoglycaemia). In order to identify studies for the present systematic review, a validated search filter devised for retrieving studies on measurement properties of instruments in PubMed was used [26]. An example search strategy is shown in the electronic supplementary material (ESM) Methods.

Study selection

Inclusion criteria consisted of any study design that included the primary development and/or validation of a hypoglycaemia-specific PROM used to assess the impact of hypoglycaemia on QoL in adults diagnosed with diabetes with any type, e.g. type 1, type 2 and gestational, and who have experienced hypoglycaemia. Studies of hypoglycaemia/hypoglycaemic episodes not associated with diabetes were excluded. Commentaries, reviews, opinion pieces and any other non-empirical work were also excluded. Studies were assessed for inclusion at title and abstract stage by one reviewer (JL). Full-text articles were scrutinised where considered as relevant or potentially relevant or where doubt existed. Twenty per cent of studies were assessed by a second reviewer (JC) to check for consistency. Disagreements were resolved through discussion.

Data extraction

Data extraction included study characteristics (e.g. language; participant characteristics; recall period; analysis model), a brief summary of results and measurement properties of the PROMs. Primary outcomes included measurement properties of identified PROMs, consistent with the COSMIN checklist: PROM development; content validity; structural validity; internal consistency; cross-cultural validity/measurement invariance; reliability; measurement error; criterion validity; hypothesis testing for construct validity; and responsiveness. Definitions of the measurement properties are detailed in Table 1. In accordance with COSMIN guidelines, all data relating to PROM measurement properties were extracted independently by two reviewers (JL and JC) against the respective COSMIN criteria. Discrepancies were resolved through discussion.

Table 1

Definitions of measurement properties

Measurement property	Definition
Content validity	The extent to which the items in a PROM are representative of the construct they are intended to measure
Structural validity	The extent to which the items in a PROM reflect the dimensionality of the construct (i.e. the items form a single [unidimensional] scale or multiple subscales [a multidimensional scale])
Reliability: internal consistency	The extent to which there is consistency of results across items in the PROM (i.e. within a specified scale or subscale)
Reliability: test–retest	The extent to which the PROM yields scores that are reproducible (stable) over time when there has been no change in the concept being assessed
Measurement error	The systematic and random error of a person’s score on the PROM that is not attributed to changes in the construct to be measured
Criterion validity	The extent to which the scores of a PROM reflect the scores of a test or measure considered to be the ‘gold standard’
Hypothesis testing for construct validity	The extent to which the scores of a PROM are consistent with hypotheses. For example, with regard to internal relationships, relationships to scores of other instruments or differences between relevant groups. It is based on the assumption that the PROM is a valid measure of the construct
Responsiveness	The ability of a PROM to detect change, as expected, over time in the construct to be measured when there is a true change in a person’s condition or treatment
Cross-cultural validity	The extent to which the measurement properties of the translated or culturally adapted PROM reflect the performance of the original version of the PROM

Content validity assessment

Content validity is the extent to which a PROM is deemed to reflect the construct of interest and, arguably, the most fundamental aspect of scale selection [27]. The methodological quality of the PROM development studies and other studies supplementing content validity were assessed using COSMIN standards [28]. The assessment involves three steps (see Fig. 1): (1) evaluation of the quality of the PROM development; (2) evaluation of the quality of any additional content validity studies on the PROM (if available); and (3) evaluation of the content validity of the PROM based on the quality and results of the available studies and the PROM itself. Steps 1 and 2 result in a rating of each COSMIN standard ranked on a four-point scale: ‘very good’, ‘adequate’, ‘doubtful’ and ‘inadequate’. Total ratings are then determined using the lowest rating for any item for that study (i.e. worst score counts) [22].

Fig. 1

COSMIN assessment of content validity

Step 3 consists of three sub-stages. Step 3a incorporates reviewer ratings of the identified PROMs whereby reviewers consider relevance, comprehensiveness and comprehensibility. We sought ratings from three key stakeholder groups: (1) researchers (including those with expertise in systematic reviewing, QoL research and psychological aspects of diabetes) (n = 6); (2) clinicians (n = 6); and (3) adults with diabetes (n = 4), including two representatives of the Hypo-RESOLVE Patient Advisory Committee (PAC). All reviewers provided independent ratings of the PROMs based on several criteria: (1) the construct of interest (i.e. does the PROM include items that are relevant in measuring the impact of hypoglycaemia on QoL?); (2) the population of interest; (3) the context of use of interest (i.e. is the PROM suitable for use in research and/or clinical practice?); (4) the appropriateness of response options; (5) the appropriateness of the recall period; (6) the comprehensiveness (i.e. does the PROM assess the impact of hypoglycaemia on QoL as a whole, or only on select domains of QoL?); (7) the suitability/clarity of the PROM instructions; (8) whether PROM items and response options are understandable; (9) the appropriateness of PROM item wording; and (10) the extent to which response options are appropriate to the question being asked. A majority rating was determined for each group (researcher, clinician and PAC). The group ratings were then consolidated to produce an overall reviewer rating for each PROM. Table 2 details how relevance, comprehensiveness and comprehensibility were assessed.

Table 2

COSMIN criteria and rating system for evaluating the content validity of the PROMs (adapted from Terwee et al [28]), with an example shown in italics

Name of PROM (or subscale)	PROM development study (+/−/±/?)	Content validity study 1 (+/−/±/?)	Content validity study 2a (+/−/±/?)	Rating of reviewers (+/−/±/?)	Overall ratings per PROM (+/−/±/?)	Quality of evidence (High, moderate, low, very low)
The ABC-QoL	Jones et al, 2015	Smith et al, 2016
Relevance
1. Are the included items relevant for the construct of interest?b	+	+		+
2. Are the included items relevant for the target population of interest?c	+	+		+
3. Are the included items relevant for the context of use and interest?d	−	−		−
4. Are the response options appropriate?	+	−		+
5. Is the recall period appropriate?	+	+		+
RELEVANCE RATING	+	+		+	+
Comprehensiveness
6. Are all key concepts covered?	−	−		−
COMPREHENSIVENESS RATING	−	−		−	−
Comprehensibility
7. Are the PROM instructions understood by the population of interest as intended?	+	+
8. Are the PROM items and response options understood by the population of interest as intended?	+	+
9. Are the PROM items appropriately worded?				+
10. Do the response options match the question?				+
COMPREHENSIBILITY RATING	±	±		±	±
CONTENT VALIDITY RATING					±	High

Step 3b involves summarising the results of all available studies to provide an overall rating of relevance, comprehensiveness and comprehensibility and an overall content validity rating. This results in an outcome of ‘sufficient’, ‘insufficient’, ‘inconsistent’ or ‘indeterminate’. Finally, in Step 3c, the overall ratings determined in Step 3b are accompanied by a grading of the quality of the evidence using a modified Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach [29]. Using the modified GRADE approach, the quality of evidence is graded as ‘high’, ‘moderate’, ‘low’ or ‘very low’. The GRADE approach uses five factors to consider the quality of the evidence: risk of bias, inconsistency, indirectness, imprecision and publication bias [29]. Detailed information of the rating process is reported elsewhere [28]. The resultant evaluation of content validity includes an overall rating of: + (‘satisfactory’); − (‘unsatisfactory’); ± (‘inconsistent’); or ? (‘indeterminate’), with a measure of the quality of the evidence to support the content validity rating (‘high’, ‘moderate’, ‘low’, ‘very low’). A worked example of content validity rating and scoring is shown in Table 2. Detailed information on the COSMIN methodology applied is reported elsewhere [28].

Assessment of other psychometric properties

Table 1 defines each of the psychometric properties assessed. As above, a COSMIN rating was determined by assessment across the criteria for measurement properties using the same rating scale (‘sufficient’, ‘insufficient’, ‘inconsistent’ or ‘indeterminate’). The assessment of the quality of the evidence was applied using the GRADE approach. This results in a rating of: + (‘satisfactory’); − (‘unsatisfactory’); ± (‘inconsistent’); or ? (‘indeterminate’), with a measure of the quality of the evidence to support the structural validity rating (‘high’, ‘moderate’, ‘low’, ‘very low’). Full information on the COSMIN methodology applied in this review is reported elsewhere [23].

Quality assurance of the review

The quality of this review was assessed against a COSMIN checklist that was designed to evaluate the quality of systematic reviews of PROMs [30] (ESM Table 1).

Overall COSMIN assessment of PROMs

The overall results of the COSMIN assessment are shown in Table 5. There are considerable evidence gaps for the measurement properties of most of the PROMs. The HFS-II, QoLHYPO and TRIM-HYPO were the only instruments that could be rated across all the measurement properties.

Table 5

Summary of psychometric properties of hypoglycaemia-specific PROMs used to assess the impact of hypoglycaemia on QoL

PROM	Content validity		Structural validity		Reliability: internal consistency		Reliability: test–retest		Hypothesis testing for construct validity
	Rating of results	Quality of evidence	Rating of results	Quality of evidence	Rating of results	Quality of evidence	Rating of results	Quality of evidence	Rating of results	Quality of evidence
FH-15	±	Low	−	Moderate	+	Moderate	NR	NR	?	Moderate
HFS	±	Moderate	−	Moderate	?	Moderate	−	Very low	NR	NR
HFS-II	±	Low	−	Moderate	+	Moderate	−	Moderate	?	Moderate
HFS-Norwegian	NR	NR	−	High	+	High	+	High	NR	NR
HFS-Singapore	NR	NR	−	High	+	High	?	High	?	High
HFS-Spanish	NR	NR	−	Moderate	?	Moderate	?	Very low	?	Moderate
HFS-Swedish	NR	NR	−	Moderate	+	Moderate	NR	NR	?	Moderate
HFS-II short-form	NR	NR	−	High	+	High	NR	NR	?	High
HABS	±	Very low	−	Moderate	+	Moderate	NR	NR	?	Moderate
HCS	±	Low	−	Moderate	+	Moderate	NR	NR	?	Moderate
QoLHYPO	±	Moderate	−	Moderate	+	Moderate	+	Moderate	?	Moderate
TRIM-HYPO	±	Low	?	Moderate	±	Moderate	−	Very low	±	Moderate

Content validity

ESM Table 2 summarises the key characteristics and COSMIN quality assessment of the PROM development studies. For five of the seven PROMs, there was evidence that adults with diabetes were involved in item generation (HFS, HABS, HCS, QoLHYPO and TRIM-HYPO). COSMIN quality ratings ranged from ‘inadequate’ (HFS, HABS and QoLHYPO), to ‘doubtful’ (HFS-II, HCS and TRIM-HYPO), to ‘very good’ (FH-15). The developers of the HFS-II short-form do not report on content validity, due to the scale being developed based on existing items in the HFS-II [31].

ESM Table 3 details characteristics of the PROM development studies. The overall quality of the PROM development studies was classified as ‘very good’ (FH-15), ‘inadequate’ (HFS, HABS and QoLHYPO) or ‘doubtful’ (HFS-II, HCS and TRIM-HYPO). Only five of the PROMs provided evidence of concept elicitation (all of which were of ‘doubtful’ or ‘inadequate’ quality) (HFS, HABS, HCS, QoLHYPO and TRIM-HYPO). The COSMIN rating for the PROM design ranged from ‘inadequate’ (HFS, HABS and QoLHYPO), to ‘doubtful’ (HFS-II, HCS and TRIM-HYPO), to ‘very good’ (FH-15). Three of the PROMs (HFS, QoLHYPO and TRIM-HYPO) reported on content validity. During the development of the HFS, health professionals were asked about the relevance and comprehensiveness of the PROM (‘doubtful’ COSMIN quality rating) [32]. For the QoLHYPO, adults with diabetes were asked about the comprehensibility, but not relevance, of the PROM (‘doubtful’ COSMIN quality rating) [33]. During the development of the TRIM-HYPO, adults with diabetes were asked about the comprehensibility and relevance of the PROM, but were not asked about comprehensiveness of the PROM ('doubtful' COSMIN quality rating) [34]. Aside from the development studies, no further studies were identified that independently assessed the content validity of the PROMs.

ESM Table 4 details the consensus ratings for the three groups of reviewers (researchers, clinicians, people living with diabetes), and an overall reviewer consensus rating for each PROM. FH-15 had an overall reviewer rating of ‘sufficient’; HFS-II, HABS, HCS and TRIM-HYPO were rated as ‘inconsistent’. For two of the PROMs (HFS and QoLHYPO), relevance, comprehensiveness and comprehensibility ratings resulted in a combination whereby COSMIN guidance is not explicit, and, thus, an overall rating could not be applied [28].

Structural validity

Twelve studies assessed the structural validity of the PROMs, all of which were reported in the development papers (ESM Table 5). No independent assessments of the structural validity were identified. Four studies examined the structural validity of a cultural adaptation/language translation of the HFS [35–38]. A further study assessed the structural validity of the short-form of HFS-II [31]. COSMIN quality ratings of the HFS-Norwegian, HFS-Singapore and HFS short-form were ‘very good’ and ratings were ‘adequate’ for the remaining PROMs. The same principles as noted above were applied to assess the quality of the evidence for these instruments. The quality of evidence for the HFS-Norwegian, HFS-Singapore and HFS-II short-form instruments was assessed as ‘high’. The HFS-Spanish, HFS-Swedish and TRIM-HYPO instruments were assessed as ‘moderate’. Many of the studies reported exploratory factor analysis (EFA) (rather than the confirmatory factor analysis required to receive a ‘satisfactory’ rating). Those studies reporting confirmatory factor analysis (language versions of the HFS) did so to examine whether the expected two-factor structure (observed for the original HFS) fitted their dataset. However, they all rejected this a priori-defined structure, and therefore went on to explore the latent structure of the tool using EFA.

Internal consistency reliability

Thirteen studies were identified that reported evidence of the internal consistency of the PROMs [31–34, 36–43]. Some were undertaken by the instrument developers and some were independent assessments (ESM Table 6). Most studies [32, 33, 36, 39, 40, 42, 43] had an ‘adequate’ COSMIN quality rating. Five studies had a ‘very good’ COSMIN quality rating [31, 34, 37, 38, 41].

Reliability (test–retest)

Seven studies were identified that assessed the test–retest reliability of a PROM measure. Four of the studies were conducted by the instrument developers (FH-15, HFS-II, QoLHYPO and TRIM-HYPO). The remaining studies were assessments of the language versions of the HFS instrument (ESM Table 7). Four studies had an ‘adequate’ COSMIN quality rating [32, 33, 36, 40]. Two studies had a ‘very good’ COSMIN quality rating [37, 38]. One study had a 'doubtful' COSMIN quality rating [34].

Hypothesis testing for construct validity

Ten studies reported on hypothesis testing for construct validity (ESM Table 8) [31, 33–36, 38–40, 42, 43]. Of these, nine were comparing with other outcome measurement instruments (convergent validity) [31, 33–36, 38, 40, 42, 43]. These were HFS-II, HFS-Spanish, HFS-Singapore, HFS-Sweden, HFS-II short-form, HABS, HCS, QoLHYPO and TRIM-HYPO. Six studies included comparisons between subgroups (discriminative or known-groups validity) [34, 38–40, 42, 43]. These were FH-15, HFS-II, HFS-Singapore, HABS, HCS and TRIM-HYPO instruments.

Other psychometric properties

No studies were found to demonstrate evidence for cross-cultural validity, measurement error, criterion validity or responsiveness.