# Quality Grading

Estimates of each of the four criteria used to evaluate interventions were graded for the quality of the methods and data employed. The quality grading scales were developed using existing scales in the fields of research design (NICE, 2008), economic evaluation (Drummond and Jefferson, 1996), and statistics (Lyon, 2008).

## Cost effectiveness

The four estimates used to construct the cost per QALY gained estimates were graded separately: cost; effect; the relationship between effect and QALY gains; and the relationship between effect and healthcare cost savings. Each estimate was given a score between 0 and 6, which were aggregated to produce an overall score for cost per QALY gained and net cost per QALY gained estimates. Figure 1 below summarises the process used to translate scores into a quality grading.

**Figure 1: Cost effectiveness quality grading**

Key to grading | ||
---|---|---|

Confidence in the cost per QALY gained estimate |
Score |
Grade |

High confidence | 13 - 18 | *** |

Medium confidence | 6 - 12 | ** |

Low confidence | 0 - 5 | * |

Confidence in the net cost per QALY gained estimate |
Score |
Grade |

High confidence | 17 - 24 | *** |

Medium confidence | 8 - 16 | ** |

Low confidence | 0 - 7 | * |

The separate scores for each of the components of the cost-effectiveness grading system were constructed from the responses to the questions summarised in Figure 2.

**Figure 2: Components of the cost-effectiveness quality grading**

## Reach

The data used to estimate the reach of an intervention were assessed for their accuracy and timeliness. A number of questions were asked about the data to produce scores for accuracy and timeliness, which were then combined into an overall score. The grading is summarised in Figure 3.

**Figure 3: Reach quality grading**

Key to grading | ||
---|---|---|

Confidence in the reach estimate |
Score |
Grade |

High confidence | 9 - 12 | *** |

Medium confidence | 4 - 8 | ** |

Low confidence | 0 - 3 | * |

**Figure 4: Components of the reach quality grading**

## Inequality Score

The data used to estimate the inequality score from an intervention were assessed for their accuracy and timeliness. A number of questions were asked about the data to produce scores for accuracy and timeliness, which were then combined into an overall score. The grading is summarised in Figure 5.

**Figure 5: Inequality score quality grading**

Key to grading | ||
---|---|---|

Confidence in the inequality score estimate |
Score |
Grade |

High confidence | 6 - 12 | *** |

Medium confidence | 2 - 6 | ** |

Low confidence | 0 - 1 | * |

The separate scores for each of the components of the inequality score grading system were constructed from the responses to the methods questions summarised in Figure 6.

**Figure 6: Components of the inequality score quality grading**

## References

Drummond M.F. and Jefferson T.O. (1996), Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ ; 313: 275-283.

Lyon, M. (2008), Assessing data quality. Monetary and Financial Statistics.

National Institute for Health and Clinical Excellence (2008), Guide to the Methods of Technology Appraisal. London, NICE.

All content is copyrighted. Please do not duplicate or redistribute without our permission.