Latest News

Researcher at the Institute for Global Health, University College London, shares his opinion on challenges and solutions in evaluating designs in quality improvement interventions

Bejoy Nambiar is a research associate at the Institute for Global Health, University College London. He has worked on improvement research across Africa in the area of maternal, newborn and child health. He is participating in the session Better Health Care: How Do We Learn About Improvement? 

The choice of comparison groups in the evaluation of complex QI interventions remains a challenge both at a conceptual level as well as at operational level. 

At a conceptual level, quality improvement models are based on Diffusion of Innovation1 concept where the rapid spread of new ideas or practices happens largely by imitation2 through individuals categorized as innovators, early adopters, early & late majority and laggards3. The ‘innovators’ form a small proportion of this category but depending on their network linkages and inter-personal relationships, they can engage the ‘early adopters’ who then go on to influence the ‘early majority’ and so on. The spread is thus organic and dictated largely by interpersonal network, which are not limited by the boundaries of intervention groups. Thus comparison groups can be considered as a hindrance to the organic spread of QI intervention. 

In quality improvement interventions, the selection of groups or clusters (typically these are facilities within a health system) are influenced by their level of commitment or “readiness” to be part of the intervention. Such facilities are bound to possess individual and organizational characteristics that are inherently different from their comparison groups. Thirdly, evaluation designs using a comparison group requires them to be selected at the beginning of the intervention.  The diffusion of innovation approach might be a more organic and perhaps pragmatic approach from an implementation perspective, but this could be in conflict with most conventional evaluation designs where the selection of a comparison group happens apriori. 

From an evaluation perspective, focusing on only the ‘low hanging fruits’ does not tell us if the intervention has the same effect on the ‘not-so-enthusiastic’ facilities. A comparison group gives the opportunity to analyse and understand the mechanisms in play with ‘innovators and early adaptors’ as well as the ‘laggards’. However, from an improvement science perspective4 psychology plays a role in the organic spread of QI interventions. Focusing on the low hanging fruit and managing to get a ‘critical mass’ of facilities that will adhere to QI principles, is a strategy that is more likely to influence the intervention acceptance by ‘later adapters & laggards’. This aspect of psychology of influencing a larger stakeholder group by building a ‘critical mass’ to eventually influence the outcome, is difficult to measure using a comparison group. It gets further complicated if the comparison groups have to be randomly allocated.

At an operational level, there are limitations to the identification and function of comparison clusters. The constitution of a comparison group is determined largely  by the focus of the evaluation design. Evaluations usually focus on probability design to tell if an intervention works or not while plausibility designs attempts to answer the ‘how’ question5. Thus probability designs focus on the measurement of intervention outcomes while plausibility designs focus on the intervention mechanisms. Comparison groups are usually a feature of probability designs and helps to measure attribution of the QI intervention to the outcome. The choice of comparison group is therefore very important and is determined by the level at which the intervention takes place as well as the level at which outcomes are measured. QI interventions can focus at different levels such as improving the processes of care or service delivery mechanism or re-organization of the healthcare systems. In some cases it is a combination of these different levels. Furthermore, these different levels are inter-dependent thus making it difficult to isolate intervention effects even in the presence of a comparison group. Having a comparison group at a higher level such as district or organizational level, raises a challenge for adequate sample size for a quantitative analysis. While having a comparator cluster at a much lower level such as individual service provider level, raises challenges for appropriate comparative sample for outcome measurement. 

For complex QI interventions that have long implementation periods, comparison sites are prone to be influenced by the intervention (referred to as ‘spill-over effect’) and contamination by other interventions cannot be ruled out6.

Given the challenges of intervention and systems complexity and intervention fidelity, a general agreement in evaluation is for plausibility designs to be considered alongside probability designs5 7-9. A little explored area in this regard is the role of comparison groups in plausibility designs where there is a need to understand differences in context and mechanisms between intervention and comparison sites. Unlike control groups in clinical trials which are methodologically considered to be inert, comparison groups in mixed methods evaluation have a different set of context and mechanisms operating, even in the absence of QI intervention and whose dynamic nature is influenced by time and internal context. 

Such a comparison group can be part of a research strategy, which combines a theory based evaluation approach along with impact evaluation designs such as step-wedge design10 and simulation models11 or Bayesian methods12 13. Recently, realist RCTs have been proposed as an approach to evaluate complex interventions such as QI14. In such trials, the choice of comparison groups is likely to be related to the key intervention processes and functions, rather than the precise activities itself15.

The Salzburg Global Seminar program Better Health Care: How do we learn about improvement? is being held with the support of the USAID ASSIST project and the New Venture Fund. For more information on the session and to register, please visit the session page: www.salzburgglobal.org/go/565

Bibliography 

1. Rogers E. Diffusion of Innovations. New York: Free Press, 1995.

2. Greenhalgh T, Robert G, MacFarlane F, et al. Diffusion of Innovations in service organizations: Systematic review and reccomendations. The Milbank Quarterly 2004;82(4):581-629.

3. Reinersten JL, Gosfield A, Rupp W, et al. Engaging Physicians in a Shared Quality Agenda. IHI Innovation Series White Paper. Cambridge, Massachusetts: Institute for Healthcare Improvement, 2007.

4. Langley M, Nolan, Nolan, Norman, Provost. The Improvement Guide, 2nd Edition. Josey-Bass 2009.

5. Habicht JP, Victora CG, Vaughan JP. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol 1999;28(1):10-8.

6. White H. Theory-Based Impact Evaluation. International Initiative for Impact Evaluation 2009;Working Paper-3:1-20.

7. Leykum LK, Pugh JA, Lanham HJ, et al. Implementation research design: integrating participatory action research into randomized controlled trials. Implementation science : IS 2009;4:69.

8. Strange V, Allen E, Oakley A, et al. Integrating Process with Outcome Data in a Randomized Controlled Trial of Sex Education. Evaluation 2006;12(3):330-52.

9. Oakley A, Strange V, Bonell C, et al. Process evaluation in randomised controlled trials of complex interventions. Bmj 2006;332(7538):413-6.

10. Speroff T, Gerald TOC. Study Designs for PDSA Quality Improvement Research. Q Manage Health Care 2004;13(1):17-32.

11. Baio G, Copas A, Ambler G, et al. Sample size calculation for a stepped wedge trial. Trials 2015;16(1):354.

12. Brown C, Lilford R. Evaluating service delivery interventions to enhance patient safety. Bmj 2008;337:a2764.

13. Brown C, Hofer T, Johal A, et al. An epistemology of patient safety research: a framework for study design and interpretation. Part 4. One size does not fit all. Qual Saf Health Care 2008;17(3):178-81.

14. Bonell C, Fletcher A, Morton M, et al. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med 2012;75(12):2299-306.

15. Hawe P, Shiell A, Riley T. Complex interventions: how "out of control" can a randomised controlled trial be? Bmj 2004;328(7455):1561-3.