Fellow, James Ndirangu, summarizes the sessions, questions and discussions of day three
On the third day of the Salzburg Global Seminar session Better Health Care: How do we learn about improvement? Fellows expanded work on their case studies and considered how they can improve rigor, attribution and generalizability.
The day started with a clarification of what is meant by ‘intervention’ within the context of quality improvement. Dr Massoud stressed that our focus is on the ‘how’ we implement what we know works, or bridging the “know-do” gap, and not on testing ‘what’ works to improve clinical outcomes. He further clarified the purpose of the meeting, as well as the expected product from this meeting. By the end of this meeting, we need to leave with better ways to do our QI work and remove or minimize questions around results we obtain. This may be a framework, guidance, or key principles document, to help us improve the scientific rigor in the implementation of QI, thereby marrying the fields of improvement and research.
The morning session focused participants to address two questions within their case study groups, namely: What are the lessons learnt on improving the rigour, generalizability and attribution? And what would you do differently to strengthen the rigour, generalizability and attribution in your case studies? Key recommendations that came out of that discussion suggest that we need to define the customer and what information they need. We should be thinking about the use of the findings and allow these facts to guide the scope of an evaluation. In the design, we should describe the context and intervention in more details, construct adaptive logic models, and evaluate whether or not we are using the right indicators, with the right monitoring mechanisms, and explore the use of existing data. When thinking about scale up, a key recommendation was to use a structured approach in the design, such as the 10 accountability questions. Further, a recommendation was made that evaluation be considered from the beginning of a QI work.
The mid-morning session allowed participants to reflect on causal thinking and causal pathways using examples such as the link between smoking, lung cancer, and mortality to demonstrate where our focus is for QI. The presenter, Dr. Mittman, suggested that studying improvement is different than studying clinical phenomena, given the tendency for the intervention in the former to be less consistent and for the main effect to be more important than contextual effects. In improvement context is much more important. Given that with QI the focus is on the ‘how’ does it work, he suggested that our goal is to provide ‘insights’ and ‘guidance’ to improve research and evaluations of QI. The discussions that followed raised the point that core components of research are also core components of the improvement process. We were reminded that with QI we are trying to do two things, 1) what’s the best way of improving processes, and equally important, 2) what’s the best way of learning.
Dr. Gareth Parry suggested that “All models are wrong but some are more useful than others”. How do you determine which models work and how they can be adapted for scale up in specific contexts or in a broader range of contexts? There are three phases of evaluating models and frameworks for adoption or scale up. The phases include innovation phase (what models exist), testing phase (do they work or can they be amended to work in specific contexts) and scale up phase (how do they spread in different contexts). Dr. Parry further explained the four levels of the Kirkpatrick Evaluation of Learning Framework. What are the participants experiences? What did the participants learn? Did they modify their behaviour? Did the organization improve their performance?
The afternoon session evaluated three studies for rigor and generalizability. Discussions centered around the ‘negative’ study findings that were in conflict with previous studies, unexpected primary outcomes, potential biases and description of context and interventions. There was a recommendation to plan a pilot before RTC trials and to consider contextual variations across different countries when considering adoption of interventions.
