Comprehensive Guide to
Multi-Source Feedback
Evidence Based
Multi-Source Feedback
While the content of the MSF instruments – the Interpersonal Skills Questionnaire, the Colleague Feedback Evaluation Tool and the Self-Assessment – may appear to be relatively simple, there is a substantial amount of research, time, money and effort invested in producing these ‘fit-for-purpose’ tools. For example, in the United Kingdom, the General Medical Council committed over GBP500,000 to ensure the tools used for their MSF program were robust, reliable and validated so data was meaningful for doctors.
A wide range of research has been conducted on the use of MSF internationally in various clinical settings. A review of eight systematic reviews demonstrated the evidence base supporting the statistical and psychometric properties of MSF is sufficient to support using the validated tool in the clinical setting.9 The structural validity of the MSF tool has been tested, confirming that MSF provides a reliable
method of performance assessment and a feasible method of assessing clinician performance in terms of response rates, time and costs.46 More recently, a critical review of MSF and its psychometrics noted that publications over the past 50 years or more in the business and health literature support MSF for quality improvement purposes.29
Specific research validating the use of the MSF in the clinical setting covers, for example:
Research validating the use of clinical MSF
Validity and reliability
MSF systems are deeply established in industry, with 360-degree type feedback routinely used a workplace assessment tool to:
- develop insights into individual strengths and opportunities for improvement
- enhance cultural change
- produce summative assessment of performance
- evaluate potential (e.g.g careers advice or selection)
- enhance team effectiveness
- identify training needs for the organisation.
Original work on the development of MSF tools for physicians was undertaken in the United States4,5 and later developed over several years by the College of Physicians and Surgeons of Alberta’s (CPSA) Physician Achievement Review (PAR) program6, and the Medical College of Canada.7,8
MSF is increasingly adopted within continuing professional development and regulatory frameworks worldwide as a method to assess medical performance and quality-assure clinical practice.9 The validity evidence for MSF used within medicine is well established, and MSF is now used in health systems across Canada, Australia, the Netherlands, New Zealand, the United Kingdom, and the United States, with interest growing in other countries.
Feasibility
Feasibility of implementation of the MSF tool is high as assessments take a short time to complete, are cost-effective and achieve high response rates.37,60-62 .
Scoring
Comparison studies with other MSF instruments are favourable.73 In addition, comparison with other workplace-based assessment demonstrates that MSF provides a valid representation of clinician performance.61,62,64
Comparison
Systematic reviews,9,61,63 meta reviews,71 scoping reviews72 and critical reviews29 have been undertaken to determine the evidence for using MSF scores in both medical practice and medical education settings.
Consquences and change
Examinations of clinicians’ use of the feedback to make changes following MSF are overall positive, with mixed results about the likelihood of change following negative feedback or where feedback is inconsistent with a clinician’s own perceptions of their performance. This prompted the recommendation that facilitated feedback be provided. Repetitive feedback and the ability to reflect on the results, together with facilitated feedback, increase the likelihood of change.35,62-65,74-77
Barriers and enables
Explorations of the barriers and enablers to using the data to make changes in practice38,55,78
Facilitated feedback
Examinations of the utility of having a facilitated reflective feedback discussion to discuss the data and the report, the clinician’s reflections on them, the strengths and opportunities identified by them and plans to use the data for improvement46,47,55
Data
Assessments of longitudinal changes in data for clinicians who participated in Physician Achievement Review on more than
one occasion66,69
Performance
Relationship with the Medical Board of Australia’s Professional Performance Framework79 and comparisons of clinician performance13,66
Quality
Measuring the quality of hospital doctors through colleague and patient feedback57,80
Academic appointments
An examination of the association between the original PAR ratings and academic appointments and teaching81
The work has also led to systematic reviews of MSF. For example, exploring the role self-assessment plays when interpreting performance feedback has led to the development of a robust model for facilitated coaching feedback about performance data (i.e. the R2C2 model, discussed in Section 5).46,47,74
CFEP Surveys actively shares research findings and provides opportunities for linkage and research exchange, including via:
CFEP Surveys’ website (www.cfepsurveys.com.au)
MSF mailing list (to connect and receive regular updates email: [email protected])
regular and ongoing conversations with key stakeholders
an annual national MSF virtual seminar, which includes international guests
and industry leaders.
Section 2: The Four Stages of Multi-Source Feedback
- Stage 1 – Objective and meaningful data collection
- Stage 2 – Analysis and reporting
- Stage 3 – Debrief and self-reflection
- Stage 4 – Action planning and CPD allocation
Section 3: How to Implement Multi-Source Feedback
Introducing MSF and applying the MSF tool requires organisational commitment and resourcing. Developing a robust implementation plan with a feasible timeline supported by an effective engagement and communication plan is essential for success.
Chapter 3 clearly outlines the key considerations when introducing an MSF program and provides suggestions for implementation strategies.
Section 4: Multi-Source Feedback Evidence Base
While the content of an MSF tool may appear to be relatively simple, there is a substantial amount of research, time, money and effort invested in producing these ‘fit-for-purpose’ tools. A wide range of research has been conducted on the use of MSF in various clinical settings that support the effectiveness and impact MSF has on quality improvement processes.
Section 5: Enablers and Success Factors of MSF
Lorem ipsum
Section 6: Limitations of Multi-Source Feedback
Chapter 1 summaries Multi-source feedback covering:
Appendices
Appendix A: Glossary