Comprehensive Guide to
Multi-Source Feedback
Four Stages of
Multi-Source Feedback
This comprehensive guide from Client Focused Evaluation Program (CFEP) Surveys supports clinicians and organisations across the health and care continuum that are considering the benefits of, or have adopted, CFEP Surveys’ Multi-Source Feedback (MSF) Tool. It helps clinicians and organisations develop a deeper understanding of how MSF aligns with the Medical Board of Australia’s Professional Performance Framework.1 It also explains how MSF can be used to inform professional development at the clinician level, and then more broadly at the organisational level, to drive continuous, data-driven quality improvement to ensure clinicians are working at the top of their scope and together contributing to the ongoing journey towards high-performing health care nationally.
Skip to stage
Stage 1 - Collecting objective and meaningful data
Objective data about a candidate’s observable workplace interpersonal skills and professional behaviours are collected through surveys from those interacting with the individual clinician. This stage also includes a self-assessment survey.
CFEP Surveys gives guidance to the MSF candidate and participating organisations when they’re identifying patient and clinical and non-clinical colleague reviewers. All identified reviewers must know the candidate and their clinical work well enough to give constructive and considerate feedback in a timely manner.
Candidates are also invited to nominate a supporting medical colleague, who would receive a copy of their aggregate report, to support debriefing and self-reflection (see Stage 2). CFEP Surveys recommends the candidate selects someone who is familiar with their work but not directly responsible for managing them or evaluating their performance. The supporting medical colleague role is an informal but essential peer support throughout the MSF process.
While concerns about the clinician selecting their own reviewers do exist, the clinician is generally in the best position to identify people who know them and have observed or experienced their practice. Further, given the formative nature of the MSF tool, the data are more useful if candidates identify reviewers who can offer a variety of perspectives and accurately assess their performance.
Figure 11 shows the process for the distribution and collation of the MSF tool instruments.
Distribution of the MSF instruments and their activies
Interpersonal Skills Questionnaire
To ensure patient confidentiality, a nominated administrator will distribute the surveys to a minimum of 40 active patients and invite feedback. The survey is easy to administer. It is available in online and paper formats. The online version is accessible via a unique URL or QR code, may be uploaded onto a portable device, and is directly submitted to CFEP Surveys via the secure portal. The single-sided paper questionnaire is designed for the patient to complete and return via sealed envelope. Both methods ensure patient anonymity and may be completed in about three minutes. For report validity, a minimum of 30 completed, valid questionnaires are required. The responses are subsequently provided to and analysed by CFEP Surveys and inform the development of the patient feedback results report. (Note: Ethics approval is not required for the survey since data collection is quality improvement-focused rather than research-focused.)34
Colleague Feedback Evaluation Tool
The names and email addresses of 15 nominated colleague reviewers (typically five doctors, five other clinical colleagues and five managerial or administrative staff) are collated and provided. CFEP Surveys subsequently invites each reviewer to complete the 15-minute online survey via CFEP’s secure portal. CFEP Surveys’ system automatically distributes reminders after one and two weeks, to secure a minimum of 12 responses for report validity.
Self-Assessment
Alongside the clinical feedback process, details of the MSF candidate are provided to CFEP Surveys which invites the MSF candidate to complete an individual 15-minute online self-assessment via CFEP Surveys’ secure portal.
Stage 2 - Analysing and reporting on MSF data
Once CFEP Surveys receives a candidate’s data, at the response rate required for data validity, it will analyse it by source (i.e. from patients or clinical colleagues and non-clinical co-workers). Analysis and reporting will ensure reviewer anonymity, noting that free-text feedback will be provided verbatim minus any personal identifiers.
Before completing MSF, it is worth confirming:
- Who will have access to the report?
- Who will see the results?
- Where the report will be stored?
- Who will have access to it in the future?
- When the candidate or participating provider organisation is completing the full MSF tool,
CFEP Surveys will give the candidate or participating provider organisation, supervisor or clinical educator (as per the arrangement) a comprehensive MSF report, with the feedback from all three instruments as well as comparator data where the items on surveys are identical.
For example:
For general practitioners
- All Practice Experience Program (PEP) reports are sent to the candidates’ registered training organisation or college only, which will then upload them to teh candidate’s RACGP portal
- For registrars, reports will typically be sent to the candidate and the registered training organisation/college via separate emails.
- For fellows (ACRRM, RACGP), a report will be sent to the candidate only. (Note: A copy won’t be supplied to the college.)
For specialists
- For registrars, reports will typically be sent to the candidate and the registered training organisation/college via separate emails.
- For participating fellows, a report will be sent to them directly.
- For participating international medical graduates (IMGs), a report will be sent to the relevant college for distribution.
Where the candidate nominates a supporting medical colleague, the candidate will provide a copy of the report to them.
CFEP Surveys provides specific information on the distribution of the report. The comprehensive MSF report gives a more rounded picture of the candidate’s performance, encompassing feedback from all three sources.
Stage 3 - Debrief and self-reflection
Research in medical education demonstrates that simply receiving a report on one’s performance is often insufficient to promote learning or a change in practice, even when gaps are readily apparent.35,36 While initial work in the development of MSF did not include facilitated debrief and self-reflection, the opportunity for candidates to discuss feedback is seen as critical to good outcomes from MSF. As such, facilitated debrief and self-reflection, ideally performed by a trained coach, is the third stage of the MSF process and the precursor to developing a targeted improvement plan at stage 4.8,37-43
Making a personal, professional or practice change in response to performance data is complex and benefits from three specific interventions:
- having a facilitated feedback conversation about the MSF report and data
- adopting a coaching approach when considering the need for change
- co-developing an annual action plan for professional development, growth and improvement.
An important and compulsory component of the MSF process is the candidate’s ability to meet with a trusted person to review the data and discuss the results. In recent implementation trials, 89 per cent of Royal Australian College of Physicians candidates agreed or strongly agreed that debrief was a valued component of the process.48
Debrief and self-reflection goals include to:
- inform the co-development of an annual action plan for continuing professional development
- normalise the process of professional review and reflection as part of a supportive whole-system learning culture encompassing professional bodies and provider organisations
- (and, for clinicians who are excelling) focus on an aspirational change or improvement they may
wish to make.
Reflective learning is an essential component of professional practice. It involves considering the results and thinking about the experiences of patients and colleagues retrospectively in order to learn from them. The debrief helps the candidate to ‘unpack’ their MSF report; where possible this should occur as soon as they receive the report.
But debrief and facilitated reflection can be challenging, even when results are positive. It requires leadership, adopting a coaching approach, and effective interpersonal skills based on trust and respect. Research has identified characteristics of the feedback conversation that enhance its effectiveness (Figure 25).49-52
Facilitated debrief and reflection help the individual candidate to:46,49-54
- be more aware of the experience of care they provide
- explore reactions to the feedback and better understand what it means to them
- translate new information into insights and knowledge about strengths
- identify and implement areas and priorities for personal development, growth
and practice improvement.8,37,42,43
When conducting facilitated debrief and fostering self-reflection, the facilitator or coach needs to pay attention to the skills required for handling both process and content.49,53
Process skills include
- reviewing the purpose of the program and session goals with the clinician
- developing the relationship throughout the session
- ensuring familiarity with the data
- using communication micro skills to explore reactions to the results, clarify understanding and provide encouragement through active listening and open questioning
- promoting reflection and self-assessment by bringing blind spots into focus
- being flexible about the content to be discussed.
Content skills
Content skills include collaborating to make sure the clinician is engaged in and committed to the discussion, goal setting and developing anticipated outcomes, and creating a tailored action plan and a follow-up plan to monitor progress and ensure accountability.
Facilitated conversation should focus on the data in each report as well as triangulated data across colleague (clinical colleagues and non-clinical co-workers) and self-assessment reports, or benchmarks with other participating clinicians. For example, clinical colleagues may provide high ratings for punctuality and reliability but non-medical co-workers might rate those items lower. The discrepancy may provide an opportunity to ask the clinician about the difference.
Considering the candidate’s self-assessment report creates space for reflective discussion, especially when the self-assessment scores differ from their reviewers’ scores.55 A difference in results creates an opportunity for further exploration.
For clinicians who are excelling, this discussion could focus on an aspirational change or improvement they may wish to make personally, professionally or organisationally.
Note: The debrief happens before the reflective period and the personalised action plan are completed.
Options for supported debrief and self-reflection
Formal debrief Process
This is a highly recommended optional service available to any clinician completing the MSF tool in full or part (i.e. patient feedback or colleague feedback and self-assessment). It includes a one-hour debriefing phone call or videoconference between the candidate and a trained MSF facilitator or coach (i.e. an experienced member of the CFEP Surveys debrief team).
CFEP Surveys’ formal debrief approach adopts the Relationship, Reaction, Content, Coaching (R2C2) feedback model,41,46,47 a strength-based approach to facilitation and coaching and action planning.
This R2C2 model is founded on three theoretical perspectives – humanism, informed self-assessment and the science of behaviour change, and includes four phases:46,47
This is a debrief conversation between the candidate and their nominated supporting medical colleague, supervisor or medical educator in which both parties discuss the report and its findings.
Theory and research inform each phase, to guide the feedback conversation and provide open questions to promote self-reflection, self-critique and self-direction. The intention is that the facilitator or coach uses the model iteratively, to explore the sections of the report that are most meaningful to the candidate and to the facilitator or coach, and then to coach the individual through the process of co-developing a purposeful action plan.
Facilitated discussion is focused on the data in the MSF report about each of the three MSF roles: communicator, collaborator, professional. In upholding the formative nature of MSF, the facilitator or coach uses a coaching approach where discussions focus on the clinician’s relative strengths and improvement areas within each role.
Coaching in the MSF context is considered: ‘a one-to-one conversation focused on the enhancement of learning and development through increasing self-awareness and a sense of personal responsibility, where the coach facilitates the self-directed learning of the coachee through questioning, active listening, and appropriate challenge in a supportive and encouraging climate’.56
Some candidates might find considering results challenging. It is important that a skilled facilitator or coach conducts the debrief and understands, plans and prepares for challenges, such as:
- how to manage the session when the clinician hasn’t reviewed their data
- how to manage the conversation when the clinician is resistant to change
- how to work with a clinician who is clearly upset by the feedback, or alternatively considers themself an overachiever.
Facilitated coaching conversations may include open-ended questions, such as:40,42
Did you focus on particular sections of the feedback report? If so, please describe which sections you focused on and why.
- What did you learn that was expected or unsurprising? Why was it expected?
- What did you learn that was unexpected or surprising? Why was it surprising?
- What did you find that seemed noteworthy or important? Why was it important?
- Is there a gap between the care you want to offer and what the report suggests?
Completing the MSF tool and this reflection process can contribute to CPD requirements for the candidate and their supporting medical colleague.
Informal Debriefing
This debrief conversation is typically conducted between the MSF candidate and their supporting medical colleague – a trusted medical colleague the candidate appoints to support them in this process. Informal debrief allows opportunity to consider and discuss the report, reflect on results, and establish insights to inform subsequent action planning. In some cases, those clinicians on training pathways receive debrief support from their clinical supervisor or medical educator. Alternatively, a medical division or departmental head of service may support debrief when a participating organisation sponsors the MSF program.
CFEP Surveys does not specify the format for informal debrief and the training provider or sponsoring organisation uses it at their discretion. CFEP Surveys recommends adopting a skilled coaching approach to focus on the clinician’s priorities for change, goals for improvement, and co-creating an action plan based on the performance data.57
When conducting informal debrief, we recommend the MSF candidate and their supporting medical colleague establish a compact, outlining those principles and agreements they will uphold during the
MSF process; for example:
Confidentiality
All data and reports will remain confidential, and discussion during the informal debrief will be conducted in strictest confidence to allow openness and honesty
Respect
Respectful consideration will be given to feedback from patients and colleagues, recognising their unique position, the spirit in which the feedback was provided, and a focus on learning to inform personal development, growth and improvement.
CFEP Surveys may provide resources and templates to support an informal debrief, including a comprehensive guide and instructional videos following the R2C2 method.
Self Reflection
The MSF program aims to normalise the process of professional review and reflection as part of a supportive whole-system learning approach encompassing professional bodies, provider organisations and health care professionals.
The feedback report will also include guidance on conducting a reflective exercise to inform the development of a personalised action plan, plus tools and templates for considering results by source (i.e. patient and colleague) and results overall.
The recommended time for completing the self-reflection exercise is 4–6 weeks after the debrief.
The supporting medical colleague or medical educator may participate in this activity, informally reflecting on the candidate’s results and providing additional insight and support to them.
Completing self-reflection helps the candidate to prioritise areas for personal development, growth
and improvement and develop the action plan. Once the candidate submits the completed reflective exercise to either the participating medical college or CFEP Surveys, they can receive CPD recognition.
Stage 4 - Action planning and CPD allocation
The MSF process culminates in the co-development of goals for personal development, growth and improvement, and an evidence-informed action plan.28,54,58,59
Co-development is critical to the action plan’s success. The candidate must make the major contribution and feel they own their personalised action plan, while the coach contributes their experience and knowledge and facilitates the process and content development. The role of the formal debrief coach is not to provide answers or solutions, but to help prioritise areas for improvement, goal setting, opportunities and strategies for professional development, growth and improvement and to help document agreed changes in an annual action plan.
Research suggests action plans and anticipated outcomes are more likely to be achieved when co-developed, as opposed to being developed by only the candidate, facilitator or coach, or medical director.28,54,58,59
MSF action planning and CPD allocation
During the action planning process, the facilitator or coach may also connect the MSF candidate with learning resources and system supports to help them achieve their improvement goals. When collaborating with professional and peak bodies, education and training institutions or CPD units, CFEP Surveys will provide a list of such resources.
Informed by MSF results and insights from the facilitated debrief session, the action plan describes the changes the candidate intends to make in the short-term (over the next six to eight weeks) and across the medium term (over the next six to 12 months). It also captures initial activity required for longer term goals (12-plus months).
CFEP Surveys’ action plan template outlines a structure in which the candidate can capture information about:
- resources they need to support the changes
- the enablers and barriers to change
- what success looks like
- how and when they will know they have achieved success.
An action plan example, articulating one meaningful goal, is provided in Figure 31. CFEP Surveys suggests the candidate creates up to three targeted goals to work on during the annual MSF cycle. They might struggle to achieve any more in the time available.
We highly recommend the candidate and the supporting medical colleague hold a follow-up facilitated coaching session one to two months following development of the action plan, to reflect on the impact of the changes made in the short-term.
The true value of the MSF process is realised as candidates embed learnings into improved patient experience of care, clinical practice, and collegiate relationships.
CFEP Surveys provides templates to support short-term review processes. Some participating organisations, colleges and training pathways, and CPD homes will require the candidate to return their completed action plan before they will recognise completion of the MSF process. In other instances, the candidate can return the action plan to CFEP Surveys via the secure portal.
CFEP Surveys provides a certificate of completion, demonstrating self-evaluation, performance review and outcomes measurement-related activity (as appropriate), as per the MBA’s CPD standards and associated requirements.
The MSF process may conclude in this annual round, with the candidate and supporting medical colleague logging CPD activity with their professional and peak body, college, CPD home or registered training organisation, or sponsoring organisation as appropriate.
Note: Professional development and improvement-related activity undertaken in line with the action plan may attract additional CPD recognition. It is important that candidates log and
self-evaluate these personalised professional development, growth and improvement activities.
CFEP Surveys’ MSF tool provides an annual cyclical process of reviewing performance, measuring outcomes, and self-evaluation. Once the candidate has completed the action plan, CFEP Surveys recommends they repeat the virtuous cycle of MSF to engender a culture of excellence, demonstrate leadership, and sustain their personal development, growth and improvement.
Section 2: The Four Stages of Multi-Source Feedback
- Stage 1 – Objective and meaningful data collection
- Stage 2 – Analysis and reporting
- Stage 3 – Debrief and self-reflection
- Stage 4 – Action planning and CPD allocation
Section 3: How to Implement Multi-Source Feedback
Introducing MSF and applying the MSF tool requires organisational commitment and resourcing. Developing a robust implementation plan with a feasible timeline supported by an effective engagement and communication plan is essential for success.
Chapter 3 clearly outlines the key considerations when introducing an MSF program and provides suggestions for implementation strategies.
Section 4: Multi-Source Feedback Evidence Base
While the content of an MSF tool may appear to be relatively simple, there is a substantial amount of research, time, money and effort invested in producing these ‘fit-for-purpose’ tools. A wide range of research has been conducted on the use of MSF in various clinical settings that support the effectiveness and impact MSF has on quality improvement processes.
Section 5: Enablers and Success Factors of MSF
Lorem ipsum
Section 6: Limitations of Multi-Source Feedback
Chapter 1 summaries Multi-source feedback covering:
Appendices
Appendix A: Glossary