Feedback, on the Learning Curve to Improved Diagnosis
By Susan Carr | Senior Writer, ImproveDx
Experts have been calling for better use of feedback to improve diagnosis for 20 years,1-3 but most organizations are still at the starting gate. Using feedback for improvement—an appealing idea with nearly universal support—is a complex endeavor and challenging to implement.
There is general agreement around the concept that feedback is central to learning.2-4 Regarding diagnosis, the term “feedback” has traditionally referred to informing a physician that they made a diagnostic error of consequence to a patient. It has therefore been viewed with dread. Referring to emergency medicine, Gordy Schiff, MD, and Bob Wears, MD, PhD, note that receiving patient-specific feedback is unusual and wields more power than feedback received in more mundane circumstances:
…a question that to an outsider would seem innocent enough – “Remember that patient you saw last week?” – is enough to induce tachycardia.1[p24]
At most organizations, feedback about diagnostic errors continues to be rare; it is hard to both give and receive. What the term “feedback” means and how best to use it for learning is increasingly the subject of discussion and research.
Using feedback to develop diagnostic calibration
One way to frame feedback in the context of diagnosis is through the concept of calibration—the alignment of a clinician’s confidence in the accuracy of their decision-making with the actual accuracy of their past decisions.5-8 Calibration reflects self-knowledge. It involves having a sense of whether one’s decision will be right or wrong and taking action accordingly. A sense of uncertainty, for example, might prompt an earlier follow-up, or the need for subspecialty consultation. Calibration develops from knowing one’s own track record and having a feel for the limitations of one’s knowledge. That insight or acumen, which Gurpreet Dhaliwal, MD, calls “a prerequisite for achieving diagnostic excellence,”5[p737] should develop throughout a clinical career. Although the kind of feedback necessary for calibration is increasingly sought after, it is still rarely available.
Under- and overconfidence are miscalibrations
Lack of feedback about performance and outcomes contributes to overconfidence, which is a miscalibration of one’s ability. In a supplement issue of The American Journal of Medicine that remains a foundational resource for diagnostic improvement, Mark L. Graber, MD, and Etna S. Berner, EdD, note that unless otherwise informed, most clinicians will assume their diagnoses are correct and develop confidence that may not reflect their performance.9
Feedback helps counteract overconfidence, especially if it is timely and includes more information than a simple right or wrong. Feedback that includes enough detail to help adjust diagnostic decision-making could come in consultation with trusted colleagues, specialists and outside resources but remains uncommon and largely unstudied. Ashley Meyer, PhD, and Hardeep Singh, MD, who have written about feedback and calibration, note:
Determining the content of feedback to make it actionable might be the biggest hurdle for improving calibration in diagnostic decision making.6[p437]
Miscalibration can also skew the other way and result in under-confidence. Especially if delivered with blame, feedback about a misdiagnosis may cause the clinician to avoid risk in the future by ordering extra tests or by observing or admitting patients “just in case.” Dr. Dhaliwal notes that using feedback, especially knowledge of misdiagnoses, to calibrate decision-making involves learning from mistakes while avoiding defensiveness:
The greatest challenge that arises in tracking diagnostic outcomes is improving rather than reacting”4[pHO3]
Different kinds of feedback provide different lessons
A study performed at the University of Kansas Medical Center more than 10 years ago is an example of collaborative feedback in real time that offers hope for improving diagnosis and calibration.10,11 Through video case-conferencing, radiologists and pathologists, together with other team members, reviewed and discussed slides and imaging studies of breast core biopsies with a benign diagnosis. The conferencing resulted in a change of diagnosis in 34% of cases, with “major impact”11[p9] in 16%.
Mark Graber points to this study as an example of feedback that helps clinicians improve performance on a particular task; in this case, interpreting a pattern in breast imaging. He adds that other kinds of feedback, eg, autopsies that show a diagnosis was completely missed, can teach humility more than practical lessons:
You learn that you aren’t perfect, that you have to be more careful. In the future, this might result in taking time to stop and think, as opposed to just relying on System I as the routine approach to diagnosis. Autopsies have disappeared, and nothing has replaced them, so we no longer encounter those really powerful demonstrations of our being wrong. (written communication, February 2020).
How other industries use feedback
Many other industries are examining the use of feedback to improve performance, including the question of whether and how to focus on success as well as failure.
The lead article in a recent issue of the Harvard Business Review (HBR), “The Feedback Fallacy,” examines and rejects three common beliefs across industries about how people learn from feedback.
- People become aware of negative traits and actions they cannot see in themselves.
- People glean new information and skills they lack and do not know they need.
- People recognize levels of accomplishment to which they might not otherwise aspire.12
Those beliefs focus on deficits of self-knowledge, skills, and perspective. In the interest of promoting workforce excellence, the authors recommend a different approach: emphasize what works well for each individual versus calling attention to mistakes and failures.
By helping your team member recognize what excellence looks like for her—by saying, “That! Yes, that!”—you’re offering her the chance to gain an insight; you’re highlighting a pattern that is already there within her so that she can recognize it, anchor it, recreate it, and refine it. That is learning.12(p10)
A response published online argues for a blend of positive and negative feedback and recognizes the value of learning from mistakes as long as the feedback is non-judgmental.
Many now recognize that trying to improve diagnosis by focusing only on what has gone wrong is itself a mistake. Highlighting “wins” or achievements in order to figure out what works is a new concept in diagnostic improvement, one which many feedback pioneers think holds promise and demands further study.
Learning from what goes well
Experts in safety science emphasize the value of learning from what goes well. They point out that workers often add resilience to complex systems, such as health care, by constantly adjusting and adapting to dynamic and sometimes hazardous conditions. Humans make mistakes, but they also and more often make things safer through their actions. This approach to creating resilient workplaces (referred to as Safety II)13 goes beyond the focus of Safety I—finding and fixing problems—to study successful operations by deeply examining how people perform their work. It is, however, far more difficult to deconstruct complex processes, such as diagnosis, than it is to identify and study distinct adverse events as aberrations in the normal course of things. This way of thinking about success and errors views humans as:
…adapters who are mostly successful in coping with the complexities, ambiguities, uncertainties, and goal conflicts of complex work settings, but whose adaptations occasionally fail.14[p70]
To incorporate actionable feedback about diagnostic successes will require extensive study of decision-making and clinical reasoning.
How best to construct and deliver feedback
As already noted, to be most useful, feedback needs to reflect the complexity of the diagnostic process, including factors related to the clinician’s cognition as well as external elements (the system), and details about the case (the patient).5 In addition to the content of the feedback, how it is delivered and by whom also make a difference. Even under ideal circumstances, receiving feedback, especially involving errors of consequence, can be emotionally challenging.15 Clearly, organizations that want to implement effective programs for supplying feedback about diagnostic errors must first establish trust with clinicians. Meaningful feedback programs are sustainable only in a culture of safety, where learning is emphasized over blame and everyone plays a role in improvement efforts.
Geisinger sets an example
The Geisinger health system, based in Danville, Pennsylvania, is one of a small number of organizations with programs specifically designed to use feedback to improve diagnosis. Geisinger’s experience offers lessons and guidance for other organizations.16
Geisinger began taking a serious look at diagnostic errors in 2014 and established a working group in 2015, which became the Committee to Improve Clinical Diagnosis (CICD) in 2017. The CICD now includes leaders from all major disciplines and key stakeholders across the organization. Geisinger partnered with a Baylor College of Medicine-based research team led by Dr. Hardeep Singh to establish the Safer Dx Learning Lab in 2017 to develop innovative ways to address diagnostic error throughout Geisinger’s large integrated system.17
When the CICD learns about a case through organizational reporting mechanisms, it sends a letter asking the department head to review the case. That correspondence begins a multi-step process to provide the clinician or team involved in the case with non-punitive, constructive feedback and to disseminate the learning throughout the department. The CICD also asks the department to share, when relevant, actions taken to improve processes, as well as any system issues it has identified, so the CICD can facilitate further discussion. When appropriate and possible, the CICD relays the feedback to clinicians in other departments—emergency medicine, for example—who were involved with the patient at a different point in care. This feedback program operates throughout the Geisinger system, including physician practices and ambulatory clinics.
The CICD adds a system focus and research perspective to existing programs for using feedback to learn from diagnostic opportunities at Geisinger. As in many other institutions, the way feedback was used varied greatly by department or unit and often depended on the local culture. Divvy Upadhyay, MD, researcher-in-residence and co-investigator at the Safer Dx Learning Lab at Geisinger, stresses the importance of working with departmental leadership to support and improve existing feedback programs (personal communication, February 2020). Furthermore, Dr. Upadhyay says, “Geisinger’s program for improving diagnosis is grounded in a commitment to fostering a culture of transparency, openness, and learning — a vision laid out by Dennis Torretti, MD, chair of the CICD and associate chief medical officer at Geisinger Medical Center” (personal communication, February 2020).
The CICD and the Safer Dx Lab team worked together with Geisinger’s Center for Professionalism and Provider Well-Being to develop a toolkit to help department leaders, who are tasked with delivering feedback, to structure positive, supportive conversations with clinicians. Dr. Upadhyay observes:
The environment of medicine does not make it easy to give or receive feedback. The CICD intentionally uses the term “diagnostic opportunities” instead of “diagnostic errors” to improve the receptivity of the learning opportunity and to reduce the defensiveness that naturally comes associated with acknowledging errors (personal communication, February 2020).
Feedback about the feedback process
In discussion with the Safer Dx Learning Lab, the CICD recognized the need and opportunity to study the feedback process itself. How do we know who would be best to deliver feedback? Are we using the most effective method? To answer these and other questions, the Lab developed a survey instrument to collect reactions from those directly involved in feedback conversations. Dr. Upadhyay and others at CICD enlisted leaders from Geisinger’s departments of emergency, hospital, and community medicine to participate in the study. Directors and other department members who deliver feedback are trained and given recommendations for scheduling and structuring the discussions with clinicians. Following actual feedback sessions about diagnostic opportunities, the givers and receivers of feedback take the brief survey. Although not yet complete, responses to the survey have been positive.16
Feedback is like apple pie
Andrew Olson, MD, who is engaged in research (funded by the Gordon and Betty Moore Foundation) about using feedback to improving diagnosis and calibration, says that feedback is like apple pie. Everyone agrees it’s a good thing and says they want more. But feedback is not a simple solution. Organizations find that is a complex process with wide-ranging implications. It has become a trigger for studying the diagnostic process, learning styles, organizational culture, safety science, and more.
Gary Klein, PhD, is a research psychologist who pioneered the field of naturalistic decision-making, which studies how people make decisions in complex, high-stakes situations. He agrees that feedback is crucial for learning and points out that simple feedback is not always sufficient.
We have to make sense of feedback in order to learn from it.… For cognitive skills, we want feedback to change the way we think, not just add more facts into our memory.18[p173]
In challenging conditions that demand cognitive excellence, feedback must provide enough information about what happened and why to foster knowledge and improve thinking and decision-making. And, as with other activities in patient safety, using feedback effectively depends on a sound organizational culture that promotes learning.
Thank you to our reviewers: Jeannine Cyr Gluck, MLS, Mark L. Graber, MD, Andrew P.J. Olson, MD, Hardeep Singh, MD, Divvy Upadhyay, MD, and Lorri Zipperer, MA.
In this Issue:
Get ImproveDx in Your Inbox
Don't miss an article. Get the Society to Improve Diagnosis (SIDM)'s newsletter delivered to your inbox.