The Resident Audio Recording Project

Abstract

Southern Illinois University Department of Internal Medicine operated the Resident Audio Recording Project (RAP), which recorded and transcribed resident outpatient communication, and provided multidisciplinary panel consensus evaluation and mentor feedback. In the data of the first 3 cohorts for academic years 2009–2012, average RAP scores were 1.6 for PGY-1, 2.0 for PGY-2, and 2.3 for PGY-3, and teachback usage was also higher by postgraduate year. The authors concluded that the RAP made communication observable and improvable, but determining the score increase as the effect of the RAP is difficult with the presented comparisons alone.

Setting and Problem

Communication with patients and caregivers is a competency evaluated under the Accreditation Council for Graduate Medical Education’s Internal Medicine Reporting Milestone No. 20. This competency is difficult to interpret and score because communication is fluid and subtle. Outpatient care lacked a consistent system for evaluating resident communication and providing feedback.

The Southern Illinois University Department of Internal Medicine developed the Resident Audio Recording Project (RAP). The RAP used recordings and transcripts of actual outpatient interviews as a shared basis for evaluating resident communication and providing feedback.

Intervention

The RAP is a collaborative, longitudinal evaluation project that began in 2008. Audio recording was selected as a cost-effective way to track residents’ clinical communication. The evaluation panel consisted of 3 to 5 faculty and university staff members from medicine, education, communication, and conversational analysis. Each of three panels worked with a designated group of residents throughout their 3 years of residency.

Each resident participated in 1 RAP session every year, completing 3 steps: recording a patient interview, receiving a panel evaluation, and receiving mentor feedback.

Step	Process
Patient interview recording	The resident recorded an outpatient interview with a consenting patient.
Panel evaluation	The interview was transcribed and provided to panel members along with the original recording. The recording was played and the transcript was reviewed during a meeting, and a consensus evaluation on the effectiveness of the interview was derived through discussion after independent evaluation by each member.
Self-evaluation and feedback	The resident conducted a self-evaluation using the transcript and their recording for 1 hour prior to the mentor interview. The resident then met with the panel mentor and received positive and negative feedback on various aspects of communication.

The report estimated that faculty and panel members spent approximately 4 to 6 hours on 1 recording. This included 90 minutes of resident time and 90 minutes of attending physician time. The report did not explain how time was totaled across these roles.

RAP scores ranged from 0 to 4 points for each of the 23 communication categories. A score of 0 was unsatisfactory, and 4 was exemplary. The available text did not specify the names of the 23 categories, the criteria for intermediate scores, or inter-rater reliability.

Outcomes to Date

Since 2008, 145 recordings had been evaluated and used to provide feedback. Residents reportedly responded positively to the feedback. Observations over 5 years indicated improvement in communication competency from internship to the final year of residency. The report did not specify the tool used to measure acceptance of the feedback.

The comparison across residency years used data from the first 3 cohorts, spanning academic years 2009–2012, with 1 recording per resident per year. The available full text contained only a table title, with no numerical values in the table. The numbers of residents and recordings included in each year’s analysis were unspecified.

Indicator	PGY-1	PGY-2	PGY-3	Test reported in original text
Average RAP score	1.6	2.0	2.3	Kruskal-Wallis H = 36.97; df = 20; P = .01
teachback usage rate	16%	45.7%	50%	Chi-square = 6.3; df = 2; P < .05

Average RAP scores increased from 1.6 for PGY-1 to 2.3 for PGY-3, and teachback usage rates were 16% and 50%, respectively. The increases in both indicators were reported as statistically significant. The reported test statistics and degrees of freedom match those in the original text.

The report described teachback as a technique for checking patient understanding. Changes in resident communication were also reported in inpatient care. The report did not provide separate measurement methods or outcome values for these changes.

Although scores and teachback usage rates differed across residency years, the absence of a comparison group makes it difficult to distinguish changes associated with residency from the effects of the RAP. The text leaves unclear how the analysis accounted for repeated measurements of the same residents.

Audio recording captures nonverbal vocal cues such as pitch and speech rate, but provides less information about body language or bedside manner than video. The transcript was described as a tool accepted by both evaluators and residents.

Conclusions

The report concluded that the RAP made communication competency visible within the program. Residents reportedly came to see communication with patients as a performance that could be reviewed and improved. The evaluation process continued to evolve as panel members discussed the criteria for good communication.

These conclusions were presented without results from a separate instrument measuring perceptions of competency or changes in departmental discussions.

Reading notes

From an HRD perspective, the RAP links self-evaluation and feedback through shared recordings and transcripts, with the 1 hour before the mentor interview reserved for residents to assess their own performance. The degree of agreement between residents’ self-evaluations and the panel’s consensus evaluations was not reported. The teachback usage rates of 16%, 45.7%, and 50% identify communication behaviors that differed across residency years more specifically than the average RAP scores across 23 categories. The evaluation process also evolved. It remains unclear how much score differences over 3 years reflected changes in performance or evaluation criteria. The RAP was described as cost-effective, with approximately 4 to 6 hours required for 1 annual evaluation. The paper did not compare costs with those of other evaluation methods.

References

Papireddy, M. R., Han, H., Hingle, S. T., Koschmann, T., Ferguson, J. A., & Sandstrom, S. (2014). The resident audio recording project: A 3-step process to improve clinical communication skills. Journal of Graduate Medical Education, 6(2), 367–369. https://doi.org/10.4300/JGME-D-14-00043.1