Educational Resources on Research Frameworks and Feedback

Faculty development supporting research execution

Health professions education research requires the ability to turn questions arising in educational practice into research problems, collect and interpret data, and explain results to peers. Conceptual and theoretical frameworks guide these choices. Feedback on presentations offers an opportunity to reconsider how research is explained and to plan subsequent work. Faculty development for researchers can combine instruction in research methods with activities that examine specific research cases and draw on expert advice.

Workshop on applying conceptual and theoretical frameworks to research design

The educational resource on using conceptual and theoretical frameworks presented a workshop designed to help novice health professions education researchers select frameworks and apply them throughout the research process. The research team defined theory as an explanation of how and why relationships between things arise. A theoretical framework referred to the implications of that theory for research design. A conceptual framework referred to the central concepts and conceptual heritage of the field in which a research problem is situated. The workshop, developed by the Medical Education Scholarship Research and Evaluation (MESRE) under the AAMC’s Group on Educational Affairs (GEA), required no prior knowledge of theory. It used a study of narrative feedback in basic medical education as a shared case. The workshop combined role-play videos with small-group and large-group discussions to examine how applying different theories to the same research case changes the research problem and design. The role-play depicted a mentor and a mentee selecting a theory and interpreting results. The introduction provided an example of applying the theory of planned behavior. A 25 to 30-minute small-group activity involved refining the research problem and specifying participants, the setting, data sources, and data collection strategies using situated learning theory (SLT). The workshop was initially developed as a 75-minute session. Its second delivery indicated that 60 minutes was sufficient to address the learning objectives. At least 1 facilitator per 10 participants and a 45 to 60-minute preparatory meeting were recommended.

The workshop was delivered in person at the Learn Serve Lead: the AAMC Annual Meeting in November 2019 and online at the GEA Regional Spring Meeting in April 2021. Participants included medical school staff, faculty, and administrators, with 55 attending in person and approximately 150 attending online. The post-workshop survey received responses from 26 (47%) and 34 (23%) participants, respectively, for a total of 60 respondents (30%). Participants rated their ability on a 3-point scale: ‘could do before workshop’, ‘could do as a result of workshop’, and ‘cannot perform’. Four open-ended questions asked what participants had learned, how they planned to apply it, and how they experienced the workshop’s delivery. Rates of learning objective attainment were calculated after excluding responses indicating that participants could already perform the task before the workshop. In this analysis, 80% (32 individuals) reported that they had learned to distinguish between conceptual frameworks and theories, and 86% (32 individuals) reported that they had learned to apply conceptual and theoretical frameworks to research problems. The proportion reporting that they could analyze how framework selection affects research design was 79% (34 individuals). Another 68% (27 individuals) reported that they could evaluate research results in light of the framework. Among the 34 online respondents, 88% reported feeling engaged in the overall presentation and 80% in the small-group discussion.

In the narrative responses, 12 individuals identified small-group and breakout room activities as strengths, 9 identified expert facilitators, and 7 each identified role-plays and distributed materials. Areas for improvement included variation in small-group experiences (7 individuals), the need for additional definitions of terms (4 individuals), and the complexity of the topic and requests for expanded materials (2 individuals each). Experiences varied according to participants’ understanding of theory, research experience, comfort with speaking, and the facilitator’s management of the discussion. Proposed adjustments included distributing SLT materials in advance and creating visual diagrams with participants. Another proposal was to explain at the outset how selecting different theories changes the research problem. Given the use of post-workshop self-reports and the overall 30% survey response rate, the results show respondents’ perceptions of their learning. Pre- and post-workshop evaluations can be used together to examine additional learning among participants with prior ability.

Feedback linking conference presentation evaluation and research development

The educational resource on conference presentation feedback combined procedures for evaluating research presentations with support for educational researchers’ development and participation in a community of practice. The Central Group on Educational Affairs (CGEA) piloted formative and summative evaluation forms at a regional conference in 2014 and continued using them thereafter. Summative evaluation was designed to select Best Presentation Award winners through numerical ratings of presentation content and delivery. Formative evaluation supported research development through individual narrative feedback. In the fall of 2020, preparations for the 2021 online conference included adding sound quality and appropriate lighting to the summative evaluation. The formative evaluation and basic operating procedures were retained. Before the event, presenters received preparation guides and blank evaluation forms. Each group of 4–5 presentations was assigned 2 summative evaluators and 1 formative evaluator. Assignments took account of evaluators’ expertise in the presentation topics, potential conflicts of interest when summative evaluators and presenters shared an affiliation, and the presentation schedule. About 2–3 days after the event, winners were determined by averaging the scores of the 2 summative evaluators. Each presenter received an individual email with a PDF combining the two types of feedback. Checklists, schedules, email templates, and evaluation forms were provided to support the use of this procedure at other events.

The program evaluation used communities of practice as a conceptual framework to examine participants’ experiences, community formation, and the usefulness of feedback. Four focus groups were conducted on Zoom in June 2021, with a total of 8 participants: 4 evaluators and 4 presenters. The research team conducted a conventional content analysis, independently reviewing the recorded and transcribed data and post-meeting notes before discussing major themes and subsequent action items. Over 8 years, 102 presentation evaluations incorporating formative and summative assessments were completed, and of 62 volunteer evaluators, 19 (31%) participated for more than 2 years. During this period, the 2020 spring conference was canceled due to COVID-19. In the focus groups, presenters described the narrative feedback as specific and actionable. They found it useful for improving presentations and subsequent research and reported that it motivated them to prepare for journal submission. Evaluators had opportunities to learn about research at other schools and participate in presentations and discussions. They viewed activities that helped researchers develop as a form of academic service.

Suggested operational improvements included orientations and repeated guidance to encourage participants to review advance instructions. Explanations of the relative weight given to presentation skills and research content in evaluations were also suggested. Formative evaluators found it time-consuming to write feedback while listening to presentations. Proposed solutions included drafting feedback in advance or having 2 evaluators take turns. Separating the roles of evaluators and session moderators at the 2021 online conference was viewed positively because it helped with time management. Other suggestions included pairing senior and junior researchers as evaluators and having evaluators share written feedback with one another. The focus group results show how useful participants found the feedback and their patterns of participation.

Reading notes

The post-workshop self-reports and conference focus groups capture participants’ perceptions of learning and usefulness. Assessing changes in research practice requires examining participants’ subsequent experiences of applying what they learned. From the perspective of learning transfer in HRD, this can inform research that tracks how problems refined in the workshop and advice received after presentations are incorporated into research proposals, data collection, and manuscript revisions.

References

• Rougas, S., Berry, A., Bierer, S. B., Blanchard, R. D., Cianciolo, A. T., Colbert-Getz, J. M., Han, H., Lipner, K., & Teal, C. R. (2022). Applying conceptual and theoretical frameworks to health professions education research: An introductory workshop. MedEdPORTAL. https://doi.org/10.15766/mep_2374-8265.11286
• Bierer, S. B., Cianciolo, A., Han, H., & Hanson, J. (2022). A systematic approach to provide feedback to presenters at virtual and face-to-face professional meetings. MedEdPORTAL. https://doi.org/10.15766/mep_2374-8265.11288