AMEE Guides to Medical Education Research Methods

From research questions to data interpretation

Medical education research methods provide procedures for framing questions about learning and practice that can be investigated through data. Measuring confidence or satisfaction with a questionnaire involves examining both the scope of the concept and how respondents understand the items. When clinical experiences are recorded through audio or photos, interpretation depends on when recordings are made, the conditions of participation, and how data are transferred and stored.

Developing questionnaire scales connecting constructs and response processes

AMEE Guide No. 87 applies social science techniques for questionnaire design to medical education, presenting a procedure for developing self-administered scales that measure a single construct with multiple items (Artino et al., 2014). Of the original articles in Medical Teacher from 2011–2012 reviewed in the study, 37 (24%) used questionnaires, as did 75% of research articles in the Journal of Graduate Medical Education during the same period. Questionnaires are useful for investigating concepts that are difficult to observe directly, such as attitudes, beliefs, and satisfaction, or behaviors that respondents can recall and report. Development begins by assessing whether a questionnaire is appropriate for the research question and reviewing constructs and existing scales in the literature. Confidence in cardiac auscultation and confidence in physical examination require different measurement scopes and items. Reliability and validity are properties of questionnaire scores and their interpretations, and validation is an ongoing process of accumulating evidence for a specific population, context, and purpose.

The seven steps of questionnaire development are a literature review, interviews and focus groups, synthesis of the two data sources, item development, expert validation, cognitive interviews, and pilot testing. Interviews and focus groups help establish how the target population understands and expresses the construct. These findings are integrated with definitions from the literature. The guide recommends asking about one thing at a time in respondents’ own language. Response anchors should suit the construct, such as confidence, frequency, or satisfaction, rather than a general degree of agreement. The authors describe development as a process that anticipates revisions and deletions, giving the example of drafting 15 items to arrive at a final set of 8 items. For narrow constructs, 6–10 items are generally proposed. Response categories are stable when there are 5 or more, and exceeding 7–9 categories rarely improves reliability. Expert validation evaluates the clarity, relevance, and comprehensiveness of the items. Cited recommendations for the number of experts range from 6–10 to a maximum of 20, with larger panels intended to establish clearer consensus. Cognitive interviews examine comprehension, memory retrieval, judgment, and response selection, typically involving 10–30 participants. In small-scale medical education research, they can be conducted with 5–6 individuals, with attention to sample bias. Think-aloud protocols and verbal probing help establish how respondents understand and answer the items.

Pilot testing involves the target population and uses the web or paper format intended for the survey. The range, variance, mean, standard deviation, and histogram for each item are reviewed, along with correlations between items and composite scores. The internal structure and relationships with other variables are also analyzed. The authors recommend confirmatory factor analysis for scales designed to measure a single construct. They also present the view that exploratory factor analysis is appropriate for new scales. Since Cronbach’s alpha is an index of internal consistency, it is interpreted after dimensionality has been examined through factor analysis. Its value can increase with the number of items. Weighted or unweighted averages that retain the response scale are proposed for composite scores. Relationships with other variables are illustrated by the expectation that suturing confidence would correlate positively with the number of successful sutures and negatively with procedural anxiety. Because the guide focuses on item and scale design, conducting a survey also requires planning for sampling, response rates, survey media, ethical review, and data management.

Design and management of smartphone data collection

AMEE Guide No. 130 draws on the experience of developing a research plan for longitudinal narrative inquiry using smartphones, synthesizing the literature and research experience into three principles and ten lessons (Atherley et al., 2021). Decisions about smartphone use depend on how well it fits the research method, theoretical background, and purpose of data collection. In ethnographic research, photos, field notes, informal interviews, and spatial diagrams can be collected together. In narrative inquiry, daily experiences can be recorded through text and audio diaries. Recording immediately after an experience can reduce recall bias and capture experiences that would be omitted from diaries written later. Real-time digitization, backups, remote checks, and automatic notifications can reduce the burden of data collection and management. Depleted batteries or research notifications buried among other alerts can affect when recordings are made. The authors describe the low dropout rate in a longitudinal audio diary study following the transition from trainees to attending physicians. They recommend examining the conditions of participation rather than assuming that familiarity with a device will sustain participation.

Researchers and participants establish a secure communication channel and negotiate recording arrangements, data transfer methods, and the extent to which data can be withdrawn. Deleting raw data and removing content already incorporated into an integrated analysis require different procedures. The data management plan specifies the choice between personal and institution-provided devices. It also specifies transfer to a secure server, transcription and de-identification, and storage and deletion. An example plan for a hypothetical participant, Sanne, includes audio diaries from September 2018 to March 2019 and two participatory interviews. The interviews incorporate concentric network maps and workspace photos. Downloaded data are assigned the code 001 and sent to a transcription service under a confidentiality agreement. The workflow includes de-identifying the transcript and deleting the identifiable version. Audio and video recordings are deleted after all data have been analyzed.

Interpretation of data collected via smartphones considers the time available to record, device status, and responses to notifications alongside the experiences captured. In clinical settings, recordings may include the faces and voices of non-consenting patients or bystanders. Identities may be inferred from combinations of dates and location data, even if faces are obscured. Notifications and recording can distract from patient care and learning. The authors propose addressing these issues in the research plan and considering an iterative, progressive ethical approval model that adjusts the scope of access as needed. The ten lessons cover alignment with research principles, data diversity and efficiency, data quantity and quality, privacy, distraction, communication, IRB, data management plans, and reflection and feedback. The sociomateriality perspective proposed as a future research direction explores how smartphone functions interact with clinical activities to affect recording and meaning-making.

Tracing changes in experience and reflection through audio diaries

AMEE Guide No. 144 draws on prior literature to propose procedures for using audio diaries in research and education (Verma, 2021). Audio diaries are a longitudinal qualitative research method in which participants record their experiences, thoughts, and emotions in specific contexts. Participants choose when and where to record and interpret their experiences in natural speech. Researchers analyze their manner of expression and vocal cues alongside the spoken content. The theoretical foundations include a relativist ontology, in which individuals perceive and construct multiple realities, and social constructivism, in which the meaning of experience is constructed through interaction. The method can be used in narrative inquiry and ethnography. In education, it is presented as a way to support reflection-in-action, reflection-on-action, and the integration of knowledge gained from clinical experience. The researcher supports reflection within the scope of the research while respecting participants’ choices about recording.

The procedure consists of an initial interview, recording and submission using guiding questions, transcript verification, completion of a minimum number of recordings, and an exit interview. In the initial interview or focus group, the scope and questions of the research are shared, and the recording device, submission method, and schedule are negotiated. Portable voice recorders can be used in rural or remote areas with limited digital access. Semi-structured guiding questions help participants develop their stories freely. The author recommends collecting a minimum of 8–12 audio diaries per participant and recording once a week for at least 3 months, with these recommendations adjusted to suit the research purpose. Exam stress and remote clerkships are considered when setting the recording schedule. Transcripts are returned to participants to verify their accuracy. The exit interview clarifies unclear expressions and provides an opportunity to reflect on the recording experience. In educational applications, learning objectives are established in the initial interview. The exit interview includes debriefing and reviewing progress toward those goals.

Analysis begins by organizing individual recordings to understand the course of each participant’s experience. Initial interviews, diaries, and exit interviews are then analyzed cross-sectionally to identify common themes before examining how those themes develop across individual timelines. Options include framework analysis to track changes in themes and environments, longitudinal case studies, interpretative phenomenological analysis, and sociolinguistic analysis. Laughter, pauses, coughing, and silence also provide data for interpreting expressions of identity and language use. Other data can supplement the recordings when questions require attention to facial expressions and gestures. Sustaining long-term participation involves monitoring changes in living conditions and motivation, capacity to transfer files, and submission delays. Ethical review addresses the recording of bystanders’ voices, distress participants experience during recording, and the effects of recording activities on behavior.

Connecting research execution and researcher growth in doctoral supervision

AMEE Guide No. 104 presents a framework for doctoral supervision in health professions education, drawing on postgraduate supervision experience across four continents and related literature (van Schalkwyk et al., 2016). The authors view doctoral supervision as an educational activity shared by the student and supervisor, using transformative learning to explain changes in thinking, behavior, and identity. The framework consists of three phases: pre-doctoral, execution, and completion. Supervision involves revisiting various activities throughout the process. Initially, the student’s motivation for enrollment, clinical workload, research experience, and differences in language, culture, and time zones are explored. Meeting frequency, feedback methods, and roles are then negotiated. Short research proposals are reviewed alongside previous writing to gauge readiness.

Supervision models include an apprenticeship model in which one or two supervisors work with one student, group supervision, and team supervision, which typically involves 3–6 people contributing according to their expertise. Five approaches—functional, enculturation, critical thinking, emancipation, and relationship development—categorize supervision activities ranging from project management to participation in the academic community and personal growth. The study underlying this classification covered nine academic disciplines and did not include health science supervisors. Applying it to HPE involves adapting the approaches to the demands of balancing clinical work and study. According to the authors, proposal development typically takes 6–12 months, and a PhD by publication generally consists of 4–5 published papers linked by a common theme, together with an introduction and conclusion. In the proposal phase, the research question, conceptual framework, methodology, degree format, ethical approval, and administrative procedures are refined. Six threshold concepts—argumentation, theorizing, framework, knowledge creation, analysis, and paradigm—are used to explain changes in how the student understands research.

In the execution phase, students learn academic practices and develop an identity as a researcher through engagement, imagination, and alignment within a community of practice. Feedback addresses reasoning, conceptualization, and interpretation alongside sentence editing. It includes conversations in which students explain their next steps in their own words. In the completion phase, the endpoint is determined by reviewing alignment between the research question and design, analysis and interpretation, and argumentation and theory. When progress is slow, scheduling, conceptual development, living conditions, and data and methods are examined. Details of meetings and support are documented. The relationship continues after degree completion through co-publication, conference participation, research collaboration, and co-supervision.

Reading notes

When researching learning transfer in HRD and workplace learning in medical education, data collection can distinguish between confidence in using educational content, reported experiences of actual use, and the meaning constructed while narrating those experiences. Guiding questions and transcript verification prompt participants to explain their experiences again. Analysis of changes in their accounts also examines the intervening interactions. In research supervision, explaining the reasons for revisions and subsequent actions gives students practice in reviewing and adjusting their research judgments.

References

• Artino, A. R., La Rochelle, J. S., Dezee, K. J., & Gehlbach, H. (2014). Developing questionnaires for educational research: AMEE Guide No. 87. Medical Teacher, 36(6), 463–474. https://doi.org/10.3109/0142159X.2014.889814
• van Schalkwyk, S. C., Murdoch-Eaton, D., Tekian, A., van der Vleuten, C., & Cilliers, F. (2016). The supervisor’s toolkit: A framework for doctoral supervision in health professions education: AMEE Guide No. 104. Medical Teacher, 38(5), 429–442. https://doi.org/10.3109/0142159X.2016.1142517
• Atherley, A., Hu, W., Teunissen, P. W., Hegazi, I., & Dolmans, D. (2021). Appraising the use of smartphones and apps when conducting qualitative medical education research: AMEE Guide No. 130. Medical Teacher, 43(1), 68–74. https://doi.org/10.1080/0142159X.2020.1838461
• Verma, A. (2021). Using audio-diaries for research and education: AMEE Guide No. 144. Medical Teacher, 43(12), 1346–1352. https://doi.org/10.1080/0142159X.2021.1972954