2. Residents As Teacher Performance Improvement (Rat Pi) Project

---

In an effort to improve medical students’ psychiatric education, we began a Residents As Teacher (RAT) program in June 2011. In preparation for this RAT PI project, we collected baseline data evaluating residents’ teaching between March and June 2011. In preparation for the 2011-2012 Academic Year (AY), we conducted an introduction to RAT workshop for all incoming residents and all current residents who were able to attend. This introduction to RAT workshop occurred in June (AY 2012, 2013). Each AY, the RAT PI program also included a six-to-eight hour seminar addressing some aspect of education and training important to psychiatrists. Altogether, a typical intern entering the resident program in AY 2011 would attend the introduction to RAT workshop and three educational seminars before graduating. To measure the RAT PI project’s outcomes, a second wave of evaluation data was collected from third-year medical students in the psychiatry clerkship, between February and June 2013.The introduction to RAT workshop consisted of four hours dedicated to providing an overview of medical student education at the institution, a review of the psychiatric clerkship syllabus and course objectives, medical student evaluations (content areas, methods, and approaches), the One Minute Preceptor model, and principles and methods of teaching in small group settings. The psychiatry clerkship director, Assistant Dean for Medical Education, and psychiatry faculty, facilitated the workshop.
The three six-hour RAT seminars focused on a specific area of education important to psychiatric practice. We chose community education, principles of instructional design and curriculum development, and patient and family education as the content areas. While not specifically targeting medical students, the seminar content was applicable and transferable to teaching medical students. The goal of the community education seminar is to prepare psychiatric residents to function as community educators. The seminar introduces residents to basic educational and health promotion concepts and the role of psychiatrists as community educators, and provides residents with the opportunity to develop community educational presentation skills and to gain feedback from peers and faculty. The instructional design and curriculum deve lopment course taught fundamental concepts and required residents to develop a one-hour instructional module on some area of psychiatry that is appropriate for third-year medical students. This seminar included assigned reading, presentations, and peer feedback. The patient and family education seminar provided information and readings on health promotions, mental health literacy, and making referrals. This seminar included role-playing, problem-based learning, the creation of a 20-minute patient education module on an assigned clinical topic, and peer feedback. In total, the RAT PI program includes a four-hour introduction to RAT workshop that is primarily focused on RAT in the clerkship and three educational seminars (one each year) on aspects of education important to psychiatry. The hope is that this approach will improve medical student education and provide psychiatrists with the training and experience to be effective educators in a variety of settings.
As an initial step in supporting RAT PI, we conducted a baseline self-assessment evaluation of psychiatry and family medicine residents. We reported that psychiatry residents in our study did not clearly understand their role as educators of both patients and medical students, compared with family medicine residents. We found that residents’ attitudes varied across the institution, and suggested that each institution and training.

3. Rat Pi Scale Development

---

Following a review of the psychiatric literature on the evaluation of residents as teachers and the general literature on teaching effectiveness, we adapted 15 items from the San Francisco State University (SFSU) ongoing study and assessment of teaching effectiveness for use in this study [16]. SFSU has been studying the measurement of teaching effectiveness since 1996. The University has conducted a number of ongoing qualitative and quantitative measurement and item analysis studies on teaching effectiveness. The SFSU study based their items on a thorough review of the teaching effectiveness literature, tapping the following domains: organization of course knowledge and content, clear communication with students, respectful, fair, content driven interactions with students, concern for student learning, creating a learning environment for students, timely feedback, course difficulty and workload, and overall teaching effectiveness[16]. These domains correspond closely to the University of Michigan Global Rating Scale (GRS) and the Stanford Faculty Development questionnaire (SFDP26). These robust items measure general teaching effectiveness across academic settings. In the present study, all 15 of the original SFSU items were revised to reflect resident teaching (i.e., “Residents are enthusiastic about teaching students, residents encourage student participation, and residents provided timely feedback on my performance and progress throughout the clerkship”). To our knowledge, this is the first time that medical school teaching effectiveness has been assessed using these items. We assessed the effectiveness of the RAT PI program using third-year medical students’ evaluations of knowledge, skills, and values/attitudes of residents’ teaching via responses to the SFSU Likert-type instrument, before and at the mid-point of a RAT PI education program.

4. Methods

---

We conducted a baseline student evaluation of residents’ teaching knowledge, skills and values/attitudes between March and June in 2011 among third-year medical students (n=53). The medical students were between the ages of 24 and 30. The majority were males. At the time of the assessment, the students were rotating through a six-week psychiatric clerkship. The medical students received both didactic and bedside instruction from the psychiatric residents. After the initial assessment, residents participated in an initial four-hour RAT workshop (2010 and 2011). Approximately 12 first, second and third year residents participated in the workshops and provided some form of instruction to the medical students. Half of the residents were females. The workshop was followed by a six-session educational seminar on community education (2011) and a six-session seminar on instructional methods (2012). To determine the effectiveness of the RAT PI program and to assess the residents' teaching, we conducted a follow up evaluation between February and June in 2013 among third-year medical students (n=60). The study was approved by the University of Oklahoma Institutional Review Board (IRB). Analysis was conducted with SAS 9.3 software. The 15-item instrument covered the three main areas of resident teaching, including knowledge, skills, and values/attitude. Responses were evaluated using a 5-point Likert scale: 1-strongly disagree, 2-disagree, 3-neutral, 4-agree, and 5-strongly agree. Descriptive statistics were determined for the three main areas. T-tests were used to determine if there were any significant differences in resident teaching performances before and after the RAT PI program. Chi-square analysis was performed on dichotomized data; responses of “Strongly Agree” and “Agree” were collapsed, and “Neutral,” “Disagree,” and “Strongly Disagree” were collapsed, creating a Less-Than-Favorable response category. The alpha for the study was set at 0.05.

5. Results

---

The medical students’ evaluations of resident teaching revealed remarkable findings at the mid-point of the RAT curriculum. Table 1 provides the statistical analysis of medical students’ responses. As previously stated, responses were categorized under the three main areas of knowledge, skill, and values/attitude of residents’ teaching, before and after RAT PI program.Table1 Figure 1 shows the medical students’ evaluations of the knowledge (82%), skills (85%), and values/attitudes (88%) of residents’ teaching after implementation of the RAT PI program. There were no overall statistical differences noted in the 3 main areas. However, a significant improvement was identified within the items of knowledge and value/attitudes of residents’ teaching. Students’ ratings of residents’ organization of didactics and lectures, and their facilitation of student learning, were higher after implementation of RAT than before implementation (p< 0.005). Residents’ encouragement for student participation (p< 0.005) was also rated higher after the program was implemented.

6. Discussion

---

Medical students' evaluations support the importance of RAT PI programs. While teaching residents the fine points of educational design and instructional methods may require more focused education and ongoing supervision, the study’s results indicate that residents’ teaching can improve through relatively short, focused training. Specifically, helping residents learn to organize their didactic presentations improves medical students’ learning. Additionally, providing residents with models of bedside instruction like the One Minute Preceptor model positively influences medical students’ clinical knowledge and skills. The increased emphasis placed on medical student instruction seems to have increased residents’ enthusiasm and a positive attitude toward medical student education, which resulted in greater participation by medical students. With few exceptions, all items assessing residents’ teaching effectiveness improved after the training workshops.
At our institution, residents provide most of the didactic lectures and clinical ward teaching during the six-week medical student clerkship rotation. As part of the initiative to improve residents’ teaching, we modified the didactic curriculum during the study period from lecture-based to role-play, and included group discussion and clinical vignettes during lessons to motivate student participation. The RAT training workshops equipped residents with the knowledge, skills and attitudes necessary to maximize the potential benefits of these new instructional methods, before these changes were implemented. Over the three-year period, we found significant improvements in students’ evaluations of these items within the knowledge, skills and values/attitudes areas of resident teaching at the mid-point student evaluation, after implementation of the RAT PI program. In aggregate, these results emphasize the important impact a coherent, organized psychiatric clerkship curriculum that is supported by resident educational training can have on medical students’ learning.
Although there were insignificant numerical differences in many items, we identified a statistically significant improvement in only two items. However, this result may be due to some items that were rated highly during the initial evaluation (i.e., “Residents were respectful and fair, residents demonstrated respect for all students, residents were sensitive and responsive to students as individuals”), leaving little room for improvement due to ceiling effects. The outcomes of this early stage of the RAT PI program emphasize the value of continuing the RAT PI program at our institution.
One limitation of this study is that, at this time, we lack long-term qualitative data to determine the effectiveness of our RAT PI program. In addition, because this study was conducted at a single institution without control groups, the findings may not be generalizable to other institutions and programs. We conducted the study over a three-year period. The changes in the pools of residents being assessed and of medical students providing assessments were another limitation of the study. Medical students from different classes could vary in their optimism and assessments of psychiatry. However, there were no marked changes in medical students’ rotation schedules or in their curriculum during the study period. Finally, medical students’ perceptions of residents’ teaching might not be completely accurate, and may possibly be affected by other factors, such as residents’ personality styles.
A previous study showed that student perceptions of the overall quality of surgery residents’ teaching instruction had a positive impact on their surgery shelf examination scores [15]. In the future, it would be interesting to assess medical students’ psychiatry shelf examination scores in relation to implementation of the RAT program. Our future goal is to conduct a resident self-assessment after full RAT implementation, and to correlate the findings of their self-assessments with medical students’ evaluations and assessments.

Acknowledgements

---

The Authors gratefully acknowledge Ms. Kathy J. Kyler, M.S., M.A., Staff Editor at the University of Oklahoma Health Sciences Center for her editorial input.