The context for this review
This formative review was of an introductory postgraduate Science Education Research Methods subject that had been redesigned to better cater to external online students. Students in this subject are mostly part-time and can enrol at any time. Feedback had shown that students found the reading and assignments challenging so the teacher had reviewed the online subject design to increase the ‘electronic scaffolding’ for students. The overall focus of the review was on how the online site & materials support student learning.
Review participants
The teacher was a senior lecturer who had won a national citation for her development of “innovative and engaging learning experiences”.
The reviewer was an experienced colleague from a different discipline in the same faculty who had a teaching and learning background. The reviewee described this long-term colleague as a ‘friend’ whose skills in online learning she had great respect for.
Reason for the review
This teacher wanted a formative review to provide her with critical feedback for further development of this “relatively new” subject that she had redesigned and run for the two previous years. Although she had made changes based on feedback for each iteration she was concerned whether the restructure of the subject to provide the same experience for both internal and external students has been effective.
What happened in this review
Aspect of the subject focused on for this review was the structure and design of this online subject to “stand on its own” and to support both internal and external students’ learning.
Forms of evidence reviewed were the subject outline and all the materials and resources for students in the online site (learning management system).
Review Process
Pre-review briefing was done using email as this pair were long-term colleagues who felt comfortable and had mutual respect for each other. The process of responding to the prompting questions was challenging and required a lot of thought by the teacher, “It is quite challenging because I actually found …writing that up was actually quite challenging because it forces you to stop and think what is it that I want to be evaluated.”
The Review was done completely online and comments by reviewer can be seen on the review criteria document.
The debrief meeting was done by sending the review document (using the framework) by email and then having a quick ‘catch up’. Because everything made “perfect sense” there was little clarification required. The teacher also knew that the reviewer would be available at any time if she needed to discuss things any further.
Time taken to prepare for this review was hard for the reviewee to estimate but was approximately three hours. Initial preparation time was 1 hour (filling out the ‘easy’ bits of the briefing questions) but to complete the briefing took more time than anticipated. A week was needed to think over what, exactly, she wanted to get out of the review and then 30 minutes to document her decisions. After the review the teacher spent 1 hour reading and thinking about the feedback - even though it “made almost perfect sense”. A final hour was spent on communication after the review including: emails, a quick meeting; and further reflection. Considering the technical details of implementing the suggestions is an ongoing process. For this experienced reviewer to complete all parts of this review was 2.5 hours (30 minutes briefing; 90 minutes for three separate ‘visits’ to the site; 30 minutes for reporting back).
Outcomes of the review
- This review was highly valued by the teacher using the template and communication process outlined in the project providing practical feedback that helped the teacher resolve issues she had struggled to identify and articulate in the review briefing. Really useful ideas were identified by the reviewer, that the teacher ‘recognised instantly’ would make the subject more interactive.
“It is a different perspective. Providing a different perspective and, yes, as I say, the feedback that has given has been excellent. It's been great constructive criticism and I know now what to do and I have this plan to act upon it...“the feedback I got on this was very constructive though it just struck a cord with me immediately and it was just like yes, yes, yes, yes! Now I’ve got to do something. So it was just like reading through and it was really a case of it just needed someone to say ‘Crystal clear, this is what you need to do to make it better’. I could see straight away, yeah, yeah, yeah, and sort of almost didn’t have to read it again. It was just so clear to me that’s what I’m supposed to do.” - This reviewee emphasised the need to have time after a review to implement the valuable feedback given. For example, some of the things she wanted to implement she felt would require developing some additional technical expertise, and this may have seemed more daunting and time consuming than it needed to. This is where ongoing collegial discussions in review pairs would be very helpful.
Points to note from this review
In this instance an earlier experience of peer review with a reviewer who had specific content expertise had proved unsuccessful, due to time constraints and some miscommunication via email about what type of feedback was being sought. The success of this second review underlines the great importance of face-to-face communication, especially for reviewing pairs that have not worked closely together before. This teacher emphasised many times the importance of the relationship between reviewers and the need for trust and “commitment to a sustained relationship, familiarity with partner’s work, ongoing contact”. She also noted the importance finding the time to enable this to develop, if it does not already exist. She recognised that a good communicative relationship was key to successful reviews and was more important than being from a similar disciplines.
More from the Reviewer
“I quite actually enjoyed the process. I didn’t find it uncomfortable; I didn’t feel as though I wouldn’t say anything because [Teacher] trusted me. But I was also considered in the way I critiqued things. If I said something that I didn’t think was working well, I felt as though I needed to give some suggestions, so not just be the critical friend…because I’m familiar with the field. Or point then in another direction.
“People want to be appreciated and want to be valued and I think this is a good way to do that and they also do want some feedback on their teaching. And they often don’t get that…that’s the main reason that they were doing it, more of a diagnostic, you know, they actually wanted some real feedback. “
“You know apart from building the relationship, I also look at how other people are teaching, other strategies and think oh gee, I wonder how that would work in [Subject], or how that will work with my students and, yeah, there’s that, there’s always that two way because it causes you to reflect on your own teaching and I think that’s probably one of the most powerful things of peer review for the individual doing the reviewing…It forces you to think about your own teaching.
The time factor. To actually do a really good job, so I think that’s the biggest issue. So you really need to set aside some time, some real quality time to have those discussions beforehand and after…I think you actually need a few different goes [to do the review], you can’t just go in and you know, you can’t, you need you know a decent amount of time for the initial conversation and you need a decent amount of time to actually go in and review what they’re asking you to look at, even if it’s really focused. But you need to go in more than once. Like if I were observing someone, I’d actually want to do it at least three times.