Pink

Research replication: Improving research methods and critical analysis in student learning


Dr Michelle Bentley, Politics and International Relations

College Excellence Teaching Prize 2017

Research replication asks students to reproduce an existing piece of published research, e.g. journal article. Students effectively deconstruct then reconstruct the research in question: they locate and examine the original data/sources (re-creating these if necessary), apply this to replicate the research findings, and critically assess the strength and validity of the author(s)’s argument. This innovative form of learning: teaches/improves student understanding of research methods by carrying out research for themselves within a guided context; enriches student awareness as to what research involves; and significantly develops capacity for critical thinking. Research replication is appropriate for formative and summative assessment. Furthermore, as this study will demonstrate, it can be successfully employed in both qualitative and quantitative methods teaching across the disciplines. The low-cost, low-resource, nature of research replication means it is highly sustainable as a form of learning.

 

Overview:

This analysis is based on a case study involving a PGT module in the Department of Politics and IR: PR5924 Transnational Security Studies. This is a core module for the MSc International Security programme, and is designed to introduce students to the field of security studies and the research methods/theories involved. The module was assessed in three ways: essay (30%), individual presentation (10%), and a 3,000-word replication study (60%).

 

Implementation:

Students were asked to select a short article (approximately 3,000 words) from a list of 10, taken from the journal Peace Review. All the articles were directly relevant to the module content. The convenor choose the articles as particularly good candidates for replication, although students were given an option to select an alternative article from the same journal should they wish, subject to approval by the convenor. Students were provided with extensive information on what research replication is/involves. Furthermore, a special in-class session was held on the assignment to ensure students knew what was expected of them, as well as time in the convenor’s office hours devoted to this. All students reported that they understood the assignment, and this was evident in the submitted work.

 

Findings:

The case study reveals four core and highly positive outcomes from using research replication as assessment. This analysis is based on observational data and focus group feedback from the students involved. In this feedback, 100% of students stated they found the assignment ‘extremely beneficial’ or ‘very beneficial’ in terms of their learning, specifically as compared to more traditional modes of assessment such as essays.

(1) Ability to carry out research

Students reported a substantially improved understanding of what research ‘is’ and how it should be carried out. Many students commented in the focus group that they did not really comprehend what research was before, but that the exercise helped them realise what their assignments (not just this one, but all assessments on the programme) demanded of them. This understanding meant they could carry out much more advanced research.

In particular, the assignment was extremely effective in developing students’ library skills, as well as other methods of locating sources. Whereas students said they had tended to rely on ‘easy to find’ sources or module reading lists in the past, this assessment forced them to track down a range of sources – including resources they had relatively little experience of, such as policy documents. This ‘research by doing’ required them to engage and experiment with new methods of research, which significantly broadened their skills in this key area. Students also reported using library services they had not previously been aware of, or were uncomfortable using in the past, e.g. inter-library loans or requesting new acquisitions. Prior to this exercise, they did not feel they could utilise these services; that these were for ‘proper academics.’

(2) Research accuracy

Building on the above point, students also reported a better understanding of the accuracy of research. This included: improved referencing skills; being able to use sources and other material correctly; and ensuring that citations were a valid representation of the initial source. By going back to the original sources, students could see how mistakes were made in analytic assessment and referencing, and how this influenced the overall success of an argument. Indeed, the level of referencing on the submitted work was significantly better than usual across the class. Furthermore, several students commented they were now significantly more aware of what plagiarism comprises as a consequence of completing the assignment.

(3) Critical thinking skills

Critical analysis is often difficult to articulate/teach and students frequently report that they do not have a proper understanding of what this entails (Tapper 2007). Research replication modes of assessment directly deal with this issue by requiring students to re-construct the critical thinking of an academic, specifically within the context of producing research. This a significant more direct means of getting students to engage in critical analysis, which then leads to greater understanding of what critical thinking means and requires.

The success of this strategy was clearly evident in the quality of the submitted work. The standard of critical analysis was outstanding compared to the usual level displayed in students’ work. Many students demonstrated extremely high levels of critical skill, being able to (amongst others): identity mistakes in the authors’ use of original sources; challenge the validity and logic of an academic argument on this basis; and suggest and support alternative explanations and approaches to the original data.

(4) High levels of student engagement and inclusion

Students were highly engaged in the assignment. This was evident in: the focus group; the students’ continued and enthusiastic commitment to the project over a number of weeks; and the quality of the students’ work (which was extremely high). Students said they found the assignment exciting and intriguing, which increased the time and effort they were prepared to invest in the project and, therefore, the learning experience it represented.

Students also reported high levels of peer engagement, in terms of sharing research skills and helping each other locate original sources. As such, this promotes inclusion. It was also especially useful for a non-standard entry student in the class who raised their mark from 58 on the essay (submitted in week 6) to 72 on the research replication paper (submitted in week 10). While this finding must take into account the student’s academic development more generally, the massive improvement in their research ability within such a short space of time means this can, at least in part, be attributed to the replication assignment.

 

Future plans:

In terms of project development, I would like to re-run this case study with final year undergraduate students. Research replication has typically been associated with postgraduate levels of education, because of the complexity of the research involved (Janz 2016). I hypothesize, however, that this can be successfully adapted and applied to undergraduates – especially in terms of preparation for the dissertation or other major research assignments. In terms of taking this forward, I have planned a focus group with my current 2nd year students to gauge their interest in, and preparedness for, research replication as a form of assessment. I will also use the focus group to determine how to adapt research replication to undergraduate level.

 

References:

Janz, N. (2016) ‘Bringing the Gold Standard into the Classroom: Replication in University Teaching’, International Studies Perspectives 17(4): 392-407

Tapper, J. (2007) ‘Student perceptions of how critical thinking is embedded in a degree program’, Higher Education Research & Development 23(2):  199-222

 

See other case studies on our Assessment and Feedback page