Examiner feedback

The latest ARC Discovery Grant for SORTI researchers: The impact of examiner feedback on doctoral learners and thesis outcomes. This is the fourth in the program of research into Doctoral Examination.

ARC Discovery Project DP 180100448

The project team

Professor Allyson Holbrook Professor Allyson Holbrook

Professor Allyson Holbrook

Professor
School of Education
Emeritus Professor Terry Lovat

Emeritus Professor Terry Lovat

Emeritus Professor
College of Human and Social Futures
Dr Janene Budd

Dr Janene Budd

Aims and significance

Feedback by examiners is enshrined in the doctoral examination process and is primarily directed to improving the thesis and subsequent publications. This is fitting given that the recommendation framework used by institutions is part grade and part gauge of what still needs to be done. However, any assessor feedback is only valuable in so far as it is actually used. Without research to establish with any certainty how decisions are determined following on from examiner reports, and in what capacity examiner feedback is accepted and used by learners, the effect on outcomes such as thesis quality, candidate development,  or candidate satisfaction cannot be ascertained. This paucity of information extends to most aspects of the final stage of examination of doctorates including how institutional protocols guide this final stage, how examiner recommendations, rankings and comment are interpreted and acted upon in decision-making and with what consistency; the nature of checks and balances over process; the supervisor role in the activities that occur and how the candidate engages with the task of responding to feedback. This project will investigate the hitherto unexplored final stage of doctoral assessment in Australian universities. It will incorporate assessment processes, practices, and specifically the engagement of students with examiner feedback. It will investigate how examiner recommendations and feedback are filtered through institutional processes and actors to influence researcher skills development and quality of thesis outcome.

Research questions

Q1. What happens between the point when examiner reports on a doctoral thesis are received and a final decision is made, what are the processes, major influences on decision-making and their effects?

Q2. How is examiner feedback used by institutions, supervisors and candidates, how is it valued and in what ways, if any, does it shape the final thesis and candidate development?

Q3. Do candidates and supervisors differ in their expectations about and satisfaction with the examination process?

Q4. How do candidate demographic characteristics, learner attributes, and expectations and experiences of the assessment process, influence student engagement with examiner feedback and thesis revisions?

Q5. How and with what understandings do students approach and undertake changes and revisions to their thesis and how do they perceive the outcomes of feedback for them as researchers and learners?

Methodology

Examiner feedback methodology

Phase 1: The protocols governing examination procedures have been sourced, as far as possible, from all Australian institution websites and entered into QSR NVivo. Available data for all 39 institutions will be collated and analysed. This analysis will inform the interviews.

Phase 2: Online surveys of graduates and supervisors primarily comprise closed questions, scored using a Likert-type scale to provide quantitative measures of satisfaction with examination processes, receptiveness to examiner feedback, and perceptions of doctoral outcomes. Phase 2 surveys are now open.

Phase 3: Interviews with Deans will seek to elucidate how decisions based on the examiner reports are made and who makes those decisions. Interviews with the Research Training Committee (RTC) members will seek to uncover the members’ perceptions of their role in the processes governing the passage of examiner reports, as well as the committee members’ involvement in determining decisions about thesis revisions and judgements about final outcomes. Interviews with Deans will commence in October 2018 and interviews with RTC members will commence in 2019.

Publications

Dally, K., Holbrook, A., Lovat, T., & Budd, J. (2018).  Invisible pedagogy? The state of research on supervisor feedback. Paper to be presented at 2108 Australian Association for Research in Education (AARE) Conference as part of a Symposium on Doctoral Education: The written feedback of supervisors and examiners, Sydney Dec 2-6, 2018