Brief Communication: A Re-Examination of the Eye Movement Data used by Hooge et al (2018) ["Is human classification by experienced untrained observers a gold standard in fixation detection?"]

Abstract

Hooge et al. (2018) asked the question: ``Is human classification by experienced untrained observers a gold standard in fixation detection?'' They conclude the answer is no. If they had entitled their paper: ``Is human classification by experienced untrained observers a gold standard in fixation detection when data quality is very poor, data are error-filled, data presentation was not optimal, and the analysis was seriously flawed?'', I would have no case to make. In the present report, I will present evidence to support my view that this latter title is justified. The low-quality data assessment is based on using a relatively imprecise eye-tracker, the absence of head restraint for any subjects, and the use of infants as the majority of subjects (60 of 70 subjects). Allowing subjects with more than 50% missing data (as much as 95%) is also evidence of low-quality data. The error-filled assessment is based on evidence that a number of the ``fixations'' classified by ``experts'' have obvious saccades within them, and that, apparently, a number of fixations were classified on the basis of no signal at all. The evidence for non-optimal data presentation stems from the fact that, in a number of cases, perfectly good data was not presented to the coders. The flaws in the analysis are evidenced by the fact that entire stretches of missing data were considered classified, and that the measurement of saccade amplitude was based on many cases in which there was no saccade at all. Without general evidence to the contrary, it is correct to assume that some human classifiers under some conditions may meet the criteria for a gold standard, and classifiers under other conditions may not. This conditionality is not recognized by Hooge et al. (2018). A fair assessment would conclude that whether or not humans can be considered a gold standard is still very much an open question.