College of Science and Engineering
Permanent URI for this collectionhttps://hdl.handle.net/10877/17053
Browse
Browsing College of Science and Engineering by Department "Computer Science"
Now showing 1 - 20 of 72
Results Per Page
Sort Options
Item A Method for the Detection of Poorly-Formed or Misclassified Saccades: A case study using the GazeCom Dataset(2022-02) Friedman, Lee; Djanian, Shagen; Komogortsev, OlegThere are many automatic methods for the detection of eye movement types like fixation and saccades. Evaluating the accuracy of these methods can be a difficult and time-consuming process. We present a method to detect misclassified or poorly formed saccades\footnote{Throughout the manuscript, when we use the word ``misclassified'', we will be referring to both misclassified or poorly formed. saccades.}, regardless of how they were classified. We developed and tested our method on saccades from the very large and publicly available GazeCom dataset. We started out by creating a total of 9 metrics (velocity shape, velocity shape amplitude, position shape, position shape amplitude, flatness, entropy, kurtosis, skewness, and the Dip Test statistic of multimodality) which will be explained below. We applied these metrics to horizontal saccades of 20, 40 and 60 ms duration. For each duration, we performed a data reduction step with factor analysis to see how these 9 metrics were naturally grouped. For every duration, there were 2 factors, one which was dominated by our velocity shape metric and one which was dominated by our entropy metric. We determined that the entropy metric was the single most valuable metric for detecting misclassified saccades. We illustrate the types of saccades that our entropy metric indicates are misclassified. Link to Python Code https://github.com/sdjanian/sacanalysisItem A Novel Evaluation of Two Related, and Two Independent Algorithms for Eye Movement Classification during Reading(2018-01) Friedman, Lee; Rigas, Ioannis; Abdulin, Evgeny; Komogortsev, OlegThis repository contains classified eye-movement data from the submitted paper, "Novel Evaluation of Two Related, and Two Independent Algorithms for Eye Movement Classification during Reading" Lee Friedman, Ioannis Rigas, Evgeny Abdulin and Oleg V. Komogortsev The Department of Computer Science, Texas State University, San Marcos, Texas. As of 2/19/2018, the third revision is under review at Behavior Research Methods. There are 4 directories included, each with exactly 20 files. These are the 20 files that were evaluated with 4 scoring methods. ONH – These data were scored by the method described in [1]. MNH – These data were scored by the method presented in the manuscript. IRF – These data were scored by the method presented in [2]. EDF – These data were scored by the EyeLink Parser. File naming convention: Take, for example, this name: “S_1051_S1_TEX_Class_EyeLink.csv”. This is data from Subject number 1051, recording session 1, the TEX (poetry reading) task and it contains classification data scored by the EyeLink Parser. “S_1066_S2_TEX_Class_IRF.csv” is data from Subject number 1066, recording session 2, the TEX (poetry reading) task and it contains classification data scored by [2]. “S_1334_S2_TEX_Class_ONH.csv” is data from Subject number 1334, recording session 2, the TEX (poetry reading) task and it contains classification data scored by [1]. Files like “S_1282_S2_TEX_Class_MNH.csv” were scored by the method described in the manuscript. The first column of every dataset is a msec timestamp. Only the first 26,000 msec of each file were processed for the manuscript. The second column of every dataset is the horizontal (X) eye position signal in degrees of visual angle. In the case of the ONH and the MNH methods, these position signals were smoothed. See manuscript for details. The third column of every dataset is the vertical (Y) eye position signal in degrees of visual angle. In the case of the ONH and the MNH methods, these position signals were smoothed. See manuscript for details. The fourth column of every dataset is the radial velocity of the eye movement signals. Please see manuscript for details of this calculation for every dataset. The fifth column of each dataset is a classification code, where 1 = fixation, 2 = saccade, 3 = post-saccadic oscillation, 4 = noise or artifact, and 5 is unclassified. Note that the IRF coded data did not use an “unclassified” category. References: [1] M. Nystrom and K. Holmqvist, "An adaptive algorithm for fixation, saccade, and glissade detection in eyetracking data," Behav Res Methods, vol. 42, no. 1, pp. 188-204, Feb 2010. [2] R. Zemblys, D. C. Niehorster, O. Komogortsev, and K. Holmqvist, "Using machine learning to detect events in eye-tracking data," Behav Res Methods, Feb 23 2017.Item ActionItem: Collaboration through Commitment and Social Tasking(2009-09-10) Ngu, Anne H. H.; Gu, Qijun; Peng, Wuxu; Roberts, MarkThe relentless growth of global organizations and businesses require collaboration among virtual teams that can be formed on-demand and cross institutional, geographical and cultural boundaries. In this paper, we propose ActionItem - a Web 2.0 collaboration tool that fosters cooperation by leveraging the idea of commitment, social tasking and parallel blogging. We describe the prototype implementation of ActionItem and give a quantitative and qualitative evaluation of this collaboration management tool in terms of collaboration provenance, efficiency and quality through case studies. The concept of social tasking for collaboration has been used successfully in many social networking sites. However, social networking tools do not manage the collaboration in a team and nor do they provide a collaborative model for objective measurement of how people work together. The workflow-based collaboration management tools have extensive management capability, but typically are only built for collaboration among a static group of participants with clearly designated roles within a fixed organizational structure oblivious to any form of social networks. Our results show that ActionItem is a nimble, inexpensive, and effective tool to support the collaboration required for loosely coupled virtual teams who do not share the same time and space.Item An Effort-Based Approach to Measuring Software Usability(2008-10-21) Mueller, Carl J.; Komogortsev, Oleg; Tamir, Dan; Feldman, LiamAn Objective Measure of Usability Using Effort Estimation Design and implementation of usable human computer interface (HCI) systems involves expensive, primarily cognitive based, usability testing and evaluation techniques. This complicates the development process and may cause software companies and software engineers that are more familiar with objective testing methodologies to reduce or completely avoid the usability testing stage, reverting to best practice techniques, and producing HCI systems that lack usability. This research is based on the assumption that usability of HCI systems is directly related to the amount of mental and physical effort expended by the user throughout the interaction. It explores and exploits the utility of an objective, relatively easy to measure, and engineering oriented usability metric. A mathematical model of interaction effort is formulated. The model transforms data related to primitive interaction events such as keyboard keystrokes, mouse key clicks and Mickys traversed by the mouse along with eye tracking data into an effort metric. A carefully crafted set of user interaction goals employing scenario based test design techniques is implemented. Data is collected using logging programs that record goal completion time along with keyboard, mouse, and eyes interaction events. The recorded information is reduced to a statistically meaningful data-set that is used to evaluate the validity of the research assumptions. Experimental results support the hypothesize. Furthermore, they are prompting several interesting finding that merit further research and investigation. This is the first research that carries the intuitive idea of relation between effort and usability all the way to the "field" by recording and processing effort based metrics obtained from subjects while interacting with real complex systems.Item An Effort-based Framework for Evaluating Software Usability Design(2010-03-12) Tamir, Dan; Mueller, Carl J.; Komogortsev, OlegOne of the major stakeholder complaints is the usability of software applications. Although there is a rich amount of material on good usability design and evaluation practice, software engineers may need an integrated framework facilitating effective quality assessment. A novel element of the framework, presented in this paper, is its effort-based measure of usability providing developers with an informative model to evaluate software quality, validate usability requirements, and identify missing functionality. Another innovative aspect of this framework is its focus on learning in the process of assessing usability measurements and building the evaluation process around Unified Modeling Languages Use Cases. The framework also provides for additional developer feedback through the notion of designer's and expert's effort representing effort necessary to complete a task. In this paper, we present an effort-based usability model in conjunction with a framework for designing and conducting the evaluation. Experimental results provide evidence of the frameworks utility.Item Angular Offset Distributions During Fixation are, More Often than Not, Multimodal(Bern Open Publishing, 2021-06) Friedman, Lee; Lohr, Dillon J.; Hanson, Timothy; Komogortsev, OlegTypically, the position error of an eye-tracking device is measured as the distance of the eye-position from the target position in two-dimensional space (angular offset). Accuracy is the mean angular offset. The mean is a highly interpretable measure of central tendency if the underlying error distribution is unimodal and normal. However, in the context of an underlying multimodal distribution, the mean is less interpretable. We will present evidence that the majority of such distributions are multimodal. Only 14.7% of fixation angular offset distributions were unimodal, and of these, only 11.5% were normally distributed. Of the entire dataset, 1.7% were unimodal and normal.) This multimodality is true even if there is only a single, continuous tracking fixation segment per trial. We present several approaches to measure accuracy in the face of multimodality. We also address the role of fixation drift in partially explaining multimodality.Item Applications of Bayesian Network Models in Predicting Types of Hematological Malignancies(Nature Publishing Group, 2018-05) Agrahari, Rupesh; Foroushani, Amir; Docking, T. Roderick; Chang, Linda; Duns, Gerben; Hudoba, Monika; Karsan, Aly; Zare, HabilNetwork analysis is the preferred approach for the detection of subtle but coordinated changes in expression of an interacting and related set of genes. We introduce a novel method based on the analyses of coexpression networks and Bayesian networks, and we use this new method to classify two types of hematological malignancies; namely, acute myeloid leukemia (AML) and myelodysplastic syndrome (MDS). Our classifier has an accuracy of 93%, a precision of 98%, and a recall of 90% on the training dataset (n = 366); which outperforms the results reported by other scholars on the same dataset. Although our training dataset consists of microarray data, our model has a remarkable performance on the RNA-Seq test dataset (n = 74, accuracy = 89%, precision = 88%, recall = 98%), which confirms that eigengenes are robust with respect to expression profiling technology. These signatures are useful in classification and correctly predicting the diagnosis. They might also provide valuable information about the underlying biology of diseases. Our network analysis approach is generalizable and can be useful for classifying other diseases based on gene expression profiles. Our previously published Pigengene package is publicly available through Bioconductor, which can be used to conveniently fit a Bayesian network to gene expression data.Item Assessment of Shift-Invariant CNN Gaze Mappings for PS-OG Eye Movement Sensors(IEEE Computer Society, 2019-08-27) Griffith, Henry; Katrychuk, Dmytro; Komogortsev, OlegPhotosensor oculography (PS-OG) eye movement sensors offer desirable performance characteristics for integration within wireless head mounted devices (HMDs), including low power consumption and high sampling rates.To address the known performance degradation of these sensors due to HMD shifts, various machine learning techniques have been proposed for mapping sensor outputs to gaze location. This paper advances the understanding of a recently introduced convolutional neural network designed to provide shift invariant gaze mapping within a specified range of sensor translations. Performance is assessed for shift training examples which better reflect the distribution of values that would be generated through manual repositioning of the HMD. The network is shown to exhibit com-parable accuracy for this realistic shift distribution versus a previously considered rectangular grid, thereby enhancing the feasibility of in-field initialization. In addition, this work further supports the practical viability of the proposed initialization process by demonstrating robust mapping performance versus training data scale. The ability to maintain reasonable accuracy for shifts extending beyond those introduced during training is also demonstrated.Item Automated Classification of Complex Oculomotor Behavior(2012-06-10) Komogortsev, Oleg V.; Dai, Zanxun; Gobert, Denise V.Complex oculomotor behavior in response to a simple step stimulus can include a variety of different types of saccadic patterns including combinations of normal saccades, simple/corrected/multi-corrected overshoots/undershoots, express, dynamic overshoots, and compound saccades depending on the state of the oculomotor plant and the neuronal control signal supplied by the brain. This paper presents an algorithmic framework that allows automated classification of such behavior. Automated classification results were compared to manually classified data used as a reference baseline. In addition, this work investigates the impact of various filtering methods and basic eye movement classification algorithms on the accuracy of classification of complex oculomotor behavior. The proposed framework can be used in clinical examination of normal and abnormal visual systems.Item Automated Natural Language Evaluators - (ANLE)(1993-12) Kaikhah, KhosrowBy the turn of the century, it is expected that most computer applications will include a natural language processing component. Both developers and consumers of NLP systems have expressed a genuine need for standard natural language system evaluators. Automated natural language evaluators appear to be the only logical solution lo the overwhelming number of NLP systems that have been produced, are being produced, and will be produced. The system developed here is based on the Benchmark Evaluation Tool [7] and is the first attempt to fully automate the evaluation process. This effort was accomplished in two phases. In phase one, we identified a subset of the Benchmark Evaluation Tool for each class of NLP systems. And in phase two, we designed and implemented a natural language generation system to generate non-causal semantically meaningful test sentences. The generation system can be queued for each class of NLP systems. We followed an Object-Oriented Design (OOD) strategy. In this approach all concepts, including semantic and syntactic rules, are defined as objects. Each test sentence is generated as a chain of words satisfying a number of semantic, syntactic, pragmatic, and contextual constraints. The constraints imposed on the generation process increase dynamically while the sentence is being generated. This strategy guarantees semantic cohesiveness while maintaining syntactic integrity. In this approach, syntactic and semantic knowledge were utilized concurrently in word-objects. Each word-object is an independent knowledge source with local knowledge that can decide whether it can be a part of the sentence being generated, when called upon by the sentence-generator to join the chain.Item Automatic Test Case Generation for Web Service Processes Using a SAT Solver(2009-02-16) Radhakrishnan, Karthikeyann; Podorozkny, RodionSuch useful properties of web services as access from any platform, great interoperability with other web services, ability to combine several web services into a larger application relatively quickly have made them an important category of software systems. One of the techniques used to increase the quality of software is testing. The adequacy of test cases and possible automation of the testing process greatly influence the quality of the produced software and timeliness of the software development process. Even though a great deal of work has been done in adapting test case generation techniques to the peculiarities of web services (e.g. [11][12][13]) we believe our work makes a useful contribution in this area. This paper proposes a novel approach to generate test cases based on the process definition model of a web service. A process definition model defines a sequence of activities that can be performed by orchestrating the capabilities of a web service. A SAT solver (such as Alloy [10]) is used to extract the paths from the process definition model. These paths are used to generate test case specifications that will test all web service capabilities involved in a process. In our opinion the main contribution of the work is an application of a static analysis method for generation of test cases for a web service guided by a goodness metric of process coverage.Item Automatic Text Summarization with Neural Networks(2004-06-22) Kaikhah, KhosrowA novel technique for summarizing news articles using neural networks is presented. A neural network is trained to learn the relevant features cf sentences that should be included in the summary of the article. The neural network is then modified to generalize and combine the relevant fertures apparent in summary sentences. Finally, the modified neural network is used as a filter to summarize news article.Item Backpropagation: In Search of Performance Parameters(World Scientific and Engineering Academy and Society, 2004-04) Enumulapally, Anil Kumar; Bu, Lingguo; Kaikhah, KhosrowThis work is an extensive study of the backpropagation network based on a new visual tool, Equal Opportunity for Recognition (EOR) for all inputs to be recalled, which is used to evaluate the overall network performance, in particular, its generalization capabilities. The new procedure, EOR, is used as a means to assess the effect of other system parameters.Item Biometric Identification via an Oculomotor Plant Mathematical Model(2009-11-19) Komogortsev, Oleg; Jayarathna, Sampath; Aragon, Cecilia R.; Mahmoud, MechehoulThere has been increased interest in reliable, non-intrusive methods of biometric identification due to the growing emphasis on security and increasing prevalence of identity theft. This paper presents a new biometric approach that involves an estimation of the unique oculomotor plant (OP) or eye globe muscle parameters from an eye movement trace. These parameters model individual properties of the human eye, including neuronal control signal, series elasticity, length tension, force velocity, and active tension. These properties can be estimated for each extraocular muscle, and have been shown to differ between individuals. We describe the algorithms used in our approach and the results of an experiment with 41 human subjects tracking a jumping dot on a screen. Our results show improvement over existing eye movement biometric identification methods. The technique of using Oculomotor Plant Mathematical Model (OPMM) parameters to model the individual eye provides a number of advantages for biometric identification: it includes both behavioral and physiological human attributes, is difficult to counterfeit, non- intrusive, and could easily be incorporated into existing biometric systems to provide an extra layer of security.Item Biometric Performance as a Function of Gallery Size(2020-01-23) Friedman, Lee; Stern, Hal; Prokopenko, Vladyslav; Djanian, Shagen; Griffith, Henry; Komogortsev, OlegMany developers of biometric systems start with modest samples before general deployment. But they are interested in how their systems will work with much larger samples. To assist them, we evaluated the effect of gallery size on biometric performance. Identification rates describe the performance of biometric identification, whereas ROC-based measures describe the performance of biometric authentication (verification). Therefore, we examined how increases in gallery size affected identification rates (i.e., Rank-1 Identification Rate, or Rank-1 IR) and ROC-based measures such as equal error rate (EER). We studied these phenomena with synthetic data as well as real data from a face recognition study. It is well known that the Rank-1 IR declines with increasing gallery size. We have provided further insight into this decline. We have shown that this relationship is linear in log(Gallery Size). We have also shown that this decline can be counteracted with the inclusion of additional information (features) for larger gallery sizes. We have also described the curves which can be used to predict how much additional information is required to stabilize the Rank-1 IR as a function of gallery size. These equations are also linear in log(gallery size). We have also shown that the entire ROC curve is not systematically affected by gallery size, and so ROC-based scalar performance metrics such as EER are also stable across gallery size. Unsurpringingly, as additional uncorrelated features are added to the model, EER decreases. We were interested in exploring what changes in similarity score distributions might accompany these declines in EERs. For this, we evaluated the effect of number of features and gallery size on key distribution characteristics (median and IQR) of the genuine and impostor similarity score distributions. We present evidence that these decreases in EER are driven primarily by decreases in the spread of the impostor similarity score distribution.Item Biometric Performance as a Function of Gallery Size(Multidisciplinary Digital Publishing Institute, 2022-09-13) Friedman, Lee; Stern, Hal; Prokopenko, Vladyslav; Djanian, Shagen; Griffith, Henry; Komogortsev, OlegMany developers of biometric systems start with modest samples before general deployment. But they are interested in how their systems will work with much larger samples. To assist them, we evaluated the effect of gallery size on biometric performance. Identification rates describe the performance of biometric identification, whereas ROC-based measures describe the performance of biometric authentication (verification). Therefore, we examined how increases in gallery size affected identification rates (i.e., Rank-1 Identification Rate, or Rank-1 IR) and ROC-based measures such as equal error rate (EER). We studied these phenomena with synthetic data as well as real data from a face recognition study. It is well known that the Rank-1 IR declines with increasing gallery size. We have provided further insight into this decline. We have shown that this relationship is linear in log(Gallery Size). We have also shown that this decline can be counteracted with the inclusion of additional information (features) for larger gallery sizes. We have also described the curves which can be used to predict how much additional information is required to stabilize the Rank-1 IR as a function of gallery size. These equations are also linear in log(gallery size). We have also shown that the entire ROC curve is not systematically affected by gallery size, and so ROC-based scalar performance metrics such as EER are also stable across gallery size. Unsurpringingly, as additional uncorrelated features are added to the model, EER decreases. We were interested in exploring what changes in similarity score distributions might accompany these declines in EERs. For this, we evaluated the effect of number of features and gallery size on key distribution characteristics (median and IQR) of the genuine and impostor similarity score distributions. We present evidence that these decreases in EER are driven primarily by decreases in the spread of the impostor similarity score distribution.Item Brief Communication: A Re-Examination of the Eye Movement Data used by Hooge et al (2018) ["Is human classification by experienced untrained observers a gold standard in fixation detection?"](2020-01) Friedman, LeeHooge et al. (2018) asked the question: ``Is human classification by experienced untrained observers a gold standard in fixation detection?'' They conclude the answer is no. If they had entitled their paper: ``Is human classification by experienced untrained observers a gold standard in fixation detection when data quality is very poor, data are error-filled, data presentation was not optimal, and the analysis was seriously flawed?'', I would have no case to make. In the present report, I will present evidence to support my view that this latter title is justified. The low-quality data assessment is based on using a relatively imprecise eye-tracker, the absence of head restraint for any subjects, and the use of infants as the majority of subjects (60 of 70 subjects). Allowing subjects with more than 50% missing data (as much as 95%) is also evidence of low-quality data. The error-filled assessment is based on evidence that a number of the ``fixations'' classified by ``experts'' have obvious saccades within them, and that, apparently, a number of fixations were classified on the basis of no signal at all. The evidence for non-optimal data presentation stems from the fact that, in a number of cases, perfectly good data was not presented to the coders. The flaws in the analysis are evidenced by the fact that entire stretches of missing data were considered classified, and that the measurement of saccade amplitude was based on many cases in which there was no saccade at all. Without general evidence to the contrary, it is correct to assume that some human classifiers under some conditions may meet the criteria for a gold standard, and classifiers under other conditions may not. This conditionality is not recognized by Hooge et al. (2018). A fair assessment would conclude that whether or not humans can be considered a gold standard is still very much an open question.Item Brief Communication: Three Errors and Two Problems in a Recent Paper: gazeNet : End-to-end Eye-movement Event Detection with Deep Neural Networks (Zemblys, Niehorster, and Holmqvist, 2019)(2020-04) Friedman, LeeA final version of this research is now published as of April 13, 2020 in Behavior Research Methods. Access to this article is through this link: Brief communication: Three errors and two problems in a recent paper: gazeNet: End-to-end eye-movement event detection with deep neural networks (Zemblys, Niehorster, and Holmqvist, 2019) https://rdcu.be/b3z4nItem Checking the Statistical Assumptions Underlying the Application of the Standard Deviation and RMS Error to Eye-Movement Time Series: A Comparison between Human and Artificial Eyes(2022-02) Friedman, Lee; Hanson, Timothy; Stern, Hal; Aziz, Samantha D.; Komogortsev, OlegSpatial precision of eye movement is often measured using the standard deviation (SD) of the eye position signal or the root mean square (RMS) of the sample-to-sample signal differences (StoS) during fixation. Both SD and RMS of StoS signal differences are common statistical measures, but there are underlying assumptions that impact their interpretation. As a single summary measure of the variability of a distribution, the SD is most useful when applied to unimodal distributions. Both measures assume stationarity, which means that the statistical properties of the signals are stable over time. Both metrics assume the samples of the signals are independent. The presence of autocorrelation indicates that the samples in the time series are not independent. We tested these assumptions with multiple fixations from two studies, a publicly available dataset that included both human and artificial eyes (“HA Dataset”, N=336 fixations), and data from our laboratory of 14 subjects (“TXstate”, N=166 fixations). Many position signal distributions were multimodal (median for HA fixations: 40.6\%, TXstate: 84\%). No fixation position signals were stationary. All position signals were statistically significantly autocorrelated (p < 0.01). Thus, the statistical assumptions of the SD were not met for any fixation. All StoS signals were unimodal. Some StoS signals were stationary (median for HA fixations: 35.6\% , TXstate: 37.5\%). Almost all StoS signals were statistically significantly autocorrelated (p < 0.01). For TXstate, 16 of 166 (9.6\%) fixations met all assumptions. Thus, the statistical assumptions of the RMS were generally not met. The failure of these assumptions calls into question the appropriateness of the SD or the RMS-StoS as metrics of precision for eye-trackers. We compared the SD and the RMS to non-parametric versions of these measures (e.g., Mean Absolute Deviation (MAD) versus SD). The parametric and non-parametric indices were highly correlated. Nonparametric measures are more robust to deviations from normality, so in this sense, they are an improvement. However, the use of nonparametric measures does not remove the requirements for independence and stationarity.Item Classification Algorithm for Saccadic Oculomotor Behavior(2010-05-04) Komogortsev, Oleg; Gobert, Denise V.; Dai, ZanxunThis paper presents a detection algorithm that allows automatic classification of hypermetric and hypometric oculomotor plant behavior in cases when saccadic behavior of the oculomotor plant is assessed during the course of the step stimulus. Such behavior can be classified with a number of oculomotor plant metrics represented by the number of overshoots, undershoots, corrected undershoots/overshoots, multi-corrected overshoots/undershoots. The algorithm presented in this paper allows for the automated classification of nine oculomotor plant metrics including dynamic overshoots and express saccades. Data from sixty-five human subjects were used to support this experimental study. The performance of the proposed algorithm was tested and compared to manual classification methods resulting in a detection accuracy of up to 72% for several of the oculomotor plant metrics.