Querying Event Sequences by Exact Match or Similarity Search: Design and Empirical Evaluation.

Specifying event sequence queries is challenging even for skilled computer professionals familiar with SQL. Most graphical user interfaces for database search use an exact match approach, which is often effective, but near misses may also be of interest. We describe a new similarity search interface, in which users specify a query by simply placing events on a blank timeline and retrieve a similarity-ranked list of results. Behind this user interface is a new similarity measure for event sequences which the users can customize by four decision criteria, enabling them to adjust the impact of missing, extra, or swapped events or the impact of time shifts. We describe a use case with Electronic Health Records based on our ongoing collaboration with hospital physicians. A controlled experiment with 18 participants compared exact match and similarity search interfaces. We report on the advantages and disadvantages of each interface and suggest a hybrid interface combining the best of both.

The effectiveness of a training method using self-modeling webcam photos for reducing musculoskeletal risk among office workers using computers.

An intervention study was conducted to examine the effectiveness of an innovative self-modeling photo-training method for reducing musculoskeletal risk among office workers using computers. Sixty workers were randomly assigned to either: 1) a control group; 2) an office training group that received personal, ergonomic training and workstation adjustments or 3) a photo-training group that received both office training and an automatic frequent-feedback system that displayed on the computer screen a photo of the worker's current sitting posture together with the correct posture photo taken earlier during office training. Musculoskeletal risk was evaluated using the Rapid Upper Limb Assessment (RULA) method before, during and after the six weeks intervention. Both training methods provided effective short-term posture improvement; however, sustained improvement was only attained with the photo-training method. Both interventions had a greater effect on older workers and on workers suffering more musculoskeletal pain. The photo-training method had a greater positive effect on women than on men.

Classification of patients by severity grades during triage in the emergency department using data mining methods.

OBJECTIVE: To test the feasibility of classifying emergency department patients into severity grades using data mining methods. DESIGN: Emergency department records of 402 patients were classified into five severity grades by two expert physicians. The Naïve Bayes and C4.5 algorithms were applied to produce classifiers from patient data into severity grades. The classifiers' results over several subsets of the data were compared with the physicians' assessments, with a random classifier, and with a classifier that selects the maximal-prevalence class. MEASUREMENTS: Positive predictive value, multiple-class extensions of sensitivity and specificity combinations, and entropy change. RESULTS: The mean accuracy of the data mining classifiers was 52.94 ± 5.89%, significantly better (P < 0.05) than the mean accuracy of a random classifier (34.60 ± 2.40%). The entropy of the input data sets was reduced through classification by a mean of 10.1%. Allowing for classification deviations of one severity grade led to mean accuracy of 85.42 ± 1.42%. The classifiers' accuracy in that case was similar to the physicians' consensus rate. Learning from consensus records led to better performance. Reducing the number of severity grades improved results in certain cases. The performance of the Naïve Bayes and C4.5 algorithms was similar; in unbalanced data sets, Naïve Bayes performed better. CONCLUSIONS: It is possible to produce a computerized classification model for the severity grade of triage patients, using data mining methods. Learning from patient records regarding which there is a consensus of several physicians is preferable to learning from each physician's patients. Either Naïve Bayes or C4.5 can be used; Naïve Bayes is preferable for unbalanced data sets. An ambiguity in the intermediate severity grades seems to hamper both the physicians' agreement and the classifiers' accuracy.

Intelligent visualization and exploration of time-oriented data of multiple patients.

OBJECTIVE: Clinicians and medical researchers alike require useful, intuitive, and intelligent tools to process large amounts of time-oriented multiple-patient data from multiple sources. For analyzing the results of clinical trials or for quality assessment purposes, an aggregated view of a group of patients is often required. To meet this need, we designed and developed the VISualizatIon of Time-Oriented RecordS (VISITORS) system, which combines intelligent temporal analysis and information visualization techniques. The VISITORS system includes tools for intelligent retrieval, visualization, exploration, and analysis of raw time-oriented data and derived (abstracted) concepts for multiple patient records. To derive meaningful interpretations from raw time-oriented data (known as temporal abstractions), we used the knowledge-based temporal-abstraction method. METHODS: The main module of the VISITORS system is an interactive, ontology-based exploration module, which enables the user to visualize raw data and abstract (derived) concepts for multiple patient records, at several levels of temporal granularity; to explore these concepts; and to display associations among raw and abstract concepts. A knowledge-based delegate function is used to convert multiple data points into one delegate value representing each temporal granule. To select the population of patients to explore, the VISITORS system includes an ontology-based temporal-aggregation specification language and a graphical expression-specification module. The expressions, applied by an external temporal mediator, retrieve a list of patients, a list of relevant time intervals, and a list of time-oriented patients' data sets, by using an expressive set of time and value constraints. RESULTS: Functionality and usability evaluation of the interactive exploration module was performed on a database of more than 1000 oncology patients by a group of 10 users-five clinicians and five medical informaticians. Both types of users were able in a short time (mean of 2.5+/-0.2min per question) to answer a set of clinical questions, including questions that require the use of specialized operators for finding associations among derived temporal abstractions, with high accuracy (mean of 98.7+/-2.4 on a predefined scale from 0 to 100). There were no significant differences between the response times and between accuracy levels of the exploration of the data using different time lines, i.e., absolute (i.e., calendrical) versus relative (referring to some clinical key event). A system usability scale (SUS) questionnaire filled out by the users demonstrated the VISITORS system to be usable (mean score for the overall group: 69.3), but the clinicians' usability assessment was significantly lower than that of the medical informaticians. CONCLUSIONS: We conclude that intelligent visualization and exploration of longitudinal data of multiple patients with the VISITORS system is feasible, functional, and usable.

A scalable architecture for incremental specification and maintenance of procedural and declarative clinical decision-support knowledge.

Clinical guidelines have been shown to improve the quality of medical care and to reduce its costs. However, most guidelines exist in a free-text representation and, without automation, are not sufficiently accessible to clinicians at the point of care. A prerequisite for automated guideline application is a machine-comprehensible representation of the guidelines. In this study, we designed and implemented a scalable architecture to support medical experts and knowledge engineers in specifying and maintaining the procedural and declarative aspects of clinical guideline knowledge, resulting in a machine comprehensible representation. The new framework significantly extends our previous work on the Digital electronic Guidelines Library (DeGeL) The current study designed and implemented a graphical framework for specification of declarative and procedural clinical knowledge, Gesher. We performed three different experiments to evaluate the functionality and usability of the major aspects of the new framework: Specification of procedural clinical knowledge, specification of declarative clinical knowledge, and exploration of a given clinical guideline. The subjects included clinicians and knowledge engineers (overall, 27 participants). The evaluations indicated high levels of completeness and correctness of the guideline specification process by both the clinicians and the knowledge engineers, although the best results, in the case of declarative-knowledge specification, were achieved by teams including a clinician and a knowledge engineer. The usability scores were high as well, although the clinicians' assessment was significantly lower than the assessment of the knowledge engineers.

Intelligent interactive visual exploration of temporal associations among multiple time-oriented patient records.

OBJECTIVES: To design, implement and evaluate the functionality and usability of a methodology and a tool for interactive exploration of time and value associations among multiple-patient longitudinal data and among meaningful concepts derivable from these data. METHODS: We developed a new, user-driven, interactive knowledge-based visualization technique, called Temporal Association Charts (TACs). TACs support the investigation of temporal and statistical associations within multiple patient records among both concepts and the temporal abstractions derived from them. The TAC methodology was implemented as part of an interactive system, called VISITORS, which supports intelligent visualization and exploration of longitudinal patient data. The TAC module was evaluated for functionality and usability by a group of ten users, five clinicians and five medical informaticians. Users were asked to answer ten questions using the VISITORS system, five of which required the use of TACs. RESULTS: Both types of users were able to answer the questions in reasonably short periods of time (a mean of 2.5 +/- 0.27 minutes) and with high accuracy (95.3 +/- 4.5 on a 0-100 scale), without a significant difference between the two groups. All five questions requiring the use of TACs were answered with similar response times and accuracy levels. Similar accuracy scores were achieved for questions requiring the use of TACs and for questions requiring the use only of general exploration operators. However, response times when using TACs were slightly longer. CONCLUSIONS: TACs are functional and usable. Their use results in a uniform performance level, regardless of the type of clinical question or user group involved.

Ability of expert physicians to structure clinical guidelines: reality versus perception.

RATIONALE, AIMS AND OBJECTIVES: Structuring Textual Clinical Guidelines (GLs) into a formal representation is a necessary prerequisite for supporting their automated application. We had developed a collaborative guideline-structuring methodology that involves expert physicians, clinical editors and knowledge engineers, to produce a machine-comprehensible representation for automated support of evidence-based, guideline-based care. Our goals in the current study were: (1) to investigate the perceptions of the expert physicians and clinical editors as to the relative importance, for the structuring process, of different aspects of the methodology; (2) to assess, for the clinical editors, the inter-correlations among (i) the reported level of understanding of the guideline structuring ontology's (knowledge scheme's) features, (ii) the reported ease of structuring each feature and (iii) the actual objective quality of structuring. METHODS: A clinical consensus regarding the contents of three guidelines was prepared by an expert in the domain of each guideline. For each guideline, two clinical editors independently structured the guideline into a semi-formal representation, using the Asbru guideline ontology's features. The quality of the resulting structuring was assessed quantitatively. Each expert physician was asked which aspects were most useful for formation of the consensus. Each clinical editor filled questionnaires relating to: (1) the level of understanding of the ontology's features (before the structuring process); (2) the usefulness of various aspects in the structuring process (after the structuring process); (3) the ease of structuring each ontological feature (after the structuring process). Subjective reports were compared with objective quantitative measures of structuring correctness. RESULTS: Expert physicians considered having medical expertise and understanding the ontological features as the aspects most useful for creation of a consensus. Clinical editors considered understanding the ontological features and the use of the structuring tools as the aspects most useful for structuring guidelines. There was a positive correlation (R = 0.87, P < 0.001) between the reported ease of understanding ontological features and the reported ease of structuring those features. However, there was no significant correlation between the reported level of understanding the features - or the reported ease of structuring by using those features - and the objective quality of the structuring of these features in actual guidelines. CONCLUSIONS: Aspects considered important for formation of a clinical consensus differ from those for structuring of guidelines. Understanding the features of a structuring ontology is positively correlated with the reported ease of using these features, but neither of these subjective reports correlated with the actual objective quality of the structuring using these features.

A methodology for evaluation of a markup-based specification of clinical guidelines.

We introduce a three-phase, nine-step methodology for specification of clinical guidelines (GLs) by expert physicians, clinical editors, and knowledge engineers, and for quantitative evaluation of the specification's quality. We applied this methodology to a particular framework for incremental GL structuring (mark-up) and to GLs in three clinical domains with encouraging results.

A quality assessment tool for markup-based clinical guidelines.

We introduce a tool for quality assessment of procedural and declarative knowledge. We developed this tool for evaluating the specification of mark-up-based clinical GLs. Using this graphical tool, the expert physician and knowledge engineer collaborate to perform scoring, using pre-defined scoring scale, each of the knowledge roles of the mark-ups, comparing it to a gold standard. The tool enables scoring the mark-ups simultaneously at different sites by different users at different locations.

A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines.

We introduce a three-phase, nine-step methodology for specification of clinical guidelines (GLs) by expert physicians, clinical editors, and knowledge engineers and for quantitative evaluation of the specification's quality. We applied this methodology to a particular framework for incremental GL structuring (mark-up) and to GLs in three clinical domains. A gold-standard mark-up was created, including 196 plans and subplans, and 326 instances of ontological knowledge roles (KRs). A completeness measure of the acquired knowledge revealed that 97% of the plans and 91% of the KR instances of the GLs were recreated by the clinical editors. A correctness measure often revealed high variability within clinical editor pairs structuring each GL, but for all GLs and clinical editors the specification quality was significantly higher than random (p<0.01). Procedural KRs were more difficult to mark-up than declarative KRs. We conclude that given an ontology-specific consensus, clinical editors with mark-up training can structure GL knowledge with high completeness, whereas the main demand for correct structuring is training in the ontology's semantics.

Learning headway estimation in driving.

OBJECTIVE: The main purpose of the present study was to examine to what extent the ability to attain a required headway of 1 or 2 s can be improved through practical driving instruction under real traffic conditions and whether the learning is sustained after a period during which there has been no controlled training. BACKGROUND: The failure of drivers to estimate headways correctly has been demonstrated in previous studies. METHODS: Two methods of training were used: time based (in seconds) and distance based (in a combination of meters and car lengths). For each method, learning curves were examined for 18 participants at speeds of 50, 80, and 100 km/hr. RESULTS: The results indicated that drivers were weak in estimating headway prior to training using both methods. The learning process was rapid for both methods and similar for all speeds; thus, after one trial with feedback, there was already a significant improvement. The learning was retained over time, for at least the 1 month examined in this study. CONCLUSION: Both the time and distance training of headway improved drivers' ability to attain required headways, with the learning being maintained over a retention interval. The learning process was based on perceptual cues from the driving scene and feedback from the experimenter, regardless of the formal training method. APPLICATION: The implications of these results are that all drivers should be trained in headway estimation using an objective distance measuring device, which can be installed on driver instruction vehicles.