The influence of preprocessing on text classification using a bag-of-words representation.

Text classification (TC) is the task of automatically assigning documents to a fixed number of categories. TC is an important component in many text applications. Many of these applications perform preprocessing. There are different types of text preprocessing, e.g., conversion of uppercase letters into lowercase letters, HTML tag removal, stopword removal, punctuation mark removal, lemmatization, correction of common misspelled words, and reduction of replicated characters. We hypothesize that the application of different combinations of preprocessing methods can improve TC results. Therefore, we performed an extensive and systematic set of TC experiments (and this is our main research contribution) to explore the impact of all possible combinations of five/six basic preprocessing methods on four benchmark text corpora (and not samples of them) using three ML methods and training and test sets. The general conclusion (at least for the datasets verified) is that it is always advisable to perform an extensive and systematic variety of preprocessing methods combined with TC experiments because it contributes to improve TC accuracy. For all the tested datasets, there was always at least one combination of basic preprocessing methods that could be recommended to significantly improve the TC using a BOW representation. For three datasets, stopword removal was the only single preprocessing method that enabled a significant improvement compared to the baseline result using a bag of 1,000-word unigrams. For some of the datasets, there was minimal improvement when we removed HTML tags, performed spelling correction or removed punctuation marks, and reduced replicated characters. However, for the fourth dataset, the stopword removal was not beneficial. Instead, the conversion of uppercase letters into lowercase letters was the only single preprocessing method that demonstrated a significant improvement compared to the baseline result. The best result for this dataset was obtained when we performed spelling correction and conversion into lowercase letters. In general, for all the datasets processed, there was always at least one combination of basic preprocessing methods that could be recommended to improve the accuracy results when using a bag-of-words representation.

A Matter of Urgency: Reducing Clinical Text Message Interruptions During Educational Sessions.

BACKGROUND: Text messaging is increasingly replacing paging as a tool to reach physicians on medical wards. However, this phenomenon has resulted in high volumes of nonurgent messages that can disrupt the learning climate. OBJECTIVE: Our objective was to reduce nonurgent educational interruptions to residents on general internal medicine. DESIGN, SETTING, PATIENTS: This was a quality improvement project conducted at an academic hospital network. Measurements and interventions took place on 8 general internal medicine inpatient teaching teams. INTERVENTION: Interventions included (1) refining the clinical communication process in collaboration with nursing leadership; (2) disseminating guidelines with posters at nursing stations; (3) introducing a noninterrupting option for message senders; (4) audit and feedback of messages; (5) adding an alert for message senders advising if a message would interrupt educational sessions; and (6) training and support to nurses and residents. MEASUREMENTS: Interruptions (text messages, phone calls, emails) received by institution-supplied team smartphones were tracked during educational hours using statistical process control charts. A 1-month record of text message content was analyzed for urgency at baseline and following the interventions. RESULTS: The interruption frequency decreased from a mean of 0.92 (95% CI, 0.88 to 0.97) to 0.59 (95% CI, 0.51 to0.67) messages per team per educational hour from January 2014 to December 2016. The proportion of nonurgent educational interruptions decreased from 223/273 (82%) messages over one month to 123/182 (68%; P < .01). CONCLUSIONS: Creation of communication guidelines and modification of text message interface with feedback from end-users were associated with a reduction in nonurgent educational interruptions. Continuous audit and feedback may be necessary to minimize nonurgent messages that disrupt educational sessions.