What have we learned about constructed response short-answer questions from students and faculty? A multi-institutional study.

PURPOSE: The purpose of this study was to enrich understanding about the perceived benefits and drawbacks of constructed response short-answer questions (CR-SAQs) in preclerkship assessment using Norcini's criteria for good assessment as a framework. METHODS: This multi-institutional study surveyed students and faculty at three institutions. A survey using Likert scale and open-ended questions was developed to evaluate faculty and student perceptions of CR-SAQs using the criteria of good assessment to determine the benefits and drawbacks. Descriptive statistics and Chi-square analyses are presented, and open responses were analyzed using directed content analysis to describe benefits and drawbacks of CR-SAQs. RESULTS: A total of 260 students (19%) and 57 faculty (48%) completed the survey. Students and faculty report that the benefits of CR-SAQs are authenticity, deeper learning (educational effect), and receiving feedback (catalytic effect). Drawbacks included feasibility, construct validity, and scoring reproducibility. Students and faculty found CR-SAQs to be both acceptable (can show your reasoning, partial credit) and unacceptable (stressful, not USMLE format). CONCLUSIONS: CR-SAQs are a method of aligning innovative curricula with assessment and could enrich the assessment toolkit for medical educators.

Asking parents about babbling at 10 months produced valid answers but did not predict language screening result 2 years later.

AIM: We evaluated the concurrent and predictive validity of questions to parents of 10-month-old children about babbling. METHODS: Children with at least one native Swedish-speaking parent were eligible for inclusion in this prospective longitudinal study. The parents were asked three questions about babbling at a routine healthcare visit. If parents reported a lack of canonical babbling (CB), children were assessed by a speech and language pathologist to evaluate the questions' concurrent validity. We then examined whether the babbling questions predicted which children would fail the routine language screening at 2.5/3 years. RESULTS: Fifteen of the 1126 children lacked CB according to the parent responses and the expert assessment confirmed 12 of these cases, providing a concurrent validity of 80%. The sensitivity to predict routine language screening was 8% (95% confidence interval 3-17), and the positive predictive value was 40% (95% confidence interval 20%-65%). However, only six of the children lacking CB at 10 months were among the 71 children who failed later language screening. CONCLUSION: This study suggests that the babbling questions could be included in the 10-month surveillance at the child health services as valid measures of babbling development, but they cannot predict language screening result at 2.5/3 years.

Use of open-ended questionnaires to examine the effects of tinnitus and its relation to patient-reported outcome measures.

OBJECTIVE: The primary aim of the study was to examine the automated linguistic analysis of the open-ended problem (PQ) and life-effects (LEQ) questionnaires to understand the psychological effects of tinnitus. DESIGN: The study used a cross-sectional design. Participants completed online questionnaires which included demographic questions, several standardised patient-reported outcome measures (PROMs), and two open-ended questions focussing on PQ and LEQ related to tinnitus. The response to open-ended questions was analysed using the Linguistic Inquiry Word Count (LIWC) software to identify the frequency of text on various linguistic dimensions relevant to tinnitus. STUDY SAMPLE: 336 individuals with tinnitus. RESULTS: The study results point to two broad findings. First, although PQ and LEQ have some similarities with PROMs (e.g. the linguistic dimension negative emotions having a weak positive correlation with anxiety and depression), no correlation with the number of dimensions suggests that the open-ended questions identify additional elements that are not captured in PROMs. Second, more linguistic dimensions from the PQ correlate with PROMs compared to LEQ suggesting that the current PROMs are problem-oriented. CONCLUSIONS: The study results support the idea that the use of open-ended questions in addition to PROMs may help optimise the efforts in examining the effects of chronic conditions such as tinnitus.

Frame of reference training for content analysis with structured teams (FORT-CAST): A framework for content analysis of open-ended survey questions using multidisciplinary coders.

In the context of a global pandemic, the need for reliable analysis of qualitative data in healthcare has never been more pressing. Open-ended questions are a feasible way for both researchers and organizational stakeholders to gain deeper insight into complex situations when timely research is needed. However, the interpretation of brief, textual responses can prove problematic. Both manual and automated/semiautomated methods of coding qualitative data have been associated with errors and costly temporal delays. Data obtained from the qualitative analysis of open-ended questions have been questioned for lacking robust insights. The present article introduces an innovative, manual, team-based method of analyzing responses to open-ended survey questions. This method was developed and implemented at the outset of the COVID-19 pandemic to understand the needs of nurses and their perceptions of organizational strategies that were implemented to address pandemic-related challenges. This framework utilizes a dedicated project management structure, general purpose software for data collection and analysis, frame-of-reference training designed for an interdisciplinary team of coders, and data analysis procedures that align with qualitative content analysis procedures. In concert, these techniques empower researchd team members with varying backgrounds and disparate levels of experience to provide unique human insights to data analysis procedures, refine the coding process, and support the abstraction of meaningful themes that were used to prioritize organizational strategies and further support nurses as the pandemic progressed.

The ability of adults with limited expressive language to engage in open-ended interviews about personal experiences.

This study examines how adults with limited expressive language (with average sentences of five words or less) respond to open-ended questions. Participants (n = 49) completed a baseline measure and were then interviewed about a personal experience using exclusively open-ended questions, followed by open-ended and directive questions about a staged event. Their interviews were coded for mean length of utterance (MLU), number of different words and six dimensions of the Narrative Assessment Profile. Descriptively, the participants were able to give some event-related detail in their narratives, but there was wide variability in narrative quality. Correlational and regression analyses indicate that their MLU was stable across contexts. The findings suggest that adults with limited expressive language can provide informative responses to open-ended questions about their experiences, and that their expressive language is likely to show stability across introductory and substantive interview phases.

Assessment to Optimize Learning Strategies: A Qualitative Study of Student and Faculty Perceptions.

Phenomenon: The format of medical knowledge assessment can promote students' use of effective learning strategies from the learning sciences literature, such as elaboration, interleaving, retrieval practice, and distributed learning. Assessment format can also influence faculty teaching. Accordingly, our institution implemented a new assessment strategy in which pre-clerkship medical students answered weekly formative quizzes with constructed response questions (also referred to as open-ended questions) and multiple-choice questions in preparation for summative open-ended question examinations, to support students' use of recommended learning strategies. Our qualitative study explored medical student and faculty perceptions of this assessment strategy on learning and teaching. Approach: We conducted semi-structured interviews with 16 second-year medical students to explore their preparation for quizzes and summative examinations. We also interviewed 10 faculty responsible for writing and grading these assessments in the pre-clerkship foundational sciences curriculum regarding their approach to writing assessments and rubrics, and their perceptions of how their teaching may have changed with this assessment strategy. We analyzed interview transcripts using thematic analysis with a priori sensitizing concepts from the learning sciences literature. Findings: We identified four major themes characterizing student and faculty perceptions of weekly formative quizzes and summative OEQ examinations. Participants found that this assessment strategy helped (1) prioritize conceptual understanding, (2) simulate clinical problem solving, and (3) engage students and faculty in continuous improvement in their approach to learning or teaching. Faculty and students also recognized challenges and potential tradeoffs associated with these assessment formats. Participants identified (4) facilitators and barriers when implementing this assessment strategy. Insights: Our findings suggested that assessment of medical knowledge through weekly formative quizzes and summative open-ended question examinations can facilitate students' use of effective learning strategies. Faculty also recognized improvements in their teaching and in quality of assessment. This format of assessment also presented some challenges and potential tradeoffs and significant institutional resources were required for implementation.

Patient-identified impact of symptoms in spinal and bulbar muscular atrophy.

INTRODUCTION: The effects of spinal bulbar muscular atrophy (SBMA) on quality of life (QoL) are not well understood. This study describes symptoms from the patient's perspective and the impact these symptoms have on QoL. METHODS: We conducted open-ended interviews with 21 adult men with genetically confirmed SBMA. Using a qualitative framework technique, we coded and analyzed interviews to identify symptoms and resulting themes. RESULTS: From these interviews, 729 quotations were extracted. We identified 200 SBMA-specific symptoms and 20 symptomatic themes. Weakness was mentioned by all interviewees. Symptoms within the domain of mental health and the specific themes of emotional issues and psychological impact were also frequently mentioned. DISCUSSION: Numerous symptoms affect QoL for patients with SBMA. We identified previously unrecognized symptoms that are important to address in enhancing clinical care for patients with SBMA and in developing tools to evaluate efficacy in future clinical trials. Muscle Nerve 57: 40-44, 2018.

Validation of the NPAQ-short - a brief questionnaire to monitor physical activity and compliance with the WHO recommendations.

BACKGROUND: Using self-reported surveys to monitor physical activity levels in the population require short items covering both time and intensity. The present study aims to 1) develop the Nordic Physical Activity Questionnaire-short from the original version of the NPAQ, 2) assess test-retest reliability and criterion validity of the NPAQ-short, and 3) test the NPAQ-short's ability to monitor compliance with the WHO recommendations on physical activity. In addition, we aimed to compare open and closed-ended answering modes for the NPAQ-short. METHODS: A sample of 122 participants were included. The NPAQ-short comprised of two questions on weekly moderate-to-vigorous physical activity (MVPA) and vigorous physical activity (VPA). It was filled in twice, two weeks apart, in open- and closed-ended versions. Physical activity was measured objectively by an Actiheart accelerometer worn 24 h/day seven consecutive days. Spearman's rank correlation and Cohen's kappa were used to assess correlations between the test and retest results, and between the objective and the self-reported measures. RESULTS: Valid data was available for 92 participants. Test-retest reliability showed Spearman's rho = 0.82 for MVPA and 0.80 for VPA. For the open-ended questions, the correlations between self-reported and objectively measured physical activity levels were Spearman's rho = 0.33 for MVPA and rho = 0.32 for VPA. For closed-ended questions, the kappa-coefficients were 0.17 for MVPA and 0.21 for VPA. When using objective and self-reported measures to monitor WHO's physical activity recommendations, the kappa correlations were 0.42 for open-ended and 0.34 for closed-ended answering modes. CONCLUSION: The NPAQ-short was found to be sufficiently reliable and valid to monitor physical activity levels in the population when using both open and closed-ended questions. However, using open-ended questions seems to be a better answering mode for self-reported surveys monitoring WHO's physical activity recommendations.

An examination of responses to surveys among Filipino-Australian migrants.

Background Surveys are frequently used to collect data. Although paper surveys are commonly used, online surveys are gaining in popularity, with the inclusion of open-ended questions (OEQs) allowing respondents to freely express their views. Little is known about how Filipino-Australian migrants respond to surveys. There is some concern about the usefulness of OEQs administered to culturally and linguistically diverse migrants, who may have limited capacity to articulate their thoughts in writing. Aim To examine the responses of Filipino-Australian migrants to a survey. Discussion A total of 552 respondents were recruited, of whom 428 (78%) completed the questionnaire online. The overall response rate to the OEQs was 69%, with higher completion rates among those given a paper-based questionnaire and those with university educations. Conclusion Filipino migrants with functional English language skills responded well to the online survey. Paper-based administration elicited more OEQ responses, which is attributed to greater interaction between participants and researchers. Those with university educations may have more capacity to express themselves in English and were therefore more likely to complete the OEQs. Implications for practice The high response rate obtained in this study suggests that among Filipino-Australian migrants who rated their English language skills and educational level highly, the translation of OEQs may not be necessary. This has important implications for resources in research. Face-to-face interaction between participants and researchers is an important strategy for increasing the rates of response to OEQs.

Predicting neuroticism with open-ended response using natural language processing.

Introduction: With rapid advancements in natural language processing (NLP), predicting personality using this technology has become a significant research interest. In personality prediction, exploring appropriate questions that elicit natural language is particularly important because questions determine the context of responses. This study aimed to predict levels of neuroticism-a core psychological trait known to predict various psychological outcomes-using responses to a series of open-ended questions developed based on the five-factor model of personality. This study examined the model's accuracy and explored the influence of item content in predicting neuroticism. Methods: A total of 425 Korean adults were recruited and responded to 18 open-ended questions about their personalities, along with the measurement of the Five-Factor Model traits. In total, 30,576 Korean sentences were collected. To develop the prediction models, the pre-trained language model KoBERT was used. Accuracy, F1 Score, Precision, and Recall were calculated as evaluation metrics. Results: The results showed that items inquiring about social comparison, unintended harm, and negative feelings performed better in predicting neuroticism than other items. For predicting depressivity, items related to negative feelings, social comparison, and emotions showed superior performance. For dependency, items related to unintended harm, social dominance, and negative feelings were the most predictive. Discussion: We identified items that performed better at neuroticism prediction than others. Prediction models developed based on open-ended questions that theoretically aligned with neuroticism exhibited superior predictive performance.

"It's Like Living with a Sassy Teenager!": A Mixed-Methods Analysis of Owners' Comments about Dogs between the Ages of 12 Weeks and 2 Years.

Owners' understanding of dog behaviour influences dog welfare. This study aimed to investigate owners' experiences of living with dogs and perceptions of dog behaviour/behaviour change. Data from an ongoing UK/ROI longitudinal study of dogs were used. Open-ended survey data (n = 3577 comments, n = 1808 dogs) when dogs were 12/16 weeks (data combined), 6, 12, 18 and 24 months were analysed to cover the dog's puppyhood/adolescence. To evaluate the usefulness of open-ended survey questions, both quantitative textual and qualitative thematic analyses were employed. Textual analysis identified an overall positive sentiment at all timepoints; the proportion of positive: negative sentiments increased with the dog's age. Words related to 'love' were the most frequent descriptors at all but the first timepoint, when 'bite' was the most frequent descriptor. Qualitative analysis helped to identify that owners attribute dog behaviour to 'Dog's biology', 'Personality/deliberate action' and 'External influences'. Analysis of open-ended survey responses helped to identify changes in perception over time. When dogs were young, owners described problematic behaviours as 'mischievous', unintentional and context-specific. Similar behaviours shown by older dogs were seen as 'deliberate'. Both positive and negative experiences of dog ownership were identified. However, as not all respondents answered open-ended questions, the generalisability of our findings is limited.

An exploration of what freshness in fruit means to consumers.

Perceived freshness is a key determinant of consumers' food choices but remains an imprecisely defined concept. A comprehensive and consumer-centric definition of freshness appears to be lacking, and the present research was situated in this knowledge gap, seeking, in particular, to explore the complexity of freshness in consumers' minds. People from the USA (n = 2092) took part in an online survey and completed a text highlighting task. Herein, participants read a text that described different aspects of freshness and the technologies used to prolong freshness during storage. While reading, they used highlighting functions within the software to indicate text content that they liked/disliked or agreed/disagreed with. The combined results from text highlighting and responses to the open-ended question "Considering the fruit you eat, why is freshness important to you?" confirmed that freshness is a complex construct with multiple dimensions and suggested that the complexity of freshness extends from food in general to specific product categories (in the case of fruit and apples). Further, the findings showed that freshness is desired by consumers because fruit are perceived as healthier and better tasting. The findings revealed negative attitudes towards stored fruit among participants, but also indicated some acceptance of the fact that some storage was unavoidable. Results provide useful insights to the development for communication strategies to increase consumer acceptance of stored apples and fruits in general.

Study of Consumer Liking of Six Chinese Vinegar Products and the Correlation between These Likings and the Volatile Profile.

As the aroma of Chinese vinegar is a key quality trait that influences consumer liking, a combination of sensory data and instrumental measurements were performed to help understand the aroma differences of six types of Chinese vinegar. A total of 52 volatile compounds, mostly ethyl acetate, acetic acid, and phenethyl alcohol, were detected in six types of Chinese vinegar using solid-phase microextraction coupled with gas chromatography-mass spectrometry (SPME-GC-MS). Combined with open-ended questions, the correlation between consumer liking and the volatile profile of the vinegar was further investigated. More consumers preferred the potato vinegar (B6) described as "having a sweet aroma and fruity vinegar aroma". The Heng-shun Jinyou balsamic vinegar (B5) was not favored by consumers with its exhibition of "too pungent vinegar aroma". Based on their preference patterns, consumers were grouped into three clusters by k-means clustering and principal component analysis (PCA). Using partial least squares regression (PLSR), the most important volatile compounds that drove consumer liking in the three clusters were obtained, among which 14 compounds such as 1-methylpyrrole-2-carboxaldehyde, ethyl acetate, and acetylfuran had the greatest impact on consumer liking, which could guide manufacturers to improve product quality and customer satisfaction.

Examining the consequences of tinnitus using the multidimensional perspective.

BACKGROUND: Tinnitus is one of the most frequent chronic conditions in adults with wide range of consequences. AIMS/OBJECTIVES: The aim of the current study was to determine the problems and life effects reported by individuals with tinnitus using the International Classification of Functioning, Disability and Health (ICF) framework. MATERIAL AND METHODS: The study used a cross-sectional survey design. A total of 344 individuals with tinnitus completed a series of questionnaires. The responses to open-ended questions were linked to ICF categories. RESULTS: Activity limitations and participation restrictions were most dominant consequence of tinnitus followed by effect on the body function with limited emphasis on the contextual factors. Frequently reported responses to body function involved emotional functions (b152), attention function (b140), and sleep functions (b134). Commonly reported responses to activity limitations and participation restrictions were recreation and leisure (d920), conversation (d350), communicating with-receiving-spoken messages (d310), listening (d115), and remunerative employment (d850). Sound intensity (e2500) and sound quality (e2501) were the frequently reported responses to environmental factors. Coping styles, past and present experiences, and lifestyle were the most frequently occurring personal factors. CONCLUSIONS AND SIGNIFICANCE: The study highlighted some key influencing factors of tinnitus in different ICF domains which can be helpful in rehabilitation planning.

Climate Change and Emotions: Analysis of People's Emotional States in Southern Ecuador.

Climate change involves multiple emotional expressions associated with specific labels, notably: 'concern,' 'guilt,' or 'scepticism.' However, there are other types of emotions that have been less analysed, such as 'powerlessness,' 'anger' and 'confusion' that are of equal importance for predicting behavioural changes toward this climatic issue. Likewise, few studies in this research field rely on qualitative data to understand and identify the causative agents for the emotional arousal. This research explores a range of emotions, mixing those that have been widely studied and those that have been hardly analysed. It also looks at the demographic parameters associated with such emotions using a population sample from southern Ecuador. The study analyses quantitative and qualitative data gathered through structured-questionnaires whereby participants were given agency to select and define how they themselves sense emotionally climate change. The results indicate that two of the five participants' most selected emotions are shared with other nations ('concern,' 'guilt'), while the other three have been less reported and studied in the climate change field ('powerlessness,' 'anger,' and 'confusion'). These emotions were found to be aroused by different reasons associated with specific demographic variables. The findings reveal the role of the cultural and local environment in the emotional arousal and its relevance for designing more effective climate communication campaigns.

Improving the Quality of Survey Data on College Campus Woman Abuse: The Contribution of a Supplementary Open-Ended Question.

Obtaining accurate survey data on the prevalence of woman abuse in institutions of higher education continues to be a major methodological challenge. Underreporting is difficult to overcome; yet, there may be effective ways of minimizing this problem. One is adding a supplementary open-ended question to a primarily quantitative questionnaire. Using data derived from the Campus Quality of Life Survey (CQLS), this article examines whether asking respondents to complete such a question increases the prevalence rates of four types of woman abuse and provides information on behaviors that are not included in widely used and validated measures of these harms.

What Did Latino Family Caregivers Expect and Learn From Education Intervention "Caregivers Like Me"?

OBJECTIVE: US Latinos historically have underutilized end-of-life (EOL) resources. This study reports the views of family caregivers before and after education intervention Caregivers Like Me geared to improve knowledge and attitudes regarding EOL resources among Latino caregivers. METHODS: This is a qualitative substudy within a multicentered cross-sectional study design. Educational intervention was offered to family caregivers of Latino elders from 3 different communities, using an audiovisual presentation, including a case-based video telenovela and pre-posttest questionnaires with open-ended questions. This study pertains to the qualitative results for the open-ended questions before and after education intervention. RESULTS: Participants (N = 145) were mostly females (79%) with a mean age of 56 ± 15 years and reported (92%) active learning from intervention. The pretest open-ended question on expected learning identified 2 themes, care for the sick and self-care. It included 5 subthemes and the most common one was how to help the sick. The posttest question on actual learning identified the same 2 themes and 4 subthemes that were similar but different from the pretest. Accepting help and knowledge of services available were the most common. CONCLUSION: The education intervention Caregivers Like Me was able to improve attitudes toward EOL care because participants who wanted to help their loved ones realized that they need to accept professional help and were made aware of the services available.

Corrigendum: The Use of Subscores in Higher Education: When Is This Useful?

[This corrects the article on p. 305 in vol. 8, PMID: 28326049.].

Graduate entry nurses' initial perspectives on nursing: Content analysis of open-ended survey questions.

BACKGROUND: Graduate entry nursing courses offer individuals with prior degrees the opportunity to gain nursing qualifications and facilitate career change. While it is known that accelerated graduate entry courses are increasingly popular, the perceptions of nursing held by such individuals and the influence this has on those seeking to enter the profession are less clearly understood. OBJECTIVE: To explore graduate entry nursing students' perceptions of nursing on entering their pre-registration course. DESIGN: A descriptive design utilising cross-section survey with two open-ended questions: What do you believe the role of the nurse is? What things have influenced that view? were asked. Demographic data were analysed using descriptive frequencies, while the two open-ended questions were analysed using summative content analysis. SETTING: One university-based postgraduate graduate entry nursing course in Australia PARTICIPANTS: Eight cohorts (n=286) commencing students with prior degrees other than nursing. RESULTS: The course attracts students from diverse backgrounds. Exposure to nursing and nurses, either as a consumer of health care or other health care role, plays a primary role in influencing career change. However, similar to those found with school leavers, there remains much misinformation about nurses' roles for students in these courses. Most identify the role of caring in nursing. For some, media representations are the only information sources. DISCUSSION: Graduate entry courses offer opportunities to attract new nurses and contribute to addressing workforce shortages. However, there is still a lack of knowledge of nursing roles among students on entry. More work is required by the profession to ensure nursing is accurately and positively represented to the community.

The Use of Subscores in Higher Education: When Is This Useful?

Assessment in higher education is challenging because teachers face more students, with less contact time as compared to primary and secondary education. Therefore, teachers and management are often interested in efficient ways of giving students diagnostic feedback and providing information on the basis of subscores is one method that is often used in large-scale standardized testing. In this article we discuss some recent psychometric literature that warns against the use of subscores in addition to the use of total scores. We illustrate how the added value of subscores can be evaluated using two college exams: A multiple choice exam and a combined open-ended question and multiple choice exam; these formats are often used in higher education and represent cases in which using subscores may be informative. We discuss the implications of our findings for future classroom evaluation.