The Effects of Questionnaire Length on the Relative Impact of Response Styles in Ambulatory Assessment.

Ambulatory assessment (AA) is becoming an increasingly popular research method in the fields of psychology and life science. Nevertheless, knowledge about the effects that design choices, such as questionnaire length (i.e., number of items per questionnaire), have on AA data quality is still surprisingly restricted. Additionally, response styles (RS), which threaten data quality, have hardly been analyzed in the context of AA. The aim of the current research was to experimentally manipulate questionnaire length and investigate the association between questionnaire length and RS in an AA study. We expected that the group with the longer (82-item) questionnaire would show greater reliance on RS relative to the substantive traits than the group with the shorter (33-item) questionnaire. Students (n = 284) received questionnaires three times a day for 14 days. We used a multigroup two-dimensional item response tree model in a multilevel structural equation modeling framework to estimate midpoint and extreme RS in our AA study. We found that the long questionnaire group showed a greater reliance on RS relative to trait-based processes than the short questionnaire group. Although further validation of our findings is necessary, we hope that researchers consider our findings when planning an AA study in the future.

The effects of assessment intensity on participant burden, compliance, within-person variance, and within-person relationships in ambulatory assessment.

Considering the very large number of studies that have applied ambulatory assessment (AA) in the last decade across diverse fields of research, knowledge about the effects that these design choices have on participants' perceived burden, data quantity (i.e., compliance with the AA protocol), and data quality (e.g., within-person relationships between time-varying variables) is surprisingly restricted. The aim of the current research was to experimentally manipulate aspects of an AA study's assessment intensity-sampling frequency (Study 1) and questionnaire length (Study 2)-and to investigate their impact on perceived burden, compliance, within-person variability, and within-person relationships between time-varying variables. In Study 1, students (n = 313) received either 3 or 9 questionnaires per day for the first 7 days of the study. In Study 2, students (n = 282) received either a 33- or 82-item questionnaire three times a day for 14 days. Within-person variability and within-person relationships were investigated with respect to momentary pleasant-unpleasant mood and state extraversion. The results of Study 1 showed that a higher sampling frequency increased perceived burden but did not affect the other aspects we investigated. In Study 2, longer questionnaire length did not affect perceived burden or compliance but yielded a smaller degree of within-person variability in momentary mood (but not in state extraversion) and a smaller within-person relationship between state extraversion and mood. Differences between Studies 1 and 2 with respect to the type of manipulation of assessment intensity are discussed.

Effects of pre-notification, invitation length, questionnaire length and reminder on participation rate: a quasi-randomised controlled trial.

BACKGROUND: Improving participation rates in epidemiologic studies using questionnaires and biological sampling is important for the generalizability of the outcome. The aim of this study was to examine the effects of pre-notification, invitation length, questionnaire length, and reminder on participation rate and to investigate whether some factors contributed to participants doing both the questionnaire and blood sampling as oppose to only one part. METHODS: Our study was embedded within the pilot testing of a large population-based study about prostate cancer screening. Our study sample consisted of 28.134 men between 50 and 69 years of age and living in the region of Stockholm (Sweden) invited to respond to a web-based questionnaire and to provide blood for prostate cancer testing. The men were randomly allocated according to birth of date to receive either: (a) a pre-notification postcard or not; (b) a shorter or a longer invitation letter; (c) a shorter or a longer web-based questionnaire, and (d) a reminder or not. The effects of the survey design factors were tested using chi-square. RESULTS: The use of a pre-notification (p < 0.0001), a longer questionnaire (p = 0.004) and the use of a reminder (p = 0.02) were associated with an increase in overall participation, i.e. responding to the questionnaire or providing blood for PCT or performing both components. CONCLUSIONS: The results of this pilot study justified the use of a pre-notification and a reminder in the following large population based study since the benefits of increased participation traded off against the greater costs incurred. Furthermore, we were able to use the longer version of the questionnaire, which allowed us to collect more information without risking a lower response rate.

Iterative Item Selection of Neighborhood Clusters: A Nonparametric and Non-IRT Method for Generating Miniature Computer Adaptive Questionnaires.

The questionnaire method has always been an important research method in psychology. The increasing prevalence of multidimensional trait measures in psychological research has led researchers to use longer questionnaires. However, questionnaires that are too long will inevitably reduce the quality of the completed questionnaires and the efficiency of collection. Computer adaptive testing (CAT) can be used to reduce the test length while preserving the measurement accuracy. However, it is more often used in aptitude testing and involves a large number of parametric assumptions. Applying CAT to psychological questionnaires often requires question-specific model design and preexperimentation. The present article proposes a nonparametric and item response theory (IRT)-independent CAT algorithm. The new algorithm is simple and highly generalizable. It can be quickly used in a variety of questionnaires and tests without being limited by theoretical assumptions in different research areas. Simulation and empirical studies were conducted to demonstrate the validity of the new algorithm in aptitude tests and personality measures.

Silence is golden, but my measures still see-why cheaper-but-noisier outcome measures in large simple trials can be more cost-effective than gold standards.

OBJECTIVE: To assess the cost-effectiveness of using cheaper-but-noisier outcome measures, such as a short questionnaire, for large simple clinical trials. BACKGROUND: To detect associations reliably, trials must avoid bias and random error. To reduce random error, we can increase the size of the trial and increase the accuracy of the outcome measurement process. However, with fixed resources, there is a trade-off between the number of participants a trial can enrol and the amount of information that can be collected on each participant during data collection. METHODS: To consider the effect on measurement error of using outcome scales with varying numbers of categories, we define and calculate the variance from categorisation that would be expected from using a category midpoint; define the analytic conditions under which such a measure is cost-effective; use meta-regression to estimate the impact of participant burden, defined as questionnaire length, on response rates; and develop an interactive web-app to allow researchers to explore the cost-effectiveness of using such a measure under plausible assumptions. RESULTS: An outcome scale with only a few categories greatly reduced the variance of non-measurement. For example, a scale with five categories reduced the variance of non-measurement by 96% for a uniform distribution. We show that a simple measure will be more cost-effective than a gold-standard measure if the relative increase in variance due to using it is less than the relative increase in cost from the gold standard, assuming it does not introduce bias in the measurement. We found an inverse power law relationship between participant burden and response rates such that a doubling the burden on participants reduces the response rate by around one third. Finally, we created an interactive web-app ( https://benjiwoolf.shinyapps.io/cheapbutnoisymeasures/ ) to allow exploration of when using a cheap-but-noisy measure will be more cost-effective using realistic parameters. CONCLUSION: Cheaper-but-noisier questionnaires containing just a few questions can be a cost-effective way of maximising power. However, their use requires a judgement on the trade-off between the potential increase in risk of information bias and the reduction in the potential of selection bias due to the expected higher response rates.

The Effects of Sampling Frequency and Questionnaire Length on Perceived Burden, Compliance, and Careless Responding in Experience Sampling Data in a Student Population.

Currently, little is known about the association between assessment intensity, burden, data quantity, and data quality in experience sampling method (ESM) studies. Researchers therefore have insufficient information to make informed decisions about the design of their ESM study. Our aim was to investigate the effects of different sampling frequencies and questionnaire lengths on burden, compliance, and careless responding. Students (n = 163) received either a 30- or 60-item questionnaire three, six, or nine times per day for 14 days. Preregistered multilevel regression analyses and analyses of variance were used to analyze the effect of design condition on momentary outcomes, changes in those outcomes over time, and retrospective outcomes. Our findings offer support for increased burden and compromised data quantity and quality with longer questionnaires, but not with increased sampling frequency. We therefore advise against the use of long ESM questionnaires, while high-sampling frequencies do not seem to be associated with negative consequences.

Reducing questionnaire length did not improve physician response rate: a randomized trial.

OBJECTIVES: To examine the effect of reducing questionnaire length on the response rate in a physician survey. STUDY DESIGN AND SETTING: A postal four double-page questionnaire on end-of-life decision making was sent to a random sample of 1,100 general practitioners, 400 elderly care physicians, and 500 medical specialists. Another random sample of 500 medical specialists received a shorter questionnaire of two double pages. After 3 months and one reminder, all nonresponding physicians received an even shorter questionnaire of one double page. RESULTS: Total response was 64% (1,456 of 2,269 eligible respondents). Response rate of medical specialists for the four double-page questionnaire was equal to that of the two double-page questionnaire (190 and 191 questionnaires were returned, respectively). The total response rate increased from 53% to 64% after sending a short one double-page questionnaire (1,203-1,456 respondents). CONCLUSION: The results of our study suggest that reducing the length of a long questionnaire in a physician survey does not necessarily improve response rate. To improve response rate and gather more information, researchers could decide to send a drastically shortened version of the questionnaire to nonresponders.