The role of mediators for the pretesting effect.

Taking a pretest (e.g., smoke - ?) before material is studied (smoke - fog) can improve later recall of that material, compared to material which was initially only studied. The goal of the present study was to evaluate for this pretesting effect the potential role of semantic mediators, i.e., of unstudied information that is semantically related to the study material. In all three experiments, subjects studied weakly associated word pairs (e.g., smoke - fog), half of which received a pretest. Subjects then either completed a recognition test (Experiment 1) or a cued-recall test (Experiments 2 and 3), during which they were presented with both the original study material and never-before-seen semantic mediators that were strongly related to the cue item of a pair (e.g., cigarette). Strikingly, presenting semantic mediators as lures led to higher false alarm rates for mediators following initial pretesting than study only (Experiment 1), and presenting semantic mediators as retrieval cues led to better recall of target items following pretesting than study only (Experiments 2 and 3). We argue that these findings support the elaboration account of the pretesting effect but are difficult to reconcile with other prominent accounts of the effect.

Urdu Receptive Language Scale (URLS): Modification & development of protocol for administration in Urdu.

Objective: To conduct pretesting of Urdu Receptive Language Scale (URLS) for scale modification and development of protocol for administration on children. Methods: This exploratory research was conducted from December 2021 to June 2022 in Lahore, Pakistan. Objective was achieved by pretesting in three steps including literature review, expert review and pilot testing on infants, toddlers and children aged 0-6 years utilizing purposive sampling technique. For expert review five experts' speech language pathologists with minimum five years of experience and for pilot study 48 normal developing children age range birth to six years were recruited. The analysis was done using content validity ratio, content validity index and Cronbach alpha. Protocol for administration of test items and stimulus with minimum three correct responses among four children's responses were finalized. Results: Of the 59 items, 57 with CVR = 0.99 were retained. Item number 3 (age 6-11 months) and item 5 (age 3.6-3.11 years) were eliminated as CVR was <0.99. CVI was one for retained items indicating acceptable validity. Cronbach's alpha was 0.95 indicating high level of internal consistency for the scale. Task groups of protocol of administration directions included: Selection of scale items, testing environment, seating arrangements, rapport building and involving the caregiver, handling the scale material and scoring the task responses. Conclusion: The modified 57 items has high validity and internal consistency with suitable protocol of administration. It is linguistically appropriate for the application on larger scale with children of different cultural backgrounds in Pakistan.

Development, validation & pilot testing of a questionnaire to assess healthcare seeking behaviour, healthcare service utilization & out-of-pocket expenditure of Particularly Vulnerable Tribal Groups of Odisha, India.

Background & objectives: Assessing healthcare seeking behaviour (HSB), healthcare utilization and related out-of-pocket expenditures of Particularly Vulnerable Tribal Groups (PVTGs) of India through a prism of the health system may help to achieve equitable health outcomes. Therefore, this comprehensive study was envisaged to examine these issues among PVTGs of Odisha, India. However, there exists no validated questionnaire to measure these variables among PVTGs. Therefore, a study questionnaire was developed for this purpose and validated. Methods: Questionnaire was constructed in four phases: questionnaire development, validity assessment, pilot testing and reliability assessment. Nine domain experts face validated questionnaire in two rounds, followed by a single round of quantitative content validity. Next, the questionnaire was pretested in three rounds using cognitive interviews and pilot-tested among 335 and 100 eligible individuals for the two sections healthcare seeking behaviour (HSB-Q) and maternal and child healthcare service utilization (MCHSU-Q). Internal consistency reliability was assessed for de novo HSB-Q. Results: On two rounds of expert-driven face validity, 55 items were eliminated from 200 items. Questionnaire showed moderate to high content validity (item-level content validity index range: 0.78 to 1, scale-level content validity index/universal agreement: 0.73; scale-level content validity index/average: 0.96 and multirater kappa statistics range: 0.6 to 1). During the pre-test, items were altered until saturation was achieved. Pilot testing helped to refine interview modalities. The Cronbach alpha and McDonald's omega assessing internal consistency of HSB-Q were 0.8 and 0.85, respectively. Interpretation & conclusions: The questionnaire was found to be valid and reliable to explore healthcare seeking behaviour, maternal and child healthcare utilization and related out-of-pocket expenditure incurred by PVTGs of Odisha, India.

Metacognitive awareness of the pretesting effect improves with self-regulation support.

The pretesting or prequestion effect refers to the counterintuitive finding that taking tests on information that one has yet to learn, during which many erroneous responses typically occur, can benefit learning relative to nontesting methods (e.g., reading) if the correct answers are studied afterwards. Using a knowledge updating approach that entailed two or three cycles of pretesting versus reading followed by a criterial test, we investigated (a) the extent to which learners develop metacognitive awareness of the pretesting effect through experience (as evidenced by predictions of criterial test performance) and (b) three forms of external support-namely, performance feedback (displaying criterial test performance for pretested versus read items), prediction reminders (displaying learners' predictions alongside performance feedback), and recall prompts (asking learners to remember criterial test performance during the first cycle prior to making predictions for the second cycle)-that might improve, or provide insights into, such awareness. Across five experiments, we found that learners generally lack awareness of the memorial benefits of pretesting, are predisposed to believing that reading is more effective even after repeatedly experiencing both techniques, and need support before they recognize that pretesting is more beneficial. Overall, these results underscore the challenge of, and highlight several means of dislodging, learners' inaccurate beliefs about the efficacy of pretesting.

Pre-testing effects are target-specific and are not driven by a generalised state of curiosity.

Guessing an answer to an unfamiliar question prior to seeing the answer leads to better memory than studying alone (the pre-testing effect), which some theories attribute to increased curiosity. A similar effect occurs in general knowledge learning: people are more likely to recall information that they were initially curious to learn. Gruber and Ranganath [(2019). How curiosity enhances hippocampus-dependent memory: The prediction, appraisal, curiosity, and exploration (PACE) framework. Trends in Cognitive Sciences, 23(12), 1014-1025] argued that unanswered questions can cause a state of curiosity during which encoding is enhanced for the missing answer, but also for incidental information presented at the time. If pre-testing similarly induces curiosity, then it too should produce better memory for incidental information. We tested this idea in three experiments that varied the order, nature and timing of the incidental material presented within a pre-testing context. All three experiments demonstrated a reliable pre-testing effect for the targets, but no benefit for the incidental material presented before the target. This pattern suggests that the pre-testing effect is highly specific and is not consistent with a generalised state of curiosity.

The pretesting effect thrives in the presence of competing information.

Taking a pretest (e.g., blanket - ?) before some target material (blanket - sheet) is studied can promote recall of that material on a subsequent final test compared to material which was initially only studied. Here, we examine whether such pretesting can shield the tested material from interference-induced forgetting, which often occurs when before final testing, related material is encountered. We applied a typical pretesting task but asked subjects, between acquisition and final testing of the target list (list 1), to study two additional lists of items with either completely new and unique pairs (e.g., atom - cell) or overlapping - and thus potentially interfering - pairs (e.g., blanket - sleep). Target-list recall on the final test showed a typical pretesting effect for unique pairs, but the size of the effect even increased for overlapping pairs, as recall of study-only pairs was impaired, whereas recall of pretest pairs was left largely unaffected. This held regardless of whether a low (Experiment 1) or high (Experiment 2) degree of learning was induced for the interfering material, suggesting that pretesting can indeed protect the tested material from interference. These findings indicate that pretesting could play a significant role in educational settings where information often needs to be retained in the presence of competing information.

Development of a factorial survey for use in an international study examining clinicians' likelihood to support the decision to initiate invasive long-term ventilation for a child (the TechChild study).

BACKGROUND: The decision to initiate invasive long-term ventilation for a child with complex medical needs can be extremely challenging. TechChild is a research programme that aims to explore the liminal space between initial consideration of such technology dependence and the final decision. This paper presents a best practice example of the development of a unique use of the factorial survey method to identify the main influencing factors in this critical juncture in a child's care. METHODS: We developed a within-subjects design factorial survey. In phase 1 (design) we defined the survey goal (dependent variable, mode and sample). We defined and constructed the factors and factor levels (independent variables) using previous qualitative research and existing scientific literature. We further refined these factors based on expert feedback from expert clinicians and a statistician. In phase two (pretesting), we subjected the survey tool to several iterations (cognitive interviewing, face validity testing, statistical review, usability testing). In phase three (piloting) testing focused on feasibility testing with members of the target population (n = 18). Ethical approval was obtained from the then host institution's Health Sciences Ethics Committee. RESULTS: Initial refinement of factors was guided by literature and interviews with clinicians and grouped into four broad categories: Clinical, Child and Family, Organisational, and Professional characteristics. Extensive iterative consultations with clinical and statistical experts, including analysis of cognitive interviews, identified best practice in terms of appropriate: inclusion and order of clinical content; cognitive load and number of factors; as well as language used to suit an international audience. The pilot study confirmed feasibility of the survey. The final survey comprised a 43-item online tool including two age-based sets of clinical vignettes, eight of which were randomly presented to each participant from a total vignette population of 480. CONCLUSIONS: This paper clearly explains the processes involved in the development of a factorial survey for the online environment that is internationally appropriate, relevant, and useful to research an increasingly important subject in modern healthcare. This paper provides a framework for researchers to apply a factorial survey approach in wider health research, making this underutilised approach more accessible to a wider audience.

Pretesting can be beneficial even when using the internet to answer questions.

Research on the pretesting effect has shown that attempting to retrieve or generate information, even when unsuccessful, can potentiate the subsequent learning and remembering of that information. In the current research, we tested the hypothesis that when information can be accessed online, people may be less likely to retrieve or generate information on their own, thus making them less likely to benefit from the pretesting effect. The results of two experiments failed to provide support for this hypothesis. Participants remembered pretested information better than non-pretested information regardless of whether they were required to attempt to retrieve answers from memory or search for the answers using Google. The results suggest that the benefits of pretesting can be observed even when people rely on the internet to answer the questions they encounter.

Viewpoint: Do not resurrect the two-stage procedure.

In 1989, Peter Freeman published a paper that challenged a commonly accepted approach for analyzing cross-over trials, the so-called two-stage procedure. Freeman himself recommended using the Bayesian approach of Andy Grieve. The flaws Freeman exposed were serious and led many statisticians to conclude that the procedure was unacceptable. Unfortunately, more than 30 years later, one still encounters its use. This note explains, using a simple simulation, why the two-stage procedure is, indeed, as Freeman showed unacceptable and should not be used.

An analysis of social marketing practice: Factors associated with success.

This paper aims to identify factors that contribute to the success of current social marketing practices. These factors include setting clear behavior change objectives and segmentation that informs communication and messaging strategies. Other factors include rigorous research (consumer research, formative research, literature review), pre-testing of interventions, developing a partnership approach, using planning methodologies/theories, and monitoring and evaluation. These success factors could be used for policymakers, governments, agencies and social marketers delivering interventions focussed on healthy lives and well-being. The examples given in this study illustrate how these factors can be achieved, providing a focus for discussion and emulation.

A little can go a long way: giving learners some context can enhance the benefits of pretesting.

Tests given to learners before they study new information can enhance the learning of that information. When responding to these pretests, learners typically generate answers that are incorrect but that are nevertheless helpful for improving the learning of the correct answers. The present research examined how providing learners with context prior to pretesting can enhance the benefits of pretesting. Across two experiments, participants were given a pretest for half of the to-be-learned information and then asked to read a passage about a fictional topic, an alien civilisation known as Yoffas (Experiment 1 and Experiment 2), or a unique fruit called the Anona (Experiment 2). Participants who read a short paragraph contextualising the to-be-learned information exhibited a significantly larger pretesting effect than participants who did not, with this interaction being observed regardless of whether memory was tested after a 5-min delay or 1-week delay, and regardless of whether contextualisation was manipulated between-subjects or within-subjects. These results suggest that what learners know prior to a pretest can have an impact on the extent to which learners benefit from that pretest.

Learning from errors: students' and instructors' practices, attitudes, and beliefs.

In some educational contexts, such as during assessments, it is essential to avoid errors. In other contexts, however, generating an error can foster valuable learning opportunities. For instance, generating errors can improve memory for correct answers. In two surveys conducted at three large public universities in North America, we investigated undergraduate students' and instructors' awareness of the pedagogical benefits of generating errors, as well as related practices, attitudes, and beliefs. Surveyed topics included the incorporation of errors into learning activities, opinions about the consequences of studying errors, and approaches to feedback. Many students had an aversion towards making errors during learning and did not use opportunities to engage in errorful generation, yet studied or analysed errors when they occurred. Many instructors had a welcoming attitude towards errors that occur during learning, yet varied in providing students with resources that facilitate errorful generation. Overall, these findings reveal the prevalence of an ambivalent approach to errors: Students and instructors avoid generating errors but prioritise learning from them when they occur. These results have important implications for the implementation of pretesting, productive failure, and other error-focused learning techniques in educational contexts.

Enhancing validity through cognitive interviewing. A methodological example using the Epilepsy Monitoring Unit Comfort Questionnaire.

AIMS: To identify problematic items, assess completeness and user-friendliness, and undertake modifications to enhance face validity of the newly developed Epilepsy Monitoring Unit Comfort Questionnaire. DESIGN: Qualitative-descriptive cross-sectional study. METHODS: Five iterative rounds of cognitive interviewing were conducted with members of the target population between July - November 2017. Think-aloud technique, verbal probing, and observation were used, to assess how respondents understood and answered questions. Data analysis was based on the framework method; an eight-column framework matrix was created for this purpose. RESULTS: In 25 interviews, problems regarding completeness of the item pool, comprehension of items, retrieval of information, judgment while finding answers and reporting the appropriate responses could be detected. According to the results, 27 items remained unchanged, 11 items were reworded, and six items were added. Instructions section of the questionnaire was reworked, too. CONCLUSION: Although time-consuming, cognitive interviewing turned out to be a valuable approach for revealing problems in an instrument, which would, otherwise, remain undetected and threaten validity.

How Does Pretesting for PubMed Knowledge Spark Student Learning?

Predoctoral students enter dental school with varying skill levels for searching biomedical databases and a tendency to overestimate their abilities. Accordingly, PubMed instruction is embedded within a required dental course and includes a graded component. This article describes a pretest/intervention/posttest developed for the PubMed session. The expectation for this new assessment was that motivation to learn PubMed would increase during the intervention if pretesting objectively showed students the difference between their self-perceived versus actual PubMed abilities. The goals were to help students better self-assess their genuine searching abilities, spark learning during the instruction session, and elicit measurable improvement in skills.

Using cognitive pre-testing methods in the development of a new evidenced-based pressure ulcer risk assessment instrument.

BACKGROUND: Variation in development methods of Pressure Ulcer Risk Assessment Instruments has led to inconsistent inclusion of risk factors and concerns about content validity. A new evidenced-based Risk Assessment Instrument, the Pressure Ulcer Risk Primary Or Secondary Evaluation Tool - PURPOSE-T was developed as part of a National Institute for Health Research (NIHR) funded Pressure Ulcer Research Programme (PURPOSE: RP-PG-0407-10056). This paper reports the pre-test phase to assess and improve PURPOSE-T acceptability, usability and confirm content validity. METHODS: A descriptive study incorporating cognitive pre-testing methods and integration of service user views was undertaken over 3 cycles comprising PURPOSE-T training, a focus group and one-to-one think-aloud interviews. Clinical nurses from 2 acute and 2 community NHS Trusts, were grouped according to job role. Focus group participants used 3 vignettes to complete PURPOSE-T assessments and then participated in the focus group. Think-aloud participants were interviewed during their completion of PURPOSE-T. After each pre-test cycle analysis was undertaken and adjustment/improvements made to PURPOSE-T in an iterative process. This incorporated the use of descriptive statistics for data completeness and decision rule compliance and directed content analysis for interview and focus group data. Data were collected April 2012-June 2012. RESULTS: Thirty-four nurses participated in 3 pre-test cycles. Data from 3 focus groups, 12 think-aloud interviews incorporating 101 PURPOSE-T assessments led to changes to improve instrument content and design, flow and format, decision support and item-specific wording. Acceptability and usability were demonstrated by improved data completion and appropriate risk pathway allocation. The pre-test also confirmed content validity with clinical nurses. CONCLUSIONS: The pre-test was an important step in the development of the preliminary PURPOSE-T and the methods used may have wider instrument development application. PURPOSE-T proposes a new approach to pressure ulcer risk assessment, incorporating a screening stage, the inclusion of skin status to distinguish between those who require primary prevention and those who require secondary prevention/treatment and the use of colour to support pathway allocation and decision making. Further clinical evaluation is planned to assess the reliability and validity of PURPOSE-T and it's impact on care processes and patient outcomes.

Multiple-choice pretesting potentiates learning of related information.

Although the testing effect has received a substantial amount of empirical attention, such research has largely focused on the effects of tests given after study. The present research examines the effect of using tests prior to study (i.e., as pretests), focusing particularly on how pretesting influences the subsequent learning of information that is not itself pretested but that is related to the pretested information. In Experiment 1, we found that multiple-choice pretesting was better for the learning of such related information than was cued-recall pretesting or a pre-fact-study control condition. In Experiment 2, we found that the increased learning of non-pretested related information following multiple-choice testing could not be attributed to increased time allocated to that information during subsequent study. Last, in Experiment 3, we showed that the benefits of multiple-choice pretesting over cued-recall pretesting for the learning of related information persist over 48 hours, thus demonstrating the promise of multiple-choice pretesting to potentiate learning in educational contexts. A possible explanation for the observed benefits of multiple-choice pretesting for enhancing the effectiveness with which related nontested information is learned during subsequent study is discussed.

Development and testing of study tools and methods to examine ethnic bias and clinical decision-making among medical students in New Zealand: The Bias and Decision-Making in Medicine (BDMM) study.

BACKGROUND: Health provider racial/ethnic bias and its relationship to clinical decision-making is an emerging area of research focus in understanding and addressing ethnic health inequities. Examining potential racial/ethnic bias among medical students may provide important information to inform medical education and training. This paper describes the development, pretesting and piloting of study content, tools and processes for an online study of racial/ethnic bias (comparing Maori and New Zealand European) and clinical decision-making among final year medical students in New Zealand (NZ). METHODS: The study was developed, pretested and piloted using a staged process (eight stages within five phases). Phase 1 included three stages: 1) scoping and conceptual framework development; 2) literature review and identification of potential measures and items; and, 3) development and adaptation of study content. Three main components were identified to assess different aspects of racial/ethnic bias: (1) implicit racial/ethnic bias using NZ-specific Implicit Association Tests (IATs); (2) explicit racial/ethnic bias using direct questions; and, (3) clinical decision-making, using chronic disease vignettes. Phase 2 (stage 4) comprised expert review and refinement. Formal pretesting (Phase 3) included construct testing using sorting and rating tasks (stage 5) and cognitive interviewing (stage 6). Phase 4 (stage 7) involved content revision and building of the web-based study, followed by pilot testing in Phase 5 (stage 8). RESULTS: Materials identified for potential inclusion performed well in construct testing among six participants. This assisted in the prioritisation and selection of measures that worked best in the New Zealand context and aligned with constructs of interest. Findings from the cognitive interviewing (nine participants) on the clarity, meaning, and acceptability of measures led to changes in the final wording of items and ordering of questions. Piloting (18 participants) confirmed the overall functionality of the web-based questionnaire, with a few minor revisions made to the final study. CONCLUSIONS: Robust processes are required in the development of study content to assess racial/ethnic bias in order to optimise the validity of specific measures, ensure acceptability and minimise potential problems. This paper has utility for other researchers in this area by informing potential development approaches and identifying possible measurement tools.

Forward and Back is Not Enough: Applying Best Practices for Translation of Pediatric Sleep Questionnaires.

Cultural differences in the experience of sleep warrant consideration in the measurement of sleep across populations. This requires careful attention to both language and culture when translating survey measures. While forward and back translation is the most commonly used approach, it has numerous limitations if used as an isolated method. Best practice guidelines recommend a multi-step team-based approach for translating questionnaires. We present our recent experience applying best practices in a study with both Spanish and English-speaking Mexican American mothers of toddlers. This work is part of a larger project that will measure parental sleep-related beliefs and parenting practices in Mexican American parents of toddlers. We utilized a team-based approach to translation and cultural adaptation, assembling a diverse, bilingual, and bicultural team. The translation process started with items and measures that we had selected, revised as needed, or created. New items were based on constructs identified in semi-structured interviews and focus groups used to explore parental sleep-related beliefs and parenting practices in the target population. Following this, our translation process included forward and back translation, harmonization and decentering, cognitive interviewing, debriefing, adjudication, and proofreading. We outline details of our process and the rationale for each step. We also highlight how each step contributes to ensuring culturally appropriate items with conceptual equivalence across languages. To ensure inclusivity and scientific rigor within the field of sleep research, investigators must utilize best practices for translations and cultural adaptations, building on the foundation of cultural constructs often identified in qualitative work.

Developing a person-centered stated preference survey for dementia with Lewy bodies: value of a personal and public involvement process.

Introduction: The development of high-quality stated preference (SP) surveys requires a rigorous design process involving engagement with representatives from the target population. However, while transparency in the reporting of the development of SP surveys is encouraged, few studies report on this process and the outcomes. Recommended stages of instrument development includes both steps for stakeholder/end-user engagement and pretesting. Pretesting typically involves interviews, often across multiple waves, with improvements made at each wave; pretesting is therefore resource intensive. The aims of this paper are to report on the outcomes of collaboration with a Lewy body dementia research advisory group during the design phase of a SP survey. We also evaluate an alternative approach to instrument development, necessitated by a resource constrained context. Method: The approach involved conducting the stages of end-user engagement and pretesting together during a public involvement event. A hybrid approach involving a focus group with breakout interviews was employed. Feedback from contributors informed the evolution of the survey instrument. Results: Changes to the survey instrument were organized into four categories: attribute modifications; choice task presentation and understanding; information presentation, clarity and content; and best-best scaling presentation. The hybrid approach facilitated group brainstorming while still allowing the researcher to assess the feasibility of choice tasks in an interview setting. However, greater individual exploration and the opportunity to trial iterative improvements across waves was not feasible with this approach. Discussion: Involvement of the research advisory group resulted in a more person-centered survey design. In a context constrained by time and budget, and with consideration of the capacity and vulnerability of the target population, the approach taken was a feasible and pragmatic mechanism for improving the design of a SP survey.

Broader benefits of the pretesting effect: Placement matters.

Taking a test before learning new information in a lesson improves memory for that information - pretesting effect. Although the specific memory benefit of a pretest on pretested information has been well documented, it remains unclear what the circumstances necessary for the broader memory benefit are - that is, the benefit of a pretest on memory of information in the lesson that was not pretested. Sometimes this broader benefit is present, but other times it disappears or reverses. We investigated if manipulating where the non-pretested information appears in a lesson - either before or after the pretested information - affects broader memory benefits. Participants read a text passage (Experiment 1) or watched a video lecture (Experiment 2) after completing a pretest on half of the lesson content. The pretested information appeared either at the beginning (prior to the non-pretested information) or at the end (after the non-pretested information) of the lesson. The final test assessed memory of both pretested and non-pretested information. We hypothesized that pretests trigger an attentional window that opens during the lesson and closes after pretested information has been identified. Any information, including non-pretested information, will benefit from being in this window because it is more likely to be processed. We found that memory of non-pretested information is better if the non-pretested information is presented at the beginning versus at the end of a lesson, regardless of delivery modality. These results indicate that the presentation order of pretested versus non-pretested information contributes to the broader memory benefits associated with pretesting.