RESUMO
PURPOSE: There is little consensus on evidence-based practice guidelines for the selection of criterion-referenced assessments. Having confidence in scores from criterion-referenced assessments requires evidence that items align with their intended constructs. The purposes of these studies were to demonstrate evidence of content validity for the revised item set of a developing social communication assessment and to provide clinicians with a model of content validity evaluations that can be generalised to the review of other assessments. METHOD: In Study 1, 10 experts rated 25 newly-developed items for how well they represented the intended construct. In Study 2, seven participants ages 14-20 were administered the Three Step Test Interview to assess their cognitive processes for responding to new items. Examinee responses were coded for construct-relevant and construct-irrelevant factors. RESULT: Twenty-three of the 25 newly-developed items were deemed representative of the intended construct by experts and elicited construct-relevant response processes from examinees. CONCLUSION: The integration of expert review and examinee cognitive interviewing provides a more complete evaluation of the alignment of the items to their intended construct. Transparent reports of the methods and findings of content validity studies strengthen the ability of clinicians to select criterion-referenced assessments that support valid decisions.
RESUMO
Interventions to facilitate improvement in adolescent social communication are more effective when they are tailored to adolescents' individual profiles of ability. Current social communication assessments for adolescents are not designed to identify their profiles of ability for settings beyond compulsory education. To address this gap, we developed the Transition Pragmatics Interview (TPI). The purpose of these studies was to evaluate the content validity of items developed for the TPI using expert review and cognitive interviewing. Cognitive interviewing is recommended in health-related measurement standards but is not widely reported for assessments of developmental language disorders. Six speech-language pathologists participated in the expert review, rating how well TPI items represented facets of social communication ability. All questions were rated as representative of their intended construct. Eight adolescents (age 14-21) of varied social communication abilities participated in the cognitive interview study to explore whether items were understood as intended by the developers. Participants responded to each item while a researcher observed their response process and asked questions to identify the respondents' thinking about the items. Transcribed responses were classified based on whether they indicated a construct-irrelevant difficulty with the item. Nine of 52 items were identified with recall difficulties, ambiguous wording or potential sources of bias. Cognitive interviewing complemented expert review by identifying issues with content validity not identified by expert review. Items with construct-irrelevant barriers to response will be modified and re-evaluated prior to field testing.
RESUMO
Purpose An important role of social communication assessment is to support intervention planning as adolescents with disabilities transition to workplaces and other adult settings. Adoption of evidence-based assessment practices would align professionals with legal and ethical mandates for schools and other settings. There is, however, limited adoption of evidence-based assessment selection. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) provide a structured approach to identifying assessments with sufficient validity and reliability backed by strong evidence. The purpose of this review was to identify directly administered social communication assessments for adolescents and to apply the COSMIN framework to evaluate their measurement properties. Method A systematic search identified 13 assessments designed for 14- to 21-year-olds with developmental disabilities putting them at risk for social communication deficits. Measurement properties were extracted from included studies and rated for sufficiency. The quality of evidence supporting the findings was graded using COSMIN guidelines. Results Five social communication assessments were recommended as having sufficient evidence of content validity and internal consistency. Remaining assessments could be recommended as gaps in the evidence for their measurement properties are filled. Conclusions The application of the COSMIN standards has the potential to advance evidence-based assessment for social communication. There were important gaps in the independent replication of measurement properties and methodological shortcomings in the evaluation of reliability, measurement error, and structural validity.
