Evaluation of a New Adolescent Social Communication Assessment: The Transition Pragmatics Interview.

PURPOSE: This study evaluated the psychometric functioning of a new criterion-referenced assessment of adolescent social communication, the Transition Pragmatics Interview (TPI), based on the synthesis model of pragmatics. Two ways of interpreting item difficulty were explored: (a) as a function of the synthesis model elements of social communication ability that items were designed to assess, and (b) as a function of the developmental level required for a successful response based on an adapted situational-discourse-semantics (SDS) model (Norris & Hoffman, 1993). METHOD: Thirty-seven participants aged 14-22 years completed the TPI. Responses were analyzed using Rasch analysis to evaluate the functioning of the scale and to determine item difficulty. Items were coded for the SDS developmental level required for an adequate response. The mean Rasch item difficulty for items at each SDS developmental level was analyzed for the five adapted SDS domains. RESULTS: Consistent with the first approach for interpreting item difficulty, TPI items varied in difficulty as a function of the element of social communication they were designed to assess (p < .001). Interpreting item difficulty based on the adapted SDS model was not supported: Items requiring higher SDS developmental levels were not more difficult than those requiring less (p = .55). CONCLUSIONS: The TPI responses fit the Rasch model, supporting the TPI as a unidimensional measure and supporting the use of all items together to compute a single number that summarizes the level of social communication for each examinee. The item ordering from least to most difficult was consistent with prior findings on adolescent social communication development. SUPPLEMENTAL MATERIAL: https://doi.org/10.23641/asha.26018545.

Evidence to support test interpretation: Evaluating the content validity of a new item set for the Transition Pragmatics Interview.

PURPOSE: There is little consensus on evidence-based practice guidelines for the selection of criterion-referenced assessments. Having confidence in scores from criterion-referenced assessments requires evidence that items align with their intended constructs. The purposes of these studies were to demonstrate evidence of content validity for the revised item set of a developing social communication assessment and to provide clinicians with a model of content validity evaluations that can be generalised to the review of other assessments. METHOD: In Study 1, 10 experts rated 25 newly-developed items for how well they represented the intended construct. In Study 2, seven participants ages 14-20 were administered the Three Step Test Interview to assess their cognitive processes for responding to new items. Examinee responses were coded for construct-relevant and construct-irrelevant factors. RESULT: Twenty-three of the 25 newly-developed items were deemed representative of the intended construct by experts and elicited construct-relevant response processes from examinees. CONCLUSION: The integration of expert review and examinee cognitive interviewing provides a more complete evaluation of the alignment of the items to their intended construct. Transparent reports of the methods and findings of content validity studies strengthen the ability of clinicians to select criterion-referenced assessments that support valid decisions.

Exploring the contributions of expert review and cognitive interviewing to evaluating the content validity of items for a new measure of adolescent social communication, the Transition Pragmatics Interview.

Interventions to facilitate improvement in adolescent social communication are more effective when they are tailored to adolescents' individual profiles of ability. Current social communication assessments for adolescents are not designed to identify their profiles of ability for settings beyond compulsory education. To address this gap, we developed the Transition Pragmatics Interview (TPI). The purpose of these studies was to evaluate the content validity of items developed for the TPI using expert review and cognitive interviewing. Cognitive interviewing is recommended in health-related measurement standards but is not widely reported for assessments of developmental language disorders. Six speech-language pathologists participated in the expert review, rating how well TPI items represented facets of social communication ability. All questions were rated as representative of their intended construct. Eight adolescents (age 14-21) of varied social communication abilities participated in the cognitive interview study to explore whether items were understood as intended by the developers. Participants responded to each item while a researcher observed their response process and asked questions to identify the respondents' thinking about the items. Transcribed responses were classified based on whether they indicated a construct-irrelevant difficulty with the item. Nine of 52 items were identified with recall difficulties, ambiguous wording or potential sources of bias. Cognitive interviewing complemented expert review by identifying issues with content validity not identified by expert review. Items with construct-irrelevant barriers to response will be modified and re-evaluated prior to field testing.

Measurement Properties of Social Communication Assessments for Transition-Age Adolescents: A Systematic Review.

Purpose An important role of social communication assessment is to support intervention planning as adolescents with disabilities transition to workplaces and other adult settings. Adoption of evidence-based assessment practices would align professionals with legal and ethical mandates for schools and other settings. There is, however, limited adoption of evidence-based assessment selection. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) provide a structured approach to identifying assessments with sufficient validity and reliability backed by strong evidence. The purpose of this review was to identify directly administered social communication assessments for adolescents and to apply the COSMIN framework to evaluate their measurement properties. Method A systematic search identified 13 assessments designed for 14- to 21-year-olds with developmental disabilities putting them at risk for social communication deficits. Measurement properties were extracted from included studies and rated for sufficiency. The quality of evidence supporting the findings was graded using COSMIN guidelines. Results Five social communication assessments were recommended as having sufficient evidence of content validity and internal consistency. Remaining assessments could be recommended as gaps in the evidence for their measurement properties are filled. Conclusions The application of the COSMIN standards has the potential to advance evidence-based assessment for social communication. There were important gaps in the independent replication of measurement properties and methodological shortcomings in the evaluation of reliability, measurement error, and structural validity.