Priority skills for equity-focused, evidence-based cancer control in community-based organizations: A group concept mapping analysis with academics and practitioners.

Introduction: Community-based organizations (CBOs) are important equity-promoting delivery channels for evidence-based interventions (EBIs). However, CBO practitioners often cannot access needed support to build EBI skills. Additionally, the capacity-building literature is hindered by inconsistent definitions, limited use of validated measures, and an emphasis on the perspectives of EBI developers versus implementers. To address these gaps, we explored commonalities and differences between CBO practitioners and academics in conceptualizing and prioritizing core EBI skills. Methods: We utilized Group Concept Mapping, a mixed-methods approach connecting qualitative data (e.g., regarding the range of critical EBI skills) and quantitative data (e.g., sorting and ranking data regarding unique skills) to create conceptual maps integrating perspectives from diverse participants. A total of 34 practitioners and 30 academics working with cancer inequities participated in the study. Results: Participants nominated 581 core skills for EBI use, and our team (including practitioners and academics) identified 98 unique skills from this list. Participants sorted them into conceptual groups, yielding five clusters: (1) using data and evaluation, (2) selecting and adapting EBIs, (3) connecting with community members, (4) building diverse and equitable partnerships, and (5) managing EBI implementation. The ordering of importance and presence of skill clusters were similar across groups. Overall, importance was rated higher than presence, suggesting capacity gaps. Conclusions: There are helpful commonalities between practitioners' and academics' views of core EBI skills in CBOs and apparent capacity gaps. However, underlying patterns suggest that differences between the groups' perceptions warrant further exploration.

An Exploration of Critical Care Professionals' Strategies to Enhance Daily Implementation of the Assess, Prevent, and Manage Pain; Both Spontaneous Awakening and Breathing Trials; Choice of Analgesia and Sedation; Delirium Assess, Prevent, and Manage; Early Mobility and Exercise; and Family Engagement and Empowerment: A Group Concept Mapping Study.

The goals of this exploratory study were to engage professionals from the Society for Critical Care Medicine ICU Liberation Collaborative ICUs to: 1) conceptualize strategies to enhance daily implementation of the Assess, prevent, and manage pain; Both spontaneous awakening and breathing trials; Choice of analgesia and sedation; Delirium assess, prevent, and manage; Early mobility and exercise; and Family engagement and empowerment (ABCDEF) bundle from different perspectives and 2) identify strategies to prioritize for implementation. DESIGN: Mixed-methods group concept mapping over 8 months using an online method. Participants provided strategies in response to a prompt about what was needed for successful daily ABCDEF bundle implementation. Responses were summarized into a set of unique statements and then rated on a 5-point scale on degree of necessity (essential) and degree to which currently used. SETTING: Sixty-eight academic, community, and federal ICUs. PARTICIPANTS: A total of 121 ICU professionals consisting of frontline and leadership professionals. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: A final set of 76 strategies (reduced from 188 responses) were suggested: education (16 strategies), collaboration (15 strategies), processes and protocols (13 strategies), feedback (10 strategies), sedation/pain practices (nine strategies), education (eight strategies), and family (five strategies). Nine strategies were rated as very essential but infrequently used: adequate staffing, adequate mobility equipment, attention to (patient's) sleep, open discussion and collaborative problem solving, nonsedation methods to address ventilator dyssynchrony, specific expectations for night and day shifts, education of whole team on interdependent nature of the bundle, and effective sleep protocol. CONCLUSIONS: In this concept mapping study, ICU professionals provided strategies that spanned a number of conceptual implementation clusters. Results can be used by ICU leaders for implementation planning to address context-specific interdisciplinary approaches to improve ABCDEF bundle implementation.

Exploring Characteristics of Academic General Surgery Residency Applicants: A Group Concept-Mapping Approach.

OBJECTIVE: Holistic review, which emphasizes qualitative attributes over objective measures, has been proposed as a method for selecting candidates for surgical residency in order to improve diversity in graduate medical education, and, ultimately, the field of surgery. This study seeks to articulate desirable traits of applicants as a first-step in standardizing the holistic review process. DESIGN: Using Group Concept Mapping, a web-based mixed-methods participatory research methodology, residency selection committee members were asked to 1) list desirable characteristics of applicants, 2) group these into categories, 3) rate their importance to academic/clinical success on a 5-point Likert scale (1 = not at all important, 5 = extremely important), and 4) rate the degree to which each characteristic is feasible to assess on a 3-point Likert scale (1 = not at all feasible, 3 = very feasible). Grouped characteristics submitted to hierarchical cluster analysis depicted committee's consensus about desirable qualities/criteria for applicants. Bivariate scatter-plots and pattern-matching graphics demonstrated which of these criteria were most important and reliably assessed. SETTING: A single academic general surgery residency training program in Western Pennsylvania. PARTICIPANTS: Members of the selection committee for the UPMC General Surgery Residency program who had participated in at least 1 prior cycle of applicant selection. RESULTS: Desirable characteristics of highly qualified applicants into an academic general surgery residency were clustered into domains of 1) scholarly work and research, 2) grades/formal assessments, 3) program fit, 4) behavioral assets, and 5) aspiration. Behavioral assets, which was felt to be the most important to clinical and academic success were considered to be the least feasible to reliably assess. Within this domain, initiative, being self-motivated, intellectual curiosity, work ethic, communication skills, maturity and self-awareness, and thoughtfulness were viewed as most frequently reliably assessed from the application and interview process. CONCLUSIONS: High quality applicants possess several behavioral assets that faculty deem are important to academic and clinical success. Adapting validated metrics for assessing these assets, may provide a solution for addressing subjectivity and other challenges scrutinized by critics of holistic review.

A large-scale, geographical approach to using group concept mapping for planning: The Teen Opinions Count (TOC) project.

The use of group concept mapping (GCM) for large-scale, multi-site planning has been limited. The Teens Opinions Count (TOC) project utilized the group concept mapping methodology to develop a framework for after school time program development for teens in 16 counties in the Southeast Michigan and Western New York regions. This extensive planning effort demonstrated a large-scale, geographical approach to the implementation of GCM, where multiple sites (in this case counties) simultaneously engaged in independent processes on the same topic, with the purpose of producing separate locale-specific conceptualizations. TOC was the largest GCM project ever conducted in terms of the number of participants and volume of data submitted. This report details GCM recruitment, data collection, data compilation, and analysis of ideas from over 20,000 teens and 2,000 adults. Although implementation was conducted at the county level, many of the separate county-level GCM processes were larger than what is typically encountered in the literature. Operationalizing GCM at such a scale required significant planning, design, communication, and data management solutions. The extensive data collection effort required the need for creative and innovative procedures to engage teens and process the sheer volume of data. The lessons learned are discussed and suggestions for future applications offered.

Differences in Community Members' and Academics' Perceptions of Factors Contributing to Food Insecurity and Obesity among Mexican Immigrants: An Application of Concept Mapping.

Community-based participatory research has been identified as an approach to research where community members, organizational representatives, and academic researchers all participate in the research process. As research partners, community members and academics may find some common ground in identification of problems, but frequently differ in their perception of the cause of those problems and struggle with arriving at a "shared interpretation" of the data. Failure to account for these differences ultimately leads to issues with the design of interventions. Group concept mapping is a rigorous mixed-methods approach that lends itself well to the conduct of community-based participatory research and the results can be used to systematically compare the conceptualized models of different groups of participants. Building from a previous study, where group concept mapping was used to describe the factors felt to contribute to weight gain and obesity among Mexican immigrant families, this study described and statistically evaluated the different conceptualizations between community members and academicians. Using a Procrustean comparison approach and permutation strategy, we assessed the spatial correspondence of concept maps generated by the two independent community members and academic groups. The results indicated the conceptualized patterns, represented by the two multidimensional scaling configurations (i.e., concept maps) produced by community and academic groups, were more similar than expected by chance, although meaningful differences existed. We describe how these differences in perspective have important implications for the cooperative development of an intervention addressing the multiple factors related to the perceived causes of obesity.

Using Concept Mapping within a Community-Academic Partnership to Examine Obesity among Mexican Immigrants.

Background: Weight gain is common following migration to a new country. Mexican immigrants have a disparate prevalence of overweight/obesity and food insecurity. Social stressors, such as unemployment, discrimination, and the threat of deportation, may fuel both food insecurity and weight gain in this population. Objectives: We sought to (1) examine community-defined causes and correlates of obesity among Mexican-Americans; (2) determine how current social stressors, policies, and programs impact food insecurity and obesity; and (3) identify community-defined priorities for preventive interventions. Methods: Group concept mapping (GCM) was used in a community-academic partnership (CAP) to describe the factors contributing to weight gain and obesity among Mexican immigrant families. Activities included community brainstorming, sorting and rating, multivariate statistical analysis, and community interpretation of results. Results: Eighty statements were generated in the brainstorming sessions. These statements were sorted into nine clusters, which were organized into three regions: (1) intrapersonal factors; (2) community-level factors; and (3) social policy-related barriers. Statements reflecting the impact of immigration-related stressors were found in all three regions, addressing participants' fears of deportation, and the prioritization of resources away from healthy eating, resulting in food insecurity. Community members identified five priority areas for intervention planning: (1) lack of exercise; (2) lack of knowledge of a healthy diet; (3) expense of healthy foods; (4) "junk" food; and (5) stress management. Conclusions: Results suggest high levels of social stress are contributing to food insecurity and obesity among Mexican immigrant families. Areas identified for intervention planning reflect the need for a multifaceted approach toward obesity prevention.

Computer-Based Training in Eating and Nutrition Facilitates Person-Centered Hospital Care: A Group Concept Mapping Study.

Studies have shown that computer-based training in eating and nutrition for hospital nursing staff increased the likelihood that patients at risk of undernutrition would receive nutritional interventions. This article seeks to provide understanding from the perspective of nursing staff of conceptually important areas for computer-based nutritional training, and their relative importance to nutritional care, following completion of the training. Group concept mapping, an integrated qualitative and quantitative methodology, was used to conceptualize important factors relating to the training experiences through four focus groups (n = 43), statement sorting (n = 38), and importance rating (n = 32), followed by multidimensional scaling and cluster analysis. Sorting of 38 statements yielded four clusters. These clusters (number of statements) were as follows: personal competence and development (10), practice close care development (10), patient safety (9), and awareness about the nutrition care process (9). First and second clusters represented "the learning organization," and third and fourth represented "quality improvement." These findings provide a conceptual basis for understanding the importance of training in eating and nutrition, which contributes to a learning organization and quality improvement, and can be linked to and facilitates person-centered nutritional care and patient safety.

Essential Public Health Competencies for Medical Students: Establishing a Consensus in Family Medicine.

Phenomenon: The integration of public health (PH) competency training into medical education, and further integration of PH and primary care, has been urged by the U.S. Institute of Medicine. However, PH competencies are numerous, and no consensus exists over which competencies are most important for adoption by current trainees. Our objective was to conduct a group concept mapping exercise with stakeholders identifying the most important and feasible PH skills to incorporate in medical and residency curricula. APPROACH: We utilized a group concept mapping technique via the Concept System Global Max ( http://www.conceptsystems.com ), where family medicine educators and PH professionals completed the phrase, "A key Public Health competency for physicians-in-training to learn is …" with 1-10 statements. The statement list was edited for duplication and other issues; stakeholders then sorted the statements and rated them for importance and feasibility of integration. Multidimensional scaling and cluster analysis were used to create a two-dimensional point map of domains of PH training, allowing visual comparison of groupings of related ideas and relative importance of these ideas. FINDINGS: There were 116 nonduplicative statements (225 total) suggested by 120 participants. Three metacategories of competencies emerged: Clinic, Community & Culture, Health System Understanding, and Population Health Science & Data. Insights: We identified and organized a set of topics that serve as a foundation for the integration of family medicine and PH education. Incorporating these topics into medical education is viewed as important and feasible by family medicine educators and PH professions.

The use of concept mapping in measurement development and evaluation: Application and future directions.

The past decade has seen an increase of measurement development research in social and health sciences that featured the use of concept mapping as a core technique. The purpose, application, and utility of concept mapping have varied across this emerging literature. Despite the variety of uses and range of outputs, little has been done to critically review how researchers have approached the application of concept mapping in the measurement development and evaluation process. This article focuses on a review of the current state of practice regarding the use of concept mapping as methodological tool in this process. We systematically reviewed 23 scale or measure development and evaluation studies, and detail the application of concept mapping in the context of traditional measurement development and psychometric testing processes. Although several limitations surfaced, we found several strengths in the contemporary application of the method. We determined concept mapping provides (a) a solid method for establishing content validity, (b) facilitates researcher decision-making, (c) insight into target population perspectives that are integrated a priori, and (d) a foundation for analytical and interpretative choices. Based on these results, we outline how concept mapping can be situated in the measurement development and evaluation processes for new instrumentation.

Concept mapping-An effective method for identifying diversity and congruity in cognitive style.

This paper investigates the effects of cognitive style for decision making on the behaviour of participants in different phases of the group concept mapping process (GCM). It is argued that cognitive style should be included directly in the coordination of the GCM process and not simply considered as yet another demographic variable. The cognitive styles were identified using the Kirton Adaption-Innovation Inventory, which locates each person's style on a continuum ranging from very adaptive to very innovative. Cognitive style could explain diversity in the participants' behaviour in different phases of the GCM process. At the same time, the concept map as a group's common cognitive construct can consolidate individual differences and serves as a tool for managing diversity in groups of participants. Some of the results were that: (a) the more adaptive participants generated ideas that fit to a particular, well-established and consensually agreed paradigm, frame of reference, theory or practice; (b) the more innovative participants produced ideas that were more general in scope and required changing a settled structure (paradigm, frame of reference, theory or practice); and (c) the empirical comparison of the map configurations through Procrustes analysis indicated a strong dissimilarity between cognitive styles.

Systems thinking and complexity: considerations for health promoting schools.

The health promoting schools concept reflects a comprehensive and integrated philosophy to improving student and personnel health and well-being. Conceptualized as a configuration of interacting, interdependent parts connected through a web of relationships that form a whole greater than the sum of its parts, school health promotion initiatives often target several levels (e.g. individual, professional, procedural and policy) simultaneously. Health promoting initiatives, such as those operationalized under the whole school approach, include several interconnected components that are coordinated to improve health outcomes in complex settings. These complex systems interventions are embedded in intricate arrangements of physical, biological, ecological, social, political and organizational relationships. Systems thinking and characteristics of complex adaptive systems are introduced in this article to provide a perspective that emphasizes the patterns of inter-relationships associated with the nonlinear, dynamic and adaptive nature of complex hierarchical systems. Four systems thinking areas: knowledge, networks, models and organizing are explored as a means to further manage the complex nature of the development and sustainability of health promoting schools. Applying systems thinking and insights about complex adaptive systems can illuminate how to address challenges found in settings with both complicated (i.e. multi-level and multisite) and complex aspects (i.e. synergistic processes and emergent outcomes).

Reasons for electronic cigarette use beyond cigarette smoking cessation: A concept mapping approach.

INTRODUCTION: Electronic cigarettes (ECIGs) continue to grow in popularity, however, limited research has examined reasons for ECIG use. METHODS: This study used an integrated, mixed-method participatory research approach called concept mapping (CM) to characterize and describe adults' reasons for using ECIGs. A total of 108 adults completed a multi-module online CM study that consisted of brainstorming statements about their reasons for ECIG use, sorting each statement into conceptually similar categories, and then rating each statement based on whether it represented a reason why they have used an ECIG in the past month. RESULTS: Participants brainstormed a total of 125 unique statements related to their reasons for ECIG use. Multivariate analyses generated a map revealing 11, interrelated components or domains that characterized their reasons for use. Importantly, reasons related to Cessation Methods, Perceived Health Benefits, Private Regard, Convenience and Conscientiousness were rated significantly higher than other categories/types of reasons related to ECIG use (p<.05). There also were significant model differences in participants' endorsement of reasons based on their demography and ECIG behaviors. CONCLUSIONS: This study shows that ECIG users are motivated to use ECIGs for many reasons. ECIG regulations should address these reasons for ECIG use in addition to smoking cessation.

Community Readiness Within Systems of Care: The Validity and Reliability of the System of Care Readiness and Implementation Measurement Scale (SOC-RIMS).

Establishing a system of care requires communities to identify ways to successfully implement strategies and support positive outcomes for children and their families. Such community transformation is complex and communities vary in terms of their readiness for implementing sustainable community interventions. Assessing community readiness and guiding implementation, specifically for the funded communities implementing a system of care, requires a well-designed tool with sound psychometric properties. This scale development study used the results of a previously published concept mapping study to create, administer, and assess the psychometric characteristics of the System of Care Readiness and Implementation Measurement Scale (SOC-RIMS). The results indicate the SOC-RIMS possesses excellent internal consistency characteristics, measures clearly discernible dimensions of community readiness, and demonstrates the target constructs exist within a broad network of content. The SOC-RIMS can be a useful part of a comprehensive assessment in communities where system of care practices, principles, and philosophies are implemented and evaluated.

Evaluating protocol lifecycle time intervals in HIV/AIDS clinical trials.

BACKGROUND: Identifying efficacious interventions for the prevention and treatment of human diseases depends on the efficient development and implementation of controlled clinical trials. Essential to reducing the time and burden of completing the clinical trial lifecycle is determining which aspects take the longest, delay other stages, and may lead to better resource utilization without diminishing scientific quality, safety, or the protection of human subjects. PURPOSE: In this study, we modeled time-to-event data to explore relationships between clinical trial protocol development and implementation times, as well as to identify potential correlates of prolonged development and implementation. METHODS: We obtained time interval and participant accrual data from 111 interventional clinical trials initiated between 2006 and 2011 by National Institutes of Health's HIV/AIDS Clinical Trials Networks. We determined the time (in days) required to complete defined phases of clinical trial protocol development and implementation. Kaplan-Meier estimates were used to assess the rates at which protocols reached specified terminal events, stratified by study purpose (therapeutic, prevention) and phase group (pilot/phase I, phase II, and phase III/IV). We also examined several potential correlates to prolonged development and implementation intervals. RESULTS: Even though phase grouping did not determine development or implementation times of either therapeutic or prevention studies, overall we observed wide variation in protocol development times. Moreover, we detected a trend toward phase III/IV therapeutic protocols exhibiting longer developmental (median 2½ years) and implementation times (>3 years). We also found that protocols exceeding the median number of days for completing the development interval had significantly longer implementation. LIMITATIONS: The use of a relatively small set of protocols may have limited our ability to detect differences across phase groupings. Some timing effects present for a specific study phase may have been masked by combining protocols into phase groupings. Presence of informative censoring, such as withdrawal of some protocols from development if they began showing signs of lost interest among investigators, complicates interpretation of Kaplan-Meier estimates. Because this study constitutes a retrospective examination over an extended period of time, it does not allow for the precise identification of relative factors impacting timing. CONCLUSION: Delays not only increase the time and cost to complete clinical trials but they also diminish their usefulness by failing to answer research questions in time. We believe that research analyzing the time spent traversing defined intervals across the clinical trial protocol development and implementation continuum can stimulate business process analyses and re-engineering efforts that could lead to reductions in the time from clinical trial concept to results, thereby accelerating progress in clinical research.

Conceptualizing and prioritizing clinical trial outcomes from the perspectives of people with Parkinson's disease versus health care professionals: a concept mapping study.

BACKGROUND: Focusing clinical investigations on outcomes that are meaningful from an end-user perspective is central in clinical research, particularly in chronic disorders such as Parkinson's disease (PD). However, little is known about how end-users such as people with PD (PwPD) and health care professionals (HCPs) view and prioritize therapeutic outcomes. PURPOSE: To compare the perspectives of PwPD and HCPs regarding prioritized areas for outcome measurement in clinical PD trials. METHODS: Concept mapping was used to identify prioritized outcomes (statements) through focus groups (n = 27; 12 PwPD, 12 HCPs, three researchers), statement sorting and importance rating (n = 38; 19 PwPD, 19 HCPs), followed by quantitative (multidimensional scaling, cluster analysis, procrustes analysis) and qualitative analysis. RESULTS: Sorting of 99 statements by PwPD and HCPs yielded 2D maps (PwPD/HCPs stress values, 0.31/0.21) with eight clusters per group. The correlation between raw sorting data of PwPD and HCPs was 0.80, and there was a significant concordance (m 12 = 0.53; P < 0.001; i.e., r = 0.68) between the spatial arrangements in their respective maps. Qualitatively, the maps from the two groups represented partially different perspectives. There were no significant differences between PwPD and HCP item importance ratings. CONCLUSION: Although similarities dominated, there were differences in how the relationships between items were perceived by the two groups, emanating from different perspectives, i.e., the clinical biomedical ("disease") versus the lived experience ("illness"). This study illustrates the clinical importance of attention to the perspective of PwPD; taking this into account is likely to provide evidence from clinical investigations that are meaningful and interpretable for end-users.

Assessing the challenges of multi-scope clinical research sites: an example from NIH HIV/AIDS clinical trials networks.

RATIONALE, AIMS, AND OBJECTIVES: Large-scale, multi-network clinical trials are seen as a means for efficient and effective utilization of resources with greater responsiveness to new discoveries. Formal structures instituted within the National Institutes of Health (NIH) HIV/AIDS Clinical Trials facilitate collaboration and coordination across networks and emphasize an integrated approach to HIV/AIDS vaccine, prevention and therapeutics clinical trials. This study examines the joint usage of clinical research sites as means of gaining efficiency, extending capacity, and adding scientific value to the networks. METHODS: A semi-structured questionnaire covering eight clinical management domains was administered to 74 (62% of sites) clinical site coordinators at single- and multi-network sites to identify challenges and efficiencies related to clinical trials management activities and coordination with multi-network units. RESULTS: Overall, respondents at multi-network sites did not report more challenges than single-network sites, but did report unique challenges to overcome including in the areas of study prioritization, community engagement, staff education and training, and policies and procedures. The majority of multi-network sites reported that such affiliations do allow for the consolidation and cost-sharing of research functions. Suggestions for increasing the efficiency or performance of multi-network sites included streamlining standards and requirements, consolidating protocol activation methods, using a single cross-network coordinating centre, and creating common budget and payment mechanisms. CONCLUSIONS: The results of this assessment provide important information to consider in the design and management of multi-network configurations for the NIH HIV/AIDS Clinical Trials Networks, as well as others contemplating and promoting the concept of multi-network settings.

Modeling the dissemination and uptake of clinical trials results.

A select set of highly cited publications from the National Institutes of Health (NIH) HIV/AIDS Clinical Trials Networks was used to illustrate the integration of time interval and citation data, modeling the progression, dissemination, and uptake of primary research findings. Following a process marker approach, the pace of initial utilization of this research was measured as the time from trial conceptualization, development and implementation, through results dissemination and uptake. Compared to earlier studies of clinical research, findings suggest that select HIV/AIDS trial results are disseminated and utilized relatively rapidly. Time-based modeling of publication results as they meet specific citation milestones enabled the observation of points at which study results were present in the literature summarizing the evidence in the field. Evaluating the pace of clinical research, results dissemination, and knowledge uptake in synthesized literature can help establish realistic expectations for the time course of clinical trials research and their relative impact toward influencing clinical practice.

Community-researcher partnerships at NIAID HIV/AIDS clinical trials sites: insights for evaluation and enhancement.

BACKGROUND: Community engagement has been a cornerstone of National Institute of Allergy and Infectious Diseases (NIAID)'s HIV/AIDS clinical trials programs since 1990. Stakeholders now consider this critical to success, hence the impetus to develop evaluation approaches. OBJECTIVES: The purpose was to assess the extent to which community advisory boards (CABs) at HIV/AIDS trials sites are being integrated into research activities. METHODS: CABs and research staff (RS) at NIAID research sites were surveyed for how each viewed (a) the frequency of activities indicative of community involvement, (b) the means for identifying, prioritizing, and supporting CAB needs, and (c) mission and operational challenges. RESULTS: Overall, CABs and RS share similar views about the frequency of community involvement activities. Cluster analysis reveals three groups of sites based on activity frequency ratings, including a group notable for CAB-RS discordance. CONCLUSIONS: Assessing differences between community and researcher perceptions about the frequency of and challenges posed by specific engagement activities may prove useful in developing evaluation tools for assessing community engagement in collaborative research settings.

Quality and rigor of the concept mapping methodology: a pooled study analysis.

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.