RESUMEN
BACKGROUND: The aim of this study was to examine the reliability of a scale to assess the methodological quality of acupuncture administered in clinical research. METHODS: We invited 36 acupuncture researchers and postgraduate students to participate in the study. Firstly, participants rated two articles using the scale. Following this initial stage, modifications were made to scale items and the exercise was repeated. Interrater reliability was assessed for individual items using the Fleiss kappa statistic, whilst the overall scale used the intraclass correlation coefficient statistic. A threshold agreement of ≥0.61 was acceptable. RESULTS: We received 26 responses and a 72% response rate. The first phase of testing found moderate reliability with intraclass correlation coefficients of 0.46 and 0.55 for the articles. The interrater reliability of the scales varied between and within the researchers (0.35, 0.60) and was more consistent with the postgraduate students (0.54, 0.54). Five items on the scale scored below the threshold and were revised for further testing. In this phase the intraclass correlation coefficient demonstrated variability between articles but improved to achieve reliability above the agreed threshold. CONCLUSION: This study provides evidence of the reliability of the NICMAN scale although improvements to a small number of items remain.
RESUMEN
BACKGROUND: Quality acupuncture influences the outcomes of clinical research, and issues associated with effective administration of acupuncture in randomized controlled trials need to be addressed when appraising studies. OBJECTIVE: The study objective was to achieve consensus on domains and items for inclusion in a rating scale to assess quality acupuncture administered in clinical research. STUDY DESIGN AND SUBJECTS: An active group of Australian acupuncture researchers initially identified a pool of items assessing quality. The Delphi consensus process was then used to select and reduce the number of items, and an additional expert panel of 42 researchers were invited to participate. Participants initially ranked items along a five-point scale for the first Delphi round, and indicated an agree or disagree response during the second round. For an item to be retained into the second round, an item had to attain greater than 80% agreement that the item described a dimension of quality acupuncture and related study design. RESULTS: Thirty-two (32) experts agreed to participate in the study. After two rounds of the Delphi process, consensus was reached on 14 domains and 26 items relating to quality acupuncture. Domains, items, and minimum standards related to study design; rationale of the intervention; criteria relating to needling stimulation either manual or electrostimulation; duration and frequency of treatment; and practitioner training. CONCLUSIONS: Items for inclusion in an instrument to assess quality acupuncture in clinical research were identified. Further development of the instrument including relative weighting of items and reliability testing is under way.