Best practice considerations on the assessment of robotic assisted surgical systems: results from an international consensus expert panel.

BACKGROUND: Health technology assessments (HTAs) of robotic assisted surgery (RAS) face several challenges in assessing the value of robotic surgical platforms. As a result of using different assessment methods, previous HTAs have reached different conclusions when evaluating RAS. While the number of available systems and surgical procedures is rapidly growing, existing frameworks for assessing MedTech provide a starting point, but specific considerations are needed for HTAs of RAS to ensure consistent results. This work aimed to discuss different approaches and produce guidance on evaluating RAS. METHODS: A consensus conference research methodology was adopted. A panel of 14 experts was assembled with international experience and representing relevant stakeholders: clinicians, health economists, HTA practitioners, policy makers, and industry. A review of previous HTAs was performed and seven key themes were extracted from the literature for consideration. Over five meetings, the panel discussed the key themes and formulated consensus statements. RESULTS: A total of ninety-eight previous HTAs were identified from twenty-five total countries. The seven key themes were evidence inclusion and exclusion, patient- and clinician-reported outcomes, the learning curve, allocation of costs, appropriate time horizons, economic analysis methods, and robotic ecosystem/wider benefits. CONCLUSIONS: Robotic surgical platforms are tools, not therapies. Their value varies according to context and should be considered across therapeutic areas and stakeholders. The principles set out in this paper should help HTA bodies at all levels to evaluate RAS. This work may serve as a case study for rapidly developing areas in MedTech that require particular consideration for HTAs.

An international consensus panel on the potential value of Digital Surgery.

OBJECTIVES: The use of digital technology in surgery is increasing rapidly, with a wide array of new applications from presurgical planning to postsurgical performance assessment. Understanding the clinical and economic value of these technologies is vital for making appropriate health policy and purchasing decisions. We explore the potential value of digital technologies in surgery and produce expert consensus on how to assess this value. DESIGN: A modified Delphi and consensus conference approach was adopted. Delphi rounds were used to generate priority topics and consensus statements for discussion. SETTING AND PARTICIPANTS: An international panel of 14 experts was assembled, representing relevant stakeholder groups: clinicians, health economists, health technology assessment experts, policy-makers and industry. PRIMARY AND SECONDARY OUTCOME MEASURES: A scoping questionnaire was used to generate research questions to be answered. A second questionnaire was used to rate the importance of these research questions. A final questionnaire was used to generate statements for discussion during three consensus conferences. After discussion, the panel voted on their level of agreement from 1 to 9; where 1=strongly disagree and 9=strongly agree. Consensus was defined as a mean level of agreement of >7. RESULTS: Four priority topics were identified: (1) how data are used in digital surgery, (2) the existing evidence base for digital surgical technologies, (3) how digital technologies may assist surgical training and education and (4) methods for the assessment of these technologies. Seven consensus statements were generated and refined, with the final level of consensus ranging from 7.1 to 8.6. CONCLUSION: Potential benefits of digital technologies in surgery include reducing unwarranted variation in surgical practice, increasing access to surgery and reducing health inequalities. Assessments to consider the value of the entire surgical ecosystem holistically are critical, especially as many digital technologies are likely to interact simultaneously in the operating theatre.

A novel immunogen to modulate cytokine production and promote immune system reconstitution in HIV-AIDS.

This 'proof of concept' study was implemented in anticipation of identifying and testing a novel antigen of human origin as a potential immunogen in a paradigm that emphasizes immunomodulation and immune system reconstitution as requisites to the development of an effective human immunodeficiency virus (HIV)-acquired immune deficiency syndrome vaccine. Fifteen HIV-infected, highly active antiretroviral therapy (HAART) naive, otherwise healthy male seropositive patients were stratified by [CD4+] into 3 groups of 5 patients: group 1 >500/mm; group 2 > 250/mm but <500/mm; and group 3 < 250/mm. Five healthy male subjects were used as controls. Replicate peripheral blood mononuclear cell (PBMC) [H]thymidine uptake phytohemaglutinin-stimulated proliferation studies, and serum cytokine assays were carried out in the presence or absence of Kveim antigen at dilutions ranging from 0.001 to 100 µg/mL. Serum cytokines [interleukin-2 (IL-2), IL-4, IL-6, interferon gamma, and tumor necrosis factor alpha] were assayed using standardized methodology. Nonparametric statistical analyses and linear regression analysis were used to test for statistical significance and strength of associations. PBMCs harvested from HIV-infected patients and incubated, ex vivo, demonstrated reproducible, antigen concentration-dependent changes in cytokine production over a range of antigen concentrations (0.001-100 µg/mL) in contrast to antigen-naive PBMCs and controls. Significant correlations were demonstrated between antigen concentration and the amount of cytokines secreted. The magnitude of the cytokine response and the patterns of cytokine secretion were HIV group-specific and could be used to identify and distinguish between the 3 groups of HIV-infected subjects. A shift toward the production of type 1-like (Th1) cytokines characteristically seen in systemic sarcoidosis and associated with effective HAART was seen when patterns of cytokine secretion were compared between antigen exposed and antigen-naive PBMCs. PBMCs harvested from seropositive HIV-infected patients and exposed to the Kveim antigen have the following properties: (1) They demonstrate proliferation and exhibit an antigen concentration-dependent secretion of cytokines. The magnitude of the cytokine response can be used to identify and distinguish between groups of seropositive patients stratified by [CD4+]. (2) These PBMCs secrete cytokines in patterns suggestive of a shift to a type 1-like (Th1) response characteristic of HAART and sarcoidosis as opposed to the type 2-like (Th2) cytokine profile characteristic of HIV-acquired immune deficiency syndrome.

A comprehensive approach to the interpretation of difference scores.

A comprehensive approach to the interpretation of difference scores is presented. Formulas for the test of statistical significance between two test scores, computed by a confidence interval, and for the calculation of the probabilities for the power of the statistical test, underinterpretation, overinterpretation, and misinterpretation are provided. Definitions and examples of their use in score interpretation are provided.

Differences scores: regression-based reliable difference and the regression-based confidence interval.

Over 50 years ago Payne and Jones (1957) developed what has been labeled the traditional reliable difference formula that continues to be useful as a significance test for the difference between two test scores. The traditional reliable difference is based on the standard error of measurement (SEM) and has been updated to a confidence interval approach. As an alternative to the traditional reliable difference, this article presents the regression-based reliable difference that is based on the standard error of estimate (SEE) and estimated true scores. This new approach should be attractive to clinicians preferring the idea of scores regressing toward the mean. The new approach is also presented in confidence interval form with an interpretation that can be viewed as a statement of all hypotheses that are tenable and consistent with the observed data and has the backing of several authorities. Two well-known conceptualizations for true score confidence intervals are the traditional and regression-based. Now clinicians favoring the regression-based conceptualization are not restricted to the use of traditional model when testing score differences using confidence intervals.

Statistical extensions of Ponterotto and Ruckdeschel's (2007) reliability matrix for estimating the adequacy of internal consistency coefficients.

The present article extends work on Ponterotto and Ruckdeschel's Reliability Matrix for estimating the adequacy of internal consistency measures. Specifically, it uses statistical tests to determine whether a calculated coefficient alpha is equal to or greater than the hypothesized population coefficient alpha identified in the Reliability Matrix. The Feldt, Woodruff, and Salih (1987) confidence interval test and Bonett's (2002) approximate z-test and N formula are applied. Select guidelines on reliability assessment and conducting statistical tests to determine adequacy of coefficient alpha are presented.

Statistical approaches to achieving sufficiently high test score reliabilities for research purposes.

The author provides statistical approaches to aid investigators in assuring that sufficiently high test score reliabilities are achieved for specific research purposes. The statistical approaches use tests of statistical significance between the obtained reliability and lowest population reliability that an investigator will tolerate. The statistical approaches work for coefficient alpha and related coefficients and for alternate-forms, split-half (2-part alpha), and retest reliabilities. The author shows that, in some instances, a formula can help to estimate the sample size necessary for the statistical test.

Statistical significance and reliability of single-subtest and cluster deviation scores.

Suppose one has a battery of K subtests and a composite for the battery is defined as the mean of the K standardized subtest scores. An individual's single-subtest deviation score is the difference between the individual's score on any single subtest and his composite score. A cluster deviation score is the difference between an examinee's average for a small set (cluster) of subtests and his composite. Formulas are given for the test of statistical significance of the individual's subtest or cluster deviation score and the internal consistency reliability of such deviation scores.

A practical use for the KR-21 reliability coefficient.

KR-21 provides a lower limit for the computed value of KR-20. KR-20 is equivalent to coefficient alpha when a test is composed of dichotomous items scored 0 or 1. Therefore, KR-21 coefficients, computed from simple summary statistics, can be used in cases in which journal authors do not provide the test score reliability. Use of KR-21 in these cases will provide the reader with a lower limit for the value of KR-20.

Score reliability for tests constructed for African-American populations.

Test score reliabilities and sample sizes (N) used to establish the reliabilities are described for a variety of tests constructed for African-American populations. The sample size was 341. The average internal consistency reliability was .74 (SD = .16) with a median value of .77. The median N was 131. The mean internal consistency reliability and median N for tests intended for assessment of individuals were only .72 and 96, respectively.

Psychometric properties of the Folstein Mini-Mental State Examination.

Criterion-referenced (Livingston) and norm-referenced (Gilmer-Feldt) techniques were used to measure the internal consistency reliability of Folstein's Mini-Mental State Examination (MMSE) on a large sample (N = 418) of elderly medical patients. Two administration and scoring variants of the MMSE Attention and Calculation section (Serial 7s only and WORLD only) were investigated. Livingston reliability coefficients (rs) were calculated for a wide range of cutoff scores. As necessary for the calculation of the Gilmer-Feldt r, a factor analysis showed that the MMSE measures three cognitive domains. Livingston's r for the most widely used MMSE cutoff score of 24 was .803 for Serial 7s and .795 for WORLD. The Gilmer-Feldt internal consistency reliability coefficient was .764 for Serial 7s and .747 for WORLD. Item analysis showed that nearly all of the MMSE items were good discriminators, but 12 were too easy. True score confidence intervals should be applied when interpreting MMSE test scores.

Validity and reliability of the Miller Forensic Assessment of Symptoms Test (M-FAST): comment on Veazey, et al. (2005).

Confidence intervals are provided for the validity coefficients calculated by Veazey, et al. for the M-FAST. Two coefficients alpha are also presented along with suggestions for different approaches to calculating the M-FAST internal consistency reliability.

Estimating the reliability of a test split into two parts of equal or unequal length.

When the reliability of test scores must be estimated by an internal consistency method, partition of the test into just 2 parts may be the only way to maintain content equivalence of the parts. If the parts are classically parallel, the Spearman-Brown formula may be validly used to estimate the reliability of total scores. If the parts differ in their standard deviations but are tau equivalent, Cronbach's alpha is appropriate. However, if the 2 parts are congeneric, that is, they are unequal in functional length or they comprise heterogeneous item types, a less well-known estimate, the Angoff-Feldt coefficient, is appropriate. Guidelines in terms of the ratio of standard deviations are proposed for choosing among Spearman-Brown, alpha, and Angoff-Feldt coefficients.

Study samples are too small to produce sufficiently precise reliability coefficients.

In a survey of journal articles, test manuals, and test critique books, the author found that a mean sample size (N) of 260 participants had been used for reliability studies on 742 tests. The distribution was skewed because the median sample size for the total sample was only 90. The median sample sizes for the internal consistency, retest, and interjudge reliabilities were 182, 64, and 36, respectively. The author presented sample size statistics for the various internal consistency methods and types of tests. In general, the author found that the sample sizes that were used in the internal consistency studies were too small to produce sufficiently precise reliability coefficients, which in turn could cause imprecise estimates of examinee true-score confidence intervals. The results also suggest that larger sample sizes have been used in the last decade compared with those that were used in earlier decades.

A breakdown of reliability coefficients by test type and reliability method, and the clinical implications of low reliability.

The author presented descriptive statistics for 937 reliability coefficients for various reliability methods (e.g., alpha) and test types (e.g., intelligence). He compared the average reliability coefficients with the reliability standards that are suggested by experts and found that most average reliabilities were less than ideal. Correlations showed that over the past several decades there has been neither a rise nor a decline in the value of internal consistency, retest, or interjudge reliability coefficients. Of the internal consistency approaches, there has been an increase in the use of coefficient alpha, whereas use of the split-half method has decreased over time. Decision analysis and true-score confidence intervals showed how low reliability can result in clinical decision errors.

Reliability of the WMS-III discrepancy comparisons.

Internal consistency reliabiities for the WMS-III Primary Indexes, Primary Index subtest, and Ability-Memory discrepancy scores are provided. The reliabilities ranged from .00 to .89.

Note on Kazlauskaite and Lynn's paper on retest reliability of the Colored Progressive Matrices.

A reanalysis of the retest reliabilities for the Colored Progressive Matrices indicates Kazlauskaite and Lynn's conclusions (2002) were not accurate.

Combining reliability coefficients: possible application to meta-analysis and reliability generalization.

Formulae for combining reliability coefficients from any number of samples are provided. These formulae produce the exact reliability one would compute if one had the raw data from the samples. Needed are the sample means, standard deviations, sample sizes, and reliability coefficients. The formulae work for coefficient alpha, KR-20, retest, alternate-forms, split-half, interrater (intraclass), Gilmer-Feldt, Angoff-Feldt, validity, and other coefficients. They may be particularly useful for meta-analytic and reliability generalization studies.

Note on Knight's analysis of the WAIS-III instruction effect on the Matrix Reasoning subtest.

Knight's 2003 analysis of the effect of the WAIS-III instructions on the Matrix Reasoning subtest was based on multiple t tests, which is a violation of conventional statistical procedures. Using this procedure significant differences were found between the group who know the subtest was untimed versus the group which did not know if the subtest was timed or untimed. Reanalysis of the data used three statistical alternatives: (a) Bonferroni correction for all possible t tests, (b) one-way analysis of variance, and (c) selected t tests with the Bonferroni correction. All three analyses yielded nonsignificant differences between means, thereby changing the conclusions of Knight's study.

A simple computational alternative to analysis of variance formulas for estimating the k-judge reliability.

Formulas requiring the computation of only three standard deviations are presented for computing the interjudge reliability coefficient for any number of judges. These formulas yield coefficients identical to those obtained from a one-way repeated-measures analysis of variance. Even researchers with small handheld calculators can use this simple approach.