Reliability and Validity of Proposed Risk Stratification Methods for Laboratory Developed Tests.

OBJECTIVES: To determine whether different laboratory developed test (LDT) risk stratification proposals would assign differing levels of risk to selected LDTs as a measure of the validity of those proposals, and whether there would be differing interrater agreement rates as a measure of the reliability of those proposals. METHODS: A total of 4 reviewers applied 6 proposals for risk stratification of 4 LDTs. Interrater agreement was calculated as a measure of the reliability of the proposals. Also, a consensus risk categorization and concordance rate for each LDT was developed as a measure of the validity of the proposals. RESULTS: Interrater agreement rates (reliability) ranged from 38% to 100%, and concordance rates (validity) ranged from 20% to 100%. CONCLUSIONS: A spectrum of reliability and validity was observed depending on the policy used and the LDT categorized. Before implementation or legislation of risk-stratification methods, large evaluations of reliability and validity should be conducted on any proposed method.