What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study.
BMJ Open
; 12(4): e053566, 2022 04 27.
Article
en En
| MEDLINE
| ID: mdl-35477872
ABSTRACT
OBJECTIVE:
Assess the suitability of clinical vignettes in benchmarking the performance of online symptom checkers (OSCs).DESIGN:
Observational study using a publicly available free OSC.PARTICIPANTS:
Healthily OSC, which provided consultations in English, was used to record consultation outcomes from two lay and four expert inputters using 139 standardised patient vignettes. Each vignette included three diagnostic solutions and a triage recommendation in one of three categories of triage urgency. A panel of three independent general practitioners interpreted the vignettes to arrive at an alternative set of diagnostic and triage solutions. Both sets of diagnostic and triage solutions were consolidated to arrive at a final consolidated version for benchmarking. MAIN OUTCOMEMEASURES:
Six inputters simulated 834 standardised patient evaluations using Healthily OSC and recorded outputs (triage solution, signposting, and whether the correct diagnostic solution appeared first or within the first three differentials). We estimated Cohen's kappa to assess how interpretations by different inputters could lead to divergent OSC output even when using the same vignette or when compared with a separate panel of physicians.RESULTS:
There was moderate agreement on triage recommendation (kappa=0.48), and substantial agreement on consultation outcomes between all inputters (kappa=0.73). OSC performance improved significantly from baseline when compared against the final consolidated diagnostic and triage solution (p<0.001).CONCLUSIONS:
Clinical vignettes are inherently limited in their utility to benchmark the diagnostic accuracy or triage safety of OSC. Real-world evidence studies involving real patients are recommended to benchmark the performance of OSC against a panel of physicians.Palabras clave
Texto completo:
1
Colección:
01-internacional
Banco de datos:
MEDLINE
Asunto principal:
Triaje
/
Benchmarking
Tipo de estudio:
Diagnostic_studies
/
Guideline
/
Observational_studies
Límite:
Humans
Idioma:
En
Revista:
BMJ Open
Año:
2022
Tipo del documento:
Article
País de afiliación:
Reino Unido