The problem with total error models in establishing performance specifications and a simple remedy.

A recent issue in this journal revisited performance specifications since the Stockholm conference. Of the three recommended methods, two use total error models to establish performance specifications. It is shown that the most commonly used total error model - the Westgard model - is deficient, yet even more complete models fail to capture all errors that comprise total error. Moreover, total error models are often set at 95% of results, which leave 5% of results as unspecified. Glucose meter performance standards are used to illustrate these problems. The Westgard model is useful to asses assay performance but not to set performance specifications. Total error can be used to set performance specifications if the specifications include 100% of the results.

Comparing Clinical Accuracy and Analytical Accuracy Between Continuous Glucose Monitors.

This study compares performance between two continuous glucose monitors (CGMs). The study design contains a mix of laboratory results (CGM vs YSI) and home results (CGM vs glucose meter). Analysis is provided for both clinical accuracy and analytical accuracy of CGM glucose measurements. Both types of accuracy are important. Error grid analysis informs about clinical accuracy. Analytical error is important as most users would prefer a CGM with a smaller spread of CGM versus reference differences. The authors provide the percentage of time that no result was obtained. Study design, data analysis, and editorial support were provided by a manufacturer of one of the products studied. This study provides a template for comparisons.

The Diabetes Technology Society Error Grid and Trend Accuracy Matrix for Glucose Monitors.

INTRODUCTION: An error grid compares measured versus reference glucose concentrations to assign clinical risk values to observed errors. Widely used error grids for blood glucose monitors (BGMs) have limited value because they do not also reflect clinical accuracy of continuous glucose monitors (CGMs). METHODS: Diabetes Technology Society (DTS) convened 89 international experts in glucose monitoring to (1) smooth the borders of the Surveillance Error Grid (SEG) zones and create a user-friendly tool-the DTS Error Grid; (2) define five risk zones of clinical point accuracy (A-E) to be identical for BGMs and CGMs; (3) determine a relationship between DTS Error Grid percent in Zone A and mean absolute relative difference (MARD) from analyzing 22 BGM and nine CGM accuracy studies; and (4) create trend risk categories (1-5) for CGM trend accuracy. RESULTS: The DTS Error Grid for point accuracy contains five risk zones (A-E) with straight-line borders that can be applied to both BGM and CGM accuracy data. In a data set combining point accuracy data from 18 BGMs, 2.6% of total data pairs equally moved from Zones A to B and vice versa (SEG compared with DTS Error Grid). For every 1% increase in percent data in Zone A, the MARD decreased by approximately 0.33%. We also created a DTS Trend Accuracy Matrix with five trend risk categories (1-5) for CGM-reported trend indicators compared with reference trends calculated from reference glucose. CONCLUSION: The DTS Error Grid combines contemporary clinician input regarding clinical point accuracy for BGMs and CGMs. The DTS Trend Accuracy Matrix assesses accuracy of CGM trend indicators.

Traditional Interference Experiments vs. Method Comparison Interference Experiments.

Two papers have appeared evaluating interferences in glucose meters. These studies are method comparisons with the added information of the medication(s) taken by the subjects. This paper contrasts a traditional interference study with the method comparison protocols. Unlike the advice in CLSI EP7, a substance that interferes should be reported even if the level of interference is clinically acceptable. The evidence of no clinically important interference in the method comparison protocol is very weak, and there is no possibility to detect statistically significant interferences. I provide an example where vitamin C at a therapeutic level was within clinical error limits, but when the concentration was at levels used to treat cancer, there was bias well above clinically acceptable limits.

Adverse Event Causes From 2022 for Four Continuous Glucose Monitors.

BACKGROUND: Adverse events for continuous glucose monitors (CGMs) represent a significant issue for people with diabetes with 281 963 CGM adverse events occurring in 2022. The process to obtain adverse events and the US Food and Drug Administration (FDA) database that contains them are reviewed. METHODS: Tables were created in SQL Server for four CGM products (Dexcom G6, all versions of Abbott Libre, Medtronic Guardian 3, and Senseonics Eversense) containing either malfunction or injury adverse events sorted by the manufacturer's chosen product code. As the product code is not always clear (or appropriate), the causes of the events were determined from the text description of the adverse event. The resulting causes were listed in decreasing order in tables for each product and event type. RESULTS: A common effect of several event causes prevented the user from obtaining a result. Inaccuracy was also a frequent complaint. Other causes were specific to that device. CONCLUSIONS: Creating tables based on manufacturer problem codes for their CGMs, followed by analysis of the adverse event text, facilitates the analysis of event causes. Analyzing adverse event data is the first step in trying to reduce the number of adverse events.

Retained Diabetes Devices-A Literature Review.

BACKGROUND: Diabetes management and treatment requires the use of many devices that frequently must puncture the skin, creating a risk of unintentional retention in the body as a retained diabetes device. In this article, we reviewed case studies about retained diabetes devices and presented analyses of the success rate of current imaging techniques in identifying retained devices and the success rate of device removal. METHODS: PubMed and Google Scholar were searched for articles about retained diabetes devices. Relevant articles that included sufficient details about discovery and removal of the device were included. The success rate of identification and the success rate of removal of retained devices were both calculated as percentages. RESULTS: Sixteen case studies of retained diabetes devices were identified. These devices included parts of continuous glucose monitors and infusion sets, a lancet, and various types of needles for insulin injection. Each case is presented with details about the year of publication, the retained diabetes device, the company that produced the device, the age and gender of the patient, the type of diabetes that the patient had, the location of the device, the reason for initial discovery of the retained device, the process of locating the device, the success rate for removal of the device, and the removal procedure of the device. Analysis revealed a 100% success rate for the use of imaging technology including X-rays and computed tomography to identify a retained diabetes device. The patients with retained diabetes devices had a 62.5% success rate for eventual removal of the device. CONCLUSIONS: With the increasing use of injected, inserted, and implanted diabetes wearables for digital health, it is likely that some of the devices will detach, break apart, or otherwise become retained in the body. It is important to be aware of available technologies to identify retained diabetes devices so that it will be possible in most cases to surgically remove these devices if they detach or become retained.

An Analysis of 2019 FDA Adverse Events for Two Insulin Pumps and Two Continuous Glucose Monitors.

US Food and Drug Administration adverse event data for 2019 were analyzed for two insulin pumps and two continuous glucose monitors (CGMs). The analyses were selective-they were guided by the text described in the adverse events. They included (1) percent using auto mode for the Medtronic 670G pump, (2) distributions of hyper and hypo glucose values for Medtronic and Tandem pumps, (3) a Parkes error grid for Dexcom CGM vs glucose meter when the complaint was inaccuracy, and (4) the most frequent events for Abbott Freestyle. We found that for the 670G pump, there were more hypo events when auto mode was on than when auto mode was off. With Dexcom CGMs, users complained about inaccurate result when most results were in the B zone. With the Abbott Freestyle, the most frequent adverse event was an allergic skin reaction.

More Focus is Needed to Reduce Adverse Events for Diabetes Devices.

Advances in devices for people with diabetes have demonstrated many improvements; yet, the number of adverse events has almost doubled from 2018 to 2019. It is a challenge to examine these events due to a difficult query tool on the FDA website. There are several possible reasons why effort is not devoted to decreasing the number of adverse events including the fact that user error is a common cause. This commentary serves to raise awareness of the adverse event problem.

Adverse Event Data for Years 2018 to 2020 for Diabetes Devices.

Unlike performance evaluations, which are often conducted under ideal conditions, adverse events occur during actual device use for people with diabetes. This report summarizes the number of adverse events for the years 2018 to 2020 for the 3 diabetes devices: blood glucose meters (BG), continuous glucose monitors (CGM), and insulin pumps. A text example of a CGM injury is provided. Possible reasons are suggested for trends. Whereas the rate per test result (events/usage) is exceedingly small, the rate per patient (events/people with diabetes that use insulin) is of concern. Hence, it is important to determine event causes and provide corrective actions. The first step is to put in place routine analysis of adverse event data for diabetes devices.

Towards more complete specifications for acceptable analytical performance - a plea for error grid analysis.

Abstract We examine limitations of common analytical performance specifications for quantitative assays. Specifications can be either clinical or regulatory. Problems with current specifications include specifying limits for only 95% of the results, having only one set of limits that demarcate no harm from minor harm, using incomplete models for total error, not accounting for the potential of user error, and not supplying sufficient protocol requirements. Error grids are recommended to address these problems as error grids account for 100% of the data and stratify errors into different severity categories. Total error estimation from a method comparison can be used to estimate the inner region of an error grid, but the outer region needs to be addressed using risk management techniques. The risk management steps, foreign to many in laboratory medicine, are outlined.

Analysis of a Point-of-Care HbA1c Assay: Is It Time for an HbA1c Error Grid?

In an article in Journal of Diabetes Science and Technology, Arnold et al have presented a thorough study of imprecision components for a point-of-care hemoglobin A1c (HbA1c) assay. An interesting and innovative approach is the combination of data from different studies to arrive at a total error estimate. But total error has the oxymoron feature of estimating performance for most (95%) but not all of the results. An HbA1c error grid would provide the severity for results that exceed the 6% requirement. Since this device is intended for Clinical Laboratory Improvement Amendments waived labs and allows for finger-stick samples, monitoring the Food and Drug Administration adverse event database (MAUDE, Manufacturer and User Facility Device Experience) is recommended.

Reducing Glucose Meter Adverse Events by Using Reliability Growth With the FDA MAUDE Database.

Glucose meter evaluations are common and provide important information about glucose meter performance versus standards. Although some meters meet guidelines and others fall short in these evaluations, most results are within the A and B zones of a glucose meter error grid. Another data source that is seldom used is the FDA adverse event database (MAUDE). This database describes glucose meter malfunctions and injury as reported by actual users and returned 10 837 adverse events across all meters for the first 7 months of 2018. Reliability growth management is an established tool to reduce failure rates. A reliability growth example is presented followed by a discussion of how this tool could be applied to reduce glucose meter failure events using the MAUDE database.

Why the Details of Glucose Meter Evaluations Matters.

In an article in the Journal of Diabetes Science and Technology, Macleod and coworkers describe an evaluation of LifeScan glucose meters that focus on the effects of sample types and comparison methods. They make a valid point that these factors influence the accuracy observed in evaluations and recommend the comparison method be the one recommended by the manufacturer for the sample type in the intended use statement. Yet, the recommended comparison method is not a reference method. The accuracy hierarchy of definitive, reference, and field methods originally described by Tietz should remind one that virtually all glucose meter evaluations use commercially available field methods as the comparison method. Finally, one should not neglect the FDA adverse event database as a way to assess glucose meter performance.

Getting More Information From Glucose Meter Evaluations.

Glucose meter evaluations are common in publications and inform whether the meter meets the ISO 15197 specification. The ISO 15197 specifications, which are universally cited, leave 1% of results unspecified, which can be thought of as typical performance of results (99%) versus rare performance (1%). Suggestions are provided to extract more information from these evaluations, including rare performance, since highly discrepant results or failure to obtain a result can be observed in a glucose meter that has met the ISO 15197 specification. It is also recommended that when manufacturers perform evaluations, they analyze adverse events contained in the FDA MAUDE database. Finally, we point out an important problem with the ISO 15197 specifications.

Why the Diabetes Technology Society Surveillance Protocol for Glucose Meters Needs to Be Revised.

The Diabetes Technology Society surveillance protocol provides a seal of approval for a glucose meter if a sufficient number of a candidate glucose meter's results meet ISO 15197:2013 limits. The protocol provides clear details about how to conduct this study and analyze the data but has two flaws. There is no specification about the size of glucose meter errors that are outside of ISO limits. A meter that has a result in the E zone of a glucose meter error grid could receive the DTS seal of approval. In addition, the protocol uses the ISO standard, which could be considered a "state of the art" standard instead of an error grid, which is a clinical standard. Remedies for these problems are to replace the ISO standard with an error grid and to include requirements for errors found in C or higher zones of an error grid.