Trends and opportunities in computable clinical phenotyping: A scoping review.

Identifying patient cohorts meeting the criteria of specific phenotypes is essential in biomedicine and particularly timely in precision medicine. Many research groups deliver pipelines that automatically retrieve and analyze data elements from one or more sources to automate this task and deliver high-performing computable phenotypes. We applied a systematic approach based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to conduct a thorough scoping review on computable clinical phenotyping. Five databases were searched using a query that combined the concepts of automation, clinical context, and phenotyping. Subsequently, four reviewers screened 7960 records (after removing over 4000 duplicates) and selected 139 that satisfied the inclusion criteria. This dataset was analyzed to extract information on target use cases, data-related topics, phenotyping methodologies, evaluation strategies, and portability of developed solutions. Most studies supported patient cohort selection without discussing the application to specific use cases, such as precision medicine. Electronic Health Records were the primary source in 87.1 % (N = 121) of all studies, and International Classification of Diseases codes were heavily used in 55.4 % (N = 77) of all studies, however, only 25.9 % (N = 36) of the records described compliance with a common data model. In terms of the presented methods, traditional Machine Learning (ML) was the dominant method, often combined with natural language processing and other approaches, while external validation and portability of computable phenotypes were pursued in many cases. These findings revealed that defining target use cases precisely, moving away from sole ML strategies, and evaluating the proposed solutions in the real setting are essential opportunities for future work. There is also momentum and an emerging need for computable phenotyping to support clinical and epidemiological research and precision medicine.

Visual storytelling enhances knowledge dissemination in biomedical science.

Research findings in biomedical science are often summarized in statistical plots and sophisticated data presentations. Such visualizations are challenging for people who lack the appropriate scientific background or even experts who work in other areas. Scientists have to maximize knowledge dissemination by improving the communication of their findings to the public. To address the need for compelling and successful information visualizations in biomedical science, we propose a new theoretical framework for Visual Storytelling and illustrate its potential application through two visual stories, one on vaccine safety and one on cancer immunotherapy. In both examples, we rely on solid data and combine multiple media (photographs, illustrations, choropleth maps, tables, graphs, and charts) with text to create powerful visual stories for the selected target audiences. If fully validated, the proposed theory may shed light into non-traditional techniques for building visual stories and further the agenda of creating compelling information visualizations.

Data-Driven Blood Glucose Pattern Classification and Anomalies Detection: Machine-Learning Applications in Type 1 Diabetes.

BACKGROUND: Diabetes mellitus is a chronic metabolic disorder that results in abnormal blood glucose (BG) regulations. The BG level is preferably maintained close to normality through self-management practices, which involves actively tracking BG levels and taking proper actions including adjusting diet and insulin medications. BG anomalies could be defined as any undesirable reading because of either a precisely known reason (normal cause variation) or an unknown reason (special cause variation) to the patient. Recently, machine-learning applications have been widely introduced within diabetes research in general and BG anomaly detection in particular. However, irrespective of their expanding and increasing popularity, there is a lack of up-to-date reviews that materialize the current trends in modeling options and strategies for BG anomaly classification and detection in people with diabetes. OBJECTIVE: This review aimed to identify, assess, and analyze the state-of-the-art machine-learning strategies and their hybrid systems focusing on BG anomaly classification and detection including glycemic variability (GV), hyperglycemia, and hypoglycemia in type 1 diabetes within the context of personalized decision support systems and BG alarm events applications, which are important constituents for optimal diabetes self-management. METHODS: A rigorous literature search was conducted between September 1 and October 1, 2017, and October 15 and November 5, 2018, through various Web-based databases. Peer-reviewed journals and articles were considered. Information from the selected literature was extracted based on predefined categories, which were based on previous research and further elaborated through brainstorming. RESULTS: The initial results were vetted using the title, abstract, and keywords and retrieved 496 papers. After a thorough assessment and screening, 47 articles remained, which were critically analyzed. The interrater agreement was measured using a Cohen kappa test, and disagreements were resolved through discussion. The state-of-the-art classes of machine learning have been developed and tested up to the task and achieved promising performance including artificial neural network, support vector machine, decision tree, genetic algorithm, Gaussian process regression, Bayesian neural network, deep belief network, and others. CONCLUSIONS: Despite the complexity of BG dynamics, there are many attempts to capture hypoglycemia and hyperglycemia incidences and the extent of an individual's GV using different approaches. Recently, the advancement of diabetes technologies and continuous accumulation of self-collected health data have paved the way for popularity of machine learning in these tasks. According to the review, most of the identified studies used a theoretical threshold, which suffers from inter- and intrapatient variation. Therefore, future studies should consider the difference among patients and also track its temporal change over time. Moreover, studies should also give more emphasis on the types of inputs used and their associated time lag. Generally, we foresee that these developments might encourage researchers to further develop and test these systems on a large-scale basis.

Evaluation of Natural Language Processing (NLP) systems to annotate drug product labeling with MedDRA terminology.

INTRODUCTION: The FDA Adverse Event Reporting System (FAERS) is a primary data source for identifying unlabeled adverse events (AEs) in a drug or biologic drug product's postmarketing phase. Many AE reports must be reviewed by drug safety experts to identify unlabeled AEs, even if the reported AEs are previously identified, labeled AEs. Integrating the labeling status of drug product AEs into FAERS could increase report triage and review efficiency. Medical Dictionary for Regulatory Activities (MedDRA) is the standard for coding AE terms in FAERS cases. However, drug manufacturers are not required to use MedDRA to describe AEs in product labels. We hypothesized that natural language processing (NLP) tools could assist in automating the extraction and MedDRA mapping of AE terms in drug product labels. MATERIALS AND METHODS: We evaluated the performance of three NLP systems, (ETHER, I2E, MetaMap) for their ability to extract AE terms from drug labels and translate the terms to MedDRA Preferred Terms (PTs). Pharmacovigilance-based annotation guidelines for extracting AE terms from drug labels were developed for this study. We compared each system's output to MedDRA PT AE lists, manually mapped by FDA pharmacovigilance experts using the guidelines, for ten drug product labels known as the "gold standard AE list" (GSL) dataset. Strict time and configuration conditions were imposed in order to test each system's capabilities under conditions of no human intervention and minimal system configuration. Each NLP system's output was evaluated for precision, recall and F measure in comparison to the GSL. A qualitative error analysis (QEA) was conducted to categorize a random sample of each NLP system's false positive and false negative errors. RESULTS: A total of 417, 278, and 250 false positive errors occurred in the ETHER, I2E, and MetaMap outputs, respectively. A total of 100, 80, and 187 false negative errors occurred in ETHER, I2E, and MetaMap outputs, respectively. Precision ranged from 64% to 77%, recall from 64% to 83% and F measure from 67% to 79%. I2E had the highest precision (77%), recall (83%) and F measure (79%). ETHER had the lowest precision (64%). MetaMap had the lowest recall (64%). The QEA found that the most prevalent false positive errors were context errors such as "Context error/General term", "Context error/Instructions or monitoring parameters", "Context error/Medical history preexisting condition underlying condition risk factor or contraindication", and "Context error/AE manifestations or secondary complication". The most prevalent false negative errors were in the "Incomplete or missed extraction" error category. Missing AE terms were typically due to long terms, or terms containing non-contiguous words which do not correspond exactly to MedDRA synonyms. MedDRA mapping errors were a minority of errors for ETHER and I2E but were the most prevalent false positive errors for MetaMap. CONCLUSIONS: The results demonstrate that it may be feasible to use NLP tools to extract and map AE terms to MedDRA PTs. However, the NLP tools we tested would need to be modified or reconfigured to lower the error rates to support their use in a regulatory setting. Tools specific for extracting AE terms from drug labels and mapping the terms to MedDRA PTs may need to be developed to support pharmacovigilance. Conducting research using additional NLP systems on a larger, diverse GSL would also be informative.

Evaluating automated approaches to anaphylaxis case classification using unstructured data from the FDA Sentinel System.

INTRODUCTION: In May 2008, the Food and Drug Administration launched the Sentinel Initiative, a multi-year program for the establishment of a national electronic monitoring system for medical product safety that led, in 2016, to the launch of the full Sentinel System. Under the Mini-Sentinel pilot, several algorithms for identifying health outcomes of interest, including one for anaphylaxis, were developed and evaluated using data available from the Sentinel common data model. PURPOSE: To evaluate whether features extracted from unstructured narrative data using natural language processing (NLP) could be used to classify anaphylaxis cases. METHODS: Using previously developed methods, we extracted features from unstructured narrative data using NLP and applied rule-based and similarity-based algorithms to identify anaphylaxis among 62 potential cases previously classified by human experts as anaphylaxis (N = 33), not anaphylaxis (N = 27), and unknown (N = 2). RESULTS: The rule-based and similarity-based approaches demonstrated almost equal performance (recall 100% vs 100%, precision 60.3% vs 57.4%, F-measure: 0.753 vs 0.729). Reasons for misclassification included the inability of the algorithms to make the same clinical judgments as human experts about the timing, severity, or presence of alternative explanations; and the identification of terms consistent with anaphylaxis but present in conditions other than anaphylaxis. CONCLUSIONS: Although precision needs to be improved before these algorithms could be used without human review, we demonstrated that applying rule-based and similarity-based algorithms to unstructured narrative information from clinical records can be used for classification of anaphylaxis in the Sentinel System. Further development and assessment of these methods in the Sentinel System are warranted.

Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic review.

We followed a systematic approach based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses to identify existing clinical natural language processing (NLP) systems that generate structured information from unstructured free text. Seven literature databases were searched with a query combining the concepts of natural language processing and structured data capture. Two reviewers screened all records for relevance during two screening phases, and information about clinical NLP systems was collected from the final set of papers. A total of 7149 records (after removing duplicates) were retrieved and screened, and 86 were determined to fit the review criteria. These papers contained information about 71 different clinical NLP systems, which were then analyzed. The NLP systems address a wide variety of important clinical and research tasks. Certain tasks are well addressed by the existing systems, while others remain as open challenges that only a small number of systems attempt, such as extraction of temporal information or normalization of concepts to standard terminologies. This review has identified many NLP systems capable of processing clinical free text and generating structured output, and the information collected and evaluated here will be important for prioritizing development of new approaches for clinical NLP.

A new algorithmic approach for the extraction of temporal associations from clinical narratives with an application to medical product safety surveillance reports.

The sheer volume of textual information that needs to be reviewed and analyzed in many clinical settings requires the automated retrieval of key clinical and temporal information. The existing natural language processing systems are often challenged by the low quality of clinical texts and do not demonstrate the required performance. In this study, we focus on medical product safety report narratives and investigate the association of the clinical events with appropriate time information. We developed a novel algorithm for tagging and extracting temporal information from the narratives, and associating it with related events. The proposed algorithm minimizes the performance dependency on text quality by relying only on shallow syntactic information and primitive properties of the extracted event and time entities. We demonstrated the effectiveness of the proposed algorithm by evaluating its tagging and time assignment capabilities on 140 randomly selected reports from the US Vaccine Adverse Event Reporting System (VAERS) and the FDA (Food and Drug Administration) Adverse Event Reporting System (FAERS). We compared the performance of our tagger with the SUTime and HeidelTime taggers, and our algorithm's event-time associations with the Temporal Awareness and Reasoning Systems for Question Interpretation (TARSQI). We further evaluated the ability of our algorithm to correctly identify the time information for the events in the 2012 Informatics for Integrating Biology and the Bedside (i2b2) Challenge corpus. For the time tagging task, our algorithm performed better than the SUTime and the HeidelTime taggers (F-measure in VAERS and FAERS: Our algorithm: 0.86 and 0.88, SUTime: 0.77 and 0.74, and HeidelTime 0.75 and 0.42, respectively). In the event-time association task, our algorithm assigned an inappropriate timestamp for 25% of the events, while the TARSQI toolkit demonstrated a considerably lower performance, assigning inappropriate timestamps in 61.5% of the same events. Our algorithm also supported the correct calculation of 69% of the event relations to the section time in the i2b2 testing set.

Decision support environment for medical product safety surveillance.

We have developed a Decision Support Environment (DSE) for medical experts at the US Food and Drug Administration (FDA). The DSE contains two integrated systems: The Event-based Text-mining of Health Electronic Records (ETHER) and the Pattern-based and Advanced Network Analyzer for Clinical Evaluation and Assessment (PANACEA). These systems assist medical experts in reviewing reports submitted to the Vaccine Adverse Event Reporting System (VAERS) and the FDA Adverse Event Reporting System (FAERS). In this manuscript, we describe the DSE architecture and key functionalities, and examine its potential contributions to the signal management process by focusing on four use cases: the identification of missing cases from a case series, the identification of duplicate case reports, retrieving cases for a case series analysis, and community detection for signal identification and characterization.

Seeing is believing: good graphic design principles for medical research.

Have you noticed when you browse a book, journal, study report, or product label how your eye is drawn to figures more than to words and tables? Statistical graphs are powerful ways to transparently and succinctly communicate the key points of medical research. Furthermore, the graphic design itself adds to the clarity of the messages in the data. The goal of this paper is to provide a mechanism for selecting the appropriate graph to thoughtfully construct quality deliverables using good graphic design principles. Examples are motivated by the efforts of a Safety Graphics Working Group that consisted of scientists from the pharmaceutical industry, Food and Drug Administration, and academic institutions.

Improving drug safety with adverse event detection using natural language processing.

INTRODUCTION: Pharmacovigilance (PV) involves monitoring and aggregating adverse event information from a variety of data sources, including health records, biomedical literature, spontaneous adverse event reports, product labels, and patient-generated content like social media posts, but the most pertinent details in these sources are typically available in narrative free-text formats. Natural language processing (NLP) techniques can be used to extract clinically relevant information from PV texts to inform decision-making. AREAS COVERED: We conducted a non-systematic literature review by querying the PubMed database to examine the uses of NLP in drug safety and distilled the findings to present our expert opinion on the topic. EXPERT OPINION: New NLP techniques and approaches continue to be applied for drug safety use cases; however, systems that are fully deployed and in use in a clinical environment remain vanishingly rare. To see high-performing NLP techniques implemented in the real setting will require long-term engagement with end users and other stakeholders and revised workflows in fully formulated business plans for the targeted use cases. Additionally, we found little to no evidence of extracted information placed into standardized data models, which should be a way to make implementations more portable and adaptable.

Precision Oncology Core Data Model to Support Clinical Genomics Decision Making.

PURPOSE: Precision oncology mandates developing standardized common data models (CDMs) to facilitate analyses and enable clinical decision making. Expert-opinion-based precision oncology initiatives are epitomized in Molecular Tumor Boards (MTBs), which process large volumes of clinical-genomic data to match genotypes with molecularly guided therapies. METHODS: We used the Johns Hopkins University MTB as a use case and developed a precision oncology core data model (Precision-DM) to capture key clinical-genomic data elements. We leveraged existing CDMs, building upon the Minimal Common Oncology Data Elements model (mCODE). Our model was defined as a set of profiles with multiple data elements, focusing on next-generation sequencing and variant annotations. Most elements were mapped to terminologies or code sets and the Fast Healthcare Interoperability Resources (FHIR). We subsequently compared our Precision-DM with existing CDMs, including the National Cancer Institute's Genomic Data Commons (NCI GDC), mCODE, OSIRIS, the clinical Genome Data Model (cGDM), and the genomic CDM (gCDM). RESULTS: Precision-DM contained 16 profiles and 355 data elements. 39% of the elements derived values from selected terminologies or code sets, and 61% were mapped to FHIR. Although we used most elements contained in mCODE, we significantly expanded the profiles to include genomic annotations, resulting in a partial overlap of 50.7% between our core model and mCODE. Limited overlap was noted between Precision-DM and OSIRIS (33.2%), NCI GDC (21.4%), cGDM (9.3%), and gCDM (7.9%). Precision-DM covered most of the mCODE elements (87.7%), with less coverage for OSIRIS (35.8%), NCI GDC (11%), cGDM (26%) and gCDM (33.3%). CONCLUSION: Precision-DM supports clinical-genomic data standardization to support the MTB use case and may allow for harmonized data pulls across health care systems, academic institutions, and community medical centers.

Molecular classification of nonsmall cell lung cancer using a 4-protein quantitative assay.

BACKGROUND: The importance of definitive histological subclassification has increased as drug trials have shown benefit associated with histology in nonsmall-cell lung cancer (NSCLC). The acuity of this problem is further exacerbated by the use of minimally invasive cytology samples. Here we describe the development and validation of a 4-protein classifier that differentiates primary lung adenocarcinomas (AC) from squamous cell carcinomas (SCC). METHODS: Quantitative immunofluorescence (AQUA) was employed to measure proteins differentially expressed between AC and SCC followed by logistic regression analysis. An objective 4-protein classifier was generated to define likelihood of AC in a training set of 343 patients followed by validation in 2 independent cohorts (n = 197 and n = 235). The assay was then tested on 11 cytology specimens. RESULTS: Statistical modeling selected thyroid transcription factor 1 (TTF1), CK5, CK13, and epidermal growth factor receptor (EGFR) to generate a weighted classifier and to identify the optimal cutpoint for differentiating AC from SCC. Using the pathologist's final diagnosis as the criterion standard, the molecular test showed a sensitivity of 96% and specificity of 93%. Blinded analysis of the validation sets yielded sensitivity and specificity of 96% and 97%, respectively. Our assay classified the cytology specimens with a specificity of 100% and sensitivity of 87.5%. CONCLUSIONS: Molecular classification of NSCLC using an objective quantitative test can be highly accurate and could be translated into a diagnostic platform for broad clinical application.

Exploring the community structure of a diabetes forum.

A health forum is a kind of social network where users share information for specific topics they create. The purpose of this study was the identification of the key actors and the user communities in such a network. We used the publicly available data from a diabetes forum to create the corresponding network and explore several algorithms for the detection of user communities. The degree centrality of the network followed the power law distribution demonstrating that only a few users were the key actors in the forum. It was also shown that it is feasible to infer the top communities from a forum using certain algorithms; the key actors participated in these communities. Our approach could be applied to other health forums and be extended to examine additional aspects.

Overcoming Major Barriers to Build Efficient Decision Support Systems in Pharmacovigilance.

Many decision support methods and systems in pharmacovigilance are built without explicitly addressing specific challenges that jeopardize their eventual success. We describe two sets of challenges and appropriate strategies to address them. The first are data-related challenges, which include using extensive multi-source data of poor quality, incomplete information integration, and inefficient data visualization. The second are user-related challenges, which encompass users' overall expectations and their engagement in developing automated solutions. Pharmacovigilance decision support systems will need to rely on advanced methods, such as natural language processing and validated mathematical models, to resolve data-related issues and provide properly contextualized data. However, sophisticated approaches will not provide a complete solution if end-users do not actively participate in their development, which will ensure tools that efficiently complement existing processes without creating unnecessary resistance. Our group has already tackled these issues and applied the proposed strategies in multiple projects.

An Evaluation of Pretrained BERT Models for Comparing Semantic Similarity Across Unstructured Clinical Trial Texts.

Processing unstructured clinical texts is often necessary to support certain tasks in biomedicine, such as matching patients to clinical trials. Among other methods, domain-specific language models have been built to utilize free-text information. This study evaluated the performance of Bidirectional Encoder Representations from Transformers (BERT) models in assessing the similarity between clinical trial texts. We compared an unstructured aggregated summary of clinical trials reviewed at the Johns Hopkins Molecular Tumor Board with the ClinicalTrials.gov records, focusing on the titles and eligibility criteria. Seven pretrained BERT-Based models were used in our analysis. Of the six biomedical-domain-specific models, only SciBERT outperformed the original BERT model by accurately assigning higher similarity scores to matched than mismatched trials. This finding is promising and shows that BERT and, likely, other language models may support patient-trial matching.

Natural Language Processing Approaches for Retrieval of Clinically Relevant Genomic Information in Cancer.

The accelerating impact of genomic data in clinical decision-making has generated a paradigm shift from treatment based on the anatomic origin of the tumor to the incorporation of key genomic features to guide therapy. Assessing the clinical validity and utility of the genomic background of a patient's cancer represents one of the emerging challenges in oncology practice, demanding the development of automated platforms for extracting clinically relevant genomic information from medical texts. We developed PubMiner, a natural language processing tool to extract and interpret cancer type, therapy, and genomic information from biomedical abstracts. Our initial focus has been the retrieval of gene names, variants, and negations, where PubMiner performed highly in terms of total recall (91.7%) with a precision of 79.7%. Our next steps include developing a web-based interface to promote personalized treatment based on each tumor's unique genomic fingerprints.

Network analysis of possible anaphylaxis cases reported to the US vaccine adverse event reporting system after H1N1 influenza vaccine.

The identification of signals from spontaneous reporting systems plays an important role in monitoring the safety of medical products. Network analysis (NA) allows the representation of complex interactions among the key elements of such systems. We developed a network for a subset of the US Vaccine Adverse Event Reporting System (VAERS) by representing the vaccines/adverse events (AEs) and their interconnections as the nodes and the edges, respectively; this subset we focused upon included possible anaphylaxis reports that were submitted for the H1N1 influenza vaccine. Subsequently, we calculated the main metrics that characterize the connectivity of the nodes and applied the island algorithm to identify the densest region in the network and, thus, identify potential safety signals. AEs associated with anaphylaxis formed a dense region in the 'anaphylaxis' network demonstrating the strength of NA techniques for pattern recognition. Additional validation and development of this approach is needed to improve future pharmacovigilance efforts.

Can FHIR Support Standardization in Post-Market Safety Surveillance?

The Fast Healthcare Interoperability Resources (FHIR) contain multiple data-exchange standards that aim at optimizing healthcare information exchange. One of them, the FHIR AdverseEvent, may support post-market safety surveillance. We examined its readiness using the Food and Drug Administration's (FDA) Adverse Event Reporting System (FAERS). Our methodology focused on mapping the public FAERS data fields to the FHIR AdverseEvent Resource elements and developing a software tool to automate this process. We mapped directly nine and indirectly two of the twenty-six FAERS elements, while six elements were assigned default values. This exploration further revealed opportunities for adding extra elements to the FHIR standard, based on critical FAERS pieces of information reviewed at the FDA. The existing version of the FHIR AdverseEvent Resource may standardize some of the FAERS information but has to be modified and extended to maximize its value in post-market safety surveillance.

Feature engineering and machine learning for causality assessment in pharmacovigilance: Lessons learned from application to the FDA Adverse Event Reporting System.

BACKGROUND: Our objective was to support the automated classification of Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) reports for their usefulness in assessing the possibility of a causal relationship between a drug product and an adverse event. METHOD: We used a data set of 326 redacted FAERS reports that was previously annotated using a modified version of the World Health Organization-Uppsala Monitoring Centre criteria for drug causality assessment by a group of SEs at the FDA and supported a similar study on the classification of reports using supervised machine learning and text engineering methods. We explored many potential features, including the incorporation of natural language processing on report text and information from external data sources, for supervised learning and developed models for predicting the classification status of reports. We then evaluated the models on a larger data set of previously unseen reports. RESULTS: The best-performing models achieved recall and F1 scores on both data sets above 0.80 for the identification of assessable reports (i.e. those containing enough information to make an informed causality assessment) and above 0.75 for the identification of reports meeting at least a Possible causality threshold. CONCLUSIONS: Causal inference from FAERS reports depends on many components with complex logical relationships that are yet to be made fully computable. Efforts focused on readily addressable tasks, such as quickly eliminating unassessable reports, fit naturally in SE's thought processes to provide real enhancements for FDA workflows.

Sixteen years of ICPC use in Norwegian primary care: looking through the facts.

BACKGROUND: The International Classification for Primary Care (ICPC) standard aims to facilitate simultaneous and longitudinal comparisons of clinical primary care practice within and across country borders; it is also used for administrative purposes. This study evaluates the use of the original ICPC-1 and the more complete ICPC-2 Norwegian versions in electronic patient records. METHODS: We performed a retrospective study of approximately 1.5 million ICPC codes and diagnoses that were collected over a 16-year period at 12 primary care sites in Norway. In the first phase of this period (transition phase, 1992-1999) physicians were allowed to not use an ICPC code in their practice while in the second phase (regular phase, 2000-2008) the use of an ICPC code was mandatory. The ICPC codes and diagnoses defined a problem event for each patient in the PROblem-oriented electronic MEDical record (PROMED). The main outcome measure of our analysis was the percentage of problem events in PROMEDs with inappropriate (or missing) ICPC codes and of diagnoses that did not map the latest ICPC-2 classification. Specific problem areas (pneumonia, anaemia, tonsillitis and diabetes) were examined in the same context. RESULTS: Codes were missing in 6.2% of the problem events; incorrect codes were observed in 4.0% of the problem events and text mismatch between the diagnoses and the expected ICPC-2 diagnoses text in 53.8% of the problem events. Missing codes were observed only during the transition phase while incorrect and inappropriate codes were used all over the 16-year period. The physicians created diagnoses that did not exist in ICPC. These 'new' diagnoses were used with varying frequency; many of them were used only once. Inappropriate ICPC-2 codes were also observed in the selected problem areas and for both phases. CONCLUSIONS: Our results strongly suggest that physicians did not adhere to the ICPC standard due to its incompleteness, i.e. lack of many clinically important diagnoses. This indicates that ICPC is inappropriate for the classification of problem events and the clinical practice in primary care.