Recommended data elements for health registries: a survey from a German funding initiative.

BACKGROUND: The selection of data elements is a decisive task within the development of a health registry. Having the right metadata is crucial for answering the particular research questions. Furthermore, the set of data elements determines the registries' readiness of interoperability and data reusability to a major extent. Six health registries shared and published their metadata within a German funding initiative. As one step in the direction of a common set of data elements, a selection of those metadata was evaluated with regard to their appropriateness for a broader usage. METHODS: Each registry was asked to contribute a 10%-selection of their data elements to an evaluation sample. The survey was set up with the online survey tool "LimeSurvey Cloud". The registries and an accompanying project participated in the survey with one vote for each project. The data elements were offered in content groups along with the question of whether the data element is appropriate for health registries on a broader scale. The question could be answered using a Likert scale with five options. Furthermore, "no answer" was allowed. The level of agreement was assessed using weighted Cohen's kappa and Kendall's coefficient of concordance. RESULTS: The evaluation sample consisted of 269 data elements. With a grade of "perhaps recommendable" or higher in the mean, 169 data elements were selected. These data elements belong preferably to groups' demography, education/occupation, medication, and nutrition. Half of the registries lost significance compared with their percentage of data elements in the evaluation sample, one remained stable. The level of concordance was adequate. CONCLUSIONS: The survey revealed a set of 169 data elements recommended for health registries. When developing a registry, this set could be valuable help in selecting the metadata appropriate to answer the registry's research questions. However, due to the high specificity of research questions, data elements beyond this set will be needed to cover the whole range of interests of a register. A broader discussion and subsequent surveys are needed to establish a common set of data elements on an international scale.

Data Quality and Data Quantity: Complements or Contradictions?

Although data quality is well defined, the relationship to data quantity remains unclear. Especially the big data approach promises advantages of volume in comparison with small samples in good quality. Aim of this study was to review this issue. Based on the experiences with six registries within a German funding initiative, the definition of data quality provided by the International Organization for Standardization (ISO) was confronted with several aspects of data quantity. The results of a literature search combining both concepts were considered additionally. Data quantity was identified as an umbrella of some inherent characteristics of data like case and data completeness. The same time, quantity could be regarded as a non inherent characteristic of data beyond the ISO standard focusing on the breadth and depth of metadata, i.e. data elements along with their value sets. The FAIR Guiding Principles take into account the latter solely. Surprisingly, the literature agreed in demanding an increase in data quality with volume, turning the big data approach inside out. A usage of data without context - as it could be the case in data mining or machine learning - is neither covered by the concept of data quality nor of data quantity.

Cross-Registry Benchmarking of Data Quality: Lessons Learned.

Feedback of data quality measures to study sites is an established procedure in the management of registries. Comparisons of data quality between registries as a whole are missing. We implemented a cross-registry benchmarking of data quality within the field of health services research for six projects. Five (2020) and six (2021) quality indicators were selected from a national recommendation. The calculation of the indicators was adjusted to the registries' specific settings. Nineteen (2020) and 29 results (2021) could be included in the yearly quality report. Seventy-four per cent (2020) and 79% (2021) of the results did not include the threshold in their 95%-confidence-limits. The benchmarking revealed several starting points for a weak-point analysis through a comparison of results with a predefined threshold as well as through comparisons among each other. In the future, a cross-registry benchmarking might be part of services provided through a health services research infrastructure.

ICT Tools for Registry Research: A Market Survey.

Registries in health research are complex systems requiring a diverse infrastructure with information and communications technology tools (ICT tools) for manifold tasks. Those tools should support not only data management but also several core and accompanying processes. Recent trends in registry research also need to be taken into account. Thirty-five vendors, suppliers, and experts were included in a survey on ICT tools for registries and cohorts. Information from 28 tools was available for a preliminary analysis. In comparison to 2015 and 2018, coverage of core processes such as registry development or data analysis and utilization increased from below 40% to 39% and higher. Recording patient-reported information and linkage to other data collections was well covered. However, near-patient trends were less supported. The market offers a rich selection of commercial and non-commercial ICT tools for registry research. Due to the manifold offers available from the market, in-house developed software should be an absolute exception.

Metadata Definition in Registries: What Is a Data Element?

Observational research benefits from a rich methodological foundation of registry development and operation published in international and national guidelines. Metadata management is an essential part of registry implementation based on concepts of data elements and value sets. The metadata from six German registries revealed vastly divergent interpretations of the concept of data elements. The different perspectives of research questions, data acquisition and data storage were all represented in the registries' catalogs of data elements. Consequently, the whole life cycle of a registry needs to be accompanied by a catalog of data elements, which has to be continuously adapted to the changing perspectives. A standard for the representation of those metadata is still missing. The FAIR Guiding Principles introduce important methodological requirements, but the tools for their fulfillment in respect to the management of metadata are still in its infancy.

FAIR and Quality Assured Data - The Use Case of Trueness.

The FAIR Guiding Principles do not address the quality of data and metadata. Therefore, data collections could be FAIR but useless. In a funding initiative of registries for health services research, trueness of data received special attention. Completeness in the definition of recall was selected to represent this dimension in a cross-registry benchmarking. The first analyses of completeness revealed a diversity of its implementation. No registry was able to present results exactly as requested in a guideline on data quality. Two registries switched to a source data verification as alternative, the three others downsized to the dimension integrity. The experiences underline that the achievement of appropriate data quality is a matter of costs and resources, whereas the current Guiding Principles quote for a transparent culture regarding data and metadata. We propose the extension to FAIR-Q, data collections should not only be findable, accessible, interoperable, and reusable, but also quality assured.

[Cross-Project Support of Registries in Development, Implementation and Operation]. / Projektübergreifende Unterstützung von Registern in Entwicklung, Umsetzung und Betrieb.

OBJECTIVE: The German Federal Ministry of Education and Research funded a project accompanying a funding initiative for registries in health services research. The aim was to provide cross-registry support initially for 16 and later 6 projects with regard to methodological, technical and structural standards. METHODS: The 16 projects were initially guided in concept development, e. g., providing a template for a registry protocol. Furthermore, an expert consultation was organized and carried out. To assist in the selection of an IT solution, a challenge workshop was hosted where different vendors presented their software for registries. The catalogs of data elements of the projects were migrated into a metadata catalog and transferred to the standard model of ISO/IEC 11179. A set of quality indicators was defined for a cross-registry quality management approach to be implemented during the operational phase. To improve data quality, the indicators were to be transmitted and evaluated on a regular basis. RESULTS: The template for a registry protocol was used by the majority of projects when applying for funding of their operational phase. At the workshop on IT solutions, 12 products for registry software were presented; however, the projects opted for other solutions for different reasons. Transferring the catalogs of data elements into a standard model enabled a comparison of attributes and value sets, which in turn enabled formulation of recommendations for important elements. A set of five quality indicators was defined for quality management, for which an initial evaluation was carried out for 2020. CONCLUSION: The template of a registry protocol serves a systematic development of a concept. The use of a uniformly structured catalog of data elements supports compliance with the FAIR principles. Monitoring of data quality can be achieved by regularly identifying quality indicators across registries.

Metadata of Registries: Results from an Initiative in Health Services Research.

Metadata management is an essential condition to follow the FAIR principles. Therefore, metadata management was one asset of an accompanying project within a funding scheme for registries in health services research. The metadata of the funded projects were acquired, combined in a database compatible with the metamodel of ISO/IEC 11179 "Information technology - Metadata registries" third edition (ISO/IEC 11179-3), and analyzed in order to support the development and the operation of the registries. In the second phase of the funding scheme, six registries delivered a complete update of their metadata. The mean number of data elements increased from 245.7 to 473.5 and the mean number of values from 569.5 to 1,306.0. The conceptual core of the database had to be extended by one third to cover the new elements. The reason for this increase remained unclear. Constraints from the grant might be causal, a deviation from an evidence-based development process as well. It is questionable, whether the revealed quality of the metadata is sufficient to fulfill the FAIR principles. The extension of the metamodel of ISO/IEC 11179-3 is in agreement with the literature. However, further research is needed to find workable solutions for metadata management.

Recent Trends in Patient Registries for Health Services Research.

BACKGROUND: Patient registries are an established methodology in health services research. Since more than 150 years, registries collect information concerning groups of similar patients to answer research questions. Elaborated recommendations about an appropriate development and an efficient operation of registries are available. However, the scene changes rapidly. OBJECTIVES: The aim of the study is to describe current trends in registry research for health services research. METHODS: Registries developed within a German funding scheme for model registries in health services research were analyzed. The observations were compared with recent recommendations of the Agency for Healthcare Research and Quality (AHRQ) on registries in the 21st century. RESULTS: Analyzing six registries from the funding scheme revealed the following trends: recruiting healthy individuals, representing familial or other interpersonal relationships, recording of patient-reported experiences or outcomes, accepting participants as study sites, active informing of participants, integrating the registry with other data collections, and transferring data from the registry to electronic patient records. This list partly complies with the issues discussed by the AHRQ. The AHRQ structured its ideas in five chapters, increasing focus on the patient, engaging patients as partners, digital health and patient registries, direct-to-patient registry, and registry networks. CONCLUSION: For the near future, it can be said that the concept and the design of a registry should place the patient in the center. Registries will be increasingly linked together and interconnected with other data collections. New challenges arise regarding the management of data quality and the interpretation of results from less controlled settings. Here, further research related to the methodology of registries is needed.

Process Coverage and Use Case Support of Health Registry Software in Germany.

Registries usually operate an IT-infrastructure supporting at least data management as one of the business processes. Several activities in Germany between 2007 and 2018 surveyed the market of respective software products. Combining a survey with representatives of software products with a workshop protocol of software demonstrations, a detailed insight into the market of IT-components arose. A comparison between 2015 and 2018 revealed little progress. The focus is still electronic data capture functionality. With the presented activities, rich material is available to assist registry developers in the planning of their IT-infrastructure and the selection of software products.

Towards a Core Set of Indicators for Data Quality of Registries.

Registries are a widely accepted method in health services research. Registry owners are faced with the challenge to document and assure data quality, vital for answering research questions and conducting quality research. Therefore a survey on indicators for data quality was conducted as part of a German funding initiative. A list of 51 pre-defined quality indicators was provided to 16 patient registry projects in a web based survey. The assessment included three criteria derived from the Rand Appropriateness Method (RAM), the application area, and three criteria representing a project-specific perspective. Considering the criteria adapted from RAM, a core set of 17 indicators could be identified. This core set covered important dimensions, such as case completeness, data completeness and validity. Adding importance as a criterion from a project-specific perspective led to a subset of six indicators. The selection of indicators identified through this survey may be applied on different use cases, e.g. a) benchmarking between registries, b) benchmarking of study sites, and c) value-based remuneration of study sites. Thus, the presented core set of indicators can be used as a basis to improve quality of registry data with a systematic approach.

Bridging Documentation and Metadata Standards: Experiences from a Funding Initiative for Registries.

Within a funding initiative for patient registries in Germany, data specifications from 15 projects were collected in a structured format. Subsequently, the data specifications were transferred into a structure following the ISO/IEC 11179-3 standard for metadata registries. The data specifications included a median of 16 documentation objects and 196 data elements per project. Only Common designations were used. The bridging to ISO/IEC 11179-3 revealed several overlaps between the specifications. For example, bridging can be used to align the multifold designations of sex and sex categories as well as to harmonize the respective value lists. Further work intends to create a metadata repository based on a community-driven approach as part of the IT-infrastructure of this funding initiative. Without this infrastructure, comparability of the data specifications is unlikely.

IT Infrastructure for Registries in Health Services Research: A Market Study in Germany.

Registries are increasingly implemented to record the practice of health care. Within a national funding scheme for registries, an accompanying project was launched to support the design of the registries' IT infrastructure amongst other tasks for 16 projects. A challenge of data management systems was organized by the accompanying project in order to enable the projects to define realistic expectations towards IT support in their research protocols. Twelve vendors participated in the challenge. They presented their solutions for selected use cases. In advance, the projects considered a sufficient authorization concept and the possibility to export data to be of highest importance. However, the systems covered mainly core processes of electronic data capture. The accompanying project will continue its support for the next stage of the funding scheme, which will be the implementation of the registries that win a competitive review of their research protocols prepared in the concept development stage.