The Kaiser Permanente Research Bank Cancer Cohort: a collaborative resource to improve cancer care and survivorship.

BACKGROUND: The Kaiser Permanente Research Bank (KPRB) is collecting biospecimens and surveys linked to electronic health records (EHR) from approximately 400,000 adult KP members. Within the KPRB, we developed a Cancer Cohort to address issues related to cancer survival, and to understand how genetic, lifestyle and environmental factors impact cancer treatment, treatment sequelae, and prognosis. We describe the Cancer Cohort design and implementation, describe cohort characteristics after 5 years of enrollment, and discuss future directions. METHODS: Cancer cases are identified using rapid case ascertainment algorithms, linkage to regional or central tumor registries, and direct outreach to KP members with a history of cancer. Enrollment is primarily through email invitation. Participants complete a consent form, survey, and donate a blood or saliva sample. All cancer types are included. RESULTS: As of December 31, 2020, the cohort included 65,225 cases (56% female, 44% male) verified in tumor registries. The largest group was diagnosed between 60 and 69 years of age (31%) and are non-Hispanic White (83%); however, 10,076 (16%) were diagnosed at ages 18-49 years, 4208 (7%) are Hispanic, 3393 (5%) are Asian, and 2389 (4%) are Black. The median survival time is 14 years. Biospecimens are available on 98% of the cohort. CONCLUSIONS: The KPRB Cancer Cohort is designed to improve our understanding of treatment efficacy and factors that contribute to long-term cancer survival. The cohort's diversity - with respect to age, race/ethnicity and geographic location - will facilitate research on factors that contribute to cancer survival disparities.

A Data Element-Function Conceptual Model for Data Quality Checks.

INTRODUCTION: In aggregate, existing data quality (DQ) checks are currently represented in heterogeneous formats, making it difficult to compare, categorize, and index checks. This study contributes a data element-function conceptual model to facilitate the categorization and indexing of DQ checks and explores the feasibility of leveraging natural language processing (NLP) for scalable acquisition of knowledge of common data elements and functions from DQ checks narratives. METHODS: The model defines a "data element", the primary focus of the check, and a "function", the qualitative or quantitative measure over a data element. We applied NLP techniques to extract both from 172 checks for Observational Health Data Sciences and Informatics (OHDSI) and 3,434 checks for Kaiser Permanente's Center for Effectiveness and Safety Research (CESR). RESULTS: The model was able to classify all checks. A total of 751 unique data elements and 24 unique functions were extracted. The top five frequent data element-function pairings for OHDSI were Person-Count (55 checks), Insurance-Distribution (17), Medication-Count (16), Condition-Count (14), and Observations-Count (13); for CESR, they were Medication-Variable Type (175), Medication-Missing (172), Medication-Existence (152), Medication-Count (127), and Socioeconomic Factors-Variable Type (114). CONCLUSIONS: This study shows the efficacy of the data element-function conceptual model for classifying DQ checks, demonstrates early promise of NLP-assisted knowledge acquisition, and reveals the great heterogeneity in the focus in DQ checks, confirming variation in intrinsic checks and use-case specific "fitness-for-use" checks.

Data Quality Assessment and Multi-Organizational Reporting: Tools to Enhance Network Knowledge.

OBJECTIVE: Multi-organizational research requires a multi-organizational data quality assessment (DQA) process that combines and compares data across participating organizations. We demonstrate how such a DQA approach complements traditional checks of internal reliability and validity by allowing for assessments of data consistency and the evaluation of data patterns in the absence of an external "gold standard." METHODS: We describe the DQA process employed by the Data Coordinating Center (DCC) for Kaiser Permanente's (KP) Center for Effectiveness and Safety Research (CESR). We emphasize the CESR DQA reporting system that compares data summaries from the eight KP organizations in a consistent, standardized manner. RESULTS: We provide examples of multi-organization comparisons from DQA to confirm expectations about different aspects of data quality. These include: 1) comparison of direct data extraction from the electronic health records (EHR) and 2) comparison of non-EHR data from disparate sources. DISCUSSION: The CESR DCC has developed codes and procedures for efficiently implementing and reporting DQA. The CESR DCC approach is to 1) distribute DQA tools to empower data managers at each organization to assess their data quality at any time, 2) summarize and disseminate findings to address data shortfalls or document idiosyncrasies, and 3) engage data managers and end-users in an exchange of knowledge about the quality and its fitness for use. CONCLUSION: The KP CESR DQA model is applicable to networks hoping to improve data quality. The multi-organizational reporting system promotes transparency of DQA, adds to network knowledge about data quality, and informs research.

Pragmatic (trial) informatics: a perspective from the NIH Health Care Systems Research Collaboratory.

Pragmatic clinical trials (PCTs) are research investigations embedded in health care settings designed to increase the efficiency of research and its relevance to clinical practice. The Health Care Systems Research Collaboratory, initiated by the National Institutes of Health Common Fund in 2010, is a pioneering cooperative aimed at identifying and overcoming operational challenges to pragmatic research. Drawing from our experience, we present 4 broad categories of informatics-related challenges: (1) using clinical data for research, (2) integrating data from heterogeneous systems, (3) using electronic health records to support intervention delivery or health system change, and (4) assessing and improving data capture to define study populations and outcomes. These challenges impact the validity, reliability, and integrity of PCTs. Achieving the full potential of PCTs and a learning health system will require meaningful partnerships between health system leadership and operations, and federally driven standards and policies to ensure that future electronic health record systems have the flexibility to support research.

A Comparison of Data Quality Assessment Checks in Six Data Sharing Networks.

OBJECTIVE: To compare rule-based data quality (DQ) assessment approaches across multiple national clinical data sharing organizations. METHODS: Six organizations with established data quality assessment (DQA) programs provided documentation or source code describing current DQ checks. DQ checks were mapped to the categories within the data verification context of the harmonized DQA terminology. To ensure all DQ checks were consistently mapped, conventions were developed and four iterations of mapping performed. Difficult-to-map DQ checks were discussed with research team members until consensus was achieved. RESULTS: Participating organizations provided 11,026 DQ checks, of which 99.97 percent were successfully mapped to a DQA category. Of the mapped DQ checks (N=11,023), 214 (1.94 percent) mapped to multiple DQA categories. The majority of DQ checks mapped to Atemporal Plausibility (49.60 percent), Value Conformance (17.84 percent), and Atemporal Completeness (12.98 percent) categories. DISCUSSION: Using the common DQA terminology, near-complete (99.97 percent) coverage across a wide range of DQA programs and specifications was reached. Comparing the distributions of mapped DQ checks revealed important differences between participating organizations. This variation may be related to the organization's stakeholder requirements, primary analytical focus, or maturity of their DQA program. Not within scope, mapping checks within the data validation context of the terminology may provide additional insights into DQA practice differences. CONCLUSION: A common DQA terminology provides a means to help organizations and researchers understand the coverage of their current DQA efforts as well as highlight potential areas for additional DQA development. Sharing DQ checks between organizations could help expand the scope of DQA across clinical data networks.

A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data.

OBJECTIVE: Harmonized data quality (DQ) assessment terms, methods, and reporting practices can establish a common understanding of the strengths and limitations of electronic health record (EHR) data for operational analytics, quality improvement, and research. Existing published DQ terms were harmonized to a comprehensive unified terminology with definitions and examples and organized into a conceptual framework to support a common approach to defining whether EHR data is 'fit' for specific uses. MATERIALS AND METHODS: DQ publications, informatics and analytics experts, managers of established DQ programs, and operational manuals from several mature EHR-based research networks were reviewed to identify potential DQ terms and categories. Two face-to-face stakeholder meetings were used to vet an initial set of DQ terms and definitions that were grouped into an overall conceptual framework. Feedback received from data producers and users was used to construct a draft set of harmonized DQ terms and categories. Multiple rounds of iterative refinement resulted in a set of terms and organizing framework consisting of DQ categories, subcategories, terms, definitions, and examples. The harmonized terminology and logical framework's inclusiveness was evaluated against ten published DQ terminologies. RESULTS: Existing DQ terms were harmonized and organized into a framework by defining three DQ categories: (1) Conformance (2) Completeness and (3) Plausibility and two DQ assessment contexts: (1) Verification and (2) Validation. Conformance and Plausibility categories were further divided into subcategories. Each category and subcategory was defined with respect to whether the data may be verified with organizational data, or validated against an accepted gold standard, depending on proposed context and uses. The coverage of the harmonized DQ terminology was validated by successfully aligning to multiple published DQ terminologies. DISCUSSION: Existing DQ concepts, community input, and expert review informed the development of a distinct set of terms, organized into categories and subcategories. The resulting DQ terms successfully encompassed a wide range of disparate DQ terminologies. Operational definitions were developed to provide guidance for implementing DQ assessment procedures. The resulting structure is an inclusive DQ framework for standardizing DQ assessment and reporting. While our analysis focused on the DQ issues often found in EHR data, the new terminology may be applicable to a wide range of electronic health data such as administrative, research, and patient-reported data. CONCLUSION: A consistent, common DQ terminology, organized into a logical framework, is an initial step in enabling data owners and users, patients, and policy makers to evaluate and communicate data quality findings in a well-defined manner with a shared vocabulary. Future work will leverage the framework and terminology to develop reusable data quality assessment and reporting methods.

Preparing for the ICD-10-CM Transition: Automated Methods for Translating ICD Codes in Clinical Phenotype Definitions.

BACKGROUND: The national mandate for health systems to transition from ICD-9-CM to ICD-10-CM in October 2015 has an impact on research activities. Clinical phenotypes defined by ICD-9-CM codes need to be converted to ICD-10-CM, which has nearly four times more codes and a very different structure than ICD-9-CM. METHODS: We used the Centers for Medicare & Medicaid Services (CMS) General Equivalent Maps (GEMs) to translate, using four different methods, condition-specific ICD-9-CM code sets used for pragmatic trials (n=32) into ICD-10-CM. We calculated the recall, precision, and F score of each method. We also used the ICD-9-CM and ICD-10-CM value sets defined for electronic quality measure as an additional evaluation of the mapping methods. RESULTS: The forward-backward mapping (FBM) method had higher precision, recall and F-score metrics than simple forward mapping (SFM). The more aggressive secondary (SM) and tertiary mapping (TM) methods resulted in higher recall but lower precision. For clinical phenotype definition, FBM was the best (F=0.67), but was close to SM (F=0.62) and TM (F=0.60), judging on the F-scores alone. The overall difference between the four methods was statistically significant (one-way ANOVA, F=5.749, p=0.001). However, pairwise comparisons between FBM, SM, and TM did not reach statistical significance. A similar trend was found for the quality measure value sets. DISCUSSION: The optimal method for using the GEMs depends on the relative importance of recall versus precision for a given use case. It appears that for clinically distinct and homogenous conditions, the recall of FBM is sufficient. The performance of all mapping methods was lower for heterogeneous conditions. Since code sets used for phenotype definition and quality measurement can be very similar, there is a possibility of cross-fertilization between the two activities. CONCLUSION: Different mapping approaches yield different collections of ICD-10-CM codes. All methods require some level of human validation.

The impact of continued intervention on weight: Five-year results from the weight loss maintenance trial.

OBJECTIVE: In the Weight Loss Maintenance (WLM) Trial, a personal contact (PC) intervention sustained greater weight loss relative to a self-directed (SD) group over 30 months. This study investigated the effects of continued intervention over an additional 30 months and overall weight change across the entire WLM Trial. METHODS: WLM had 3 phases. Phase 1 was a 6-month weight loss program. In Phase 2, those who lost ≥4 kg were randomized to a 30-month maintenance trial. In Phase 3, PC participants (n = 196, three sites) were re-randomized to no further intervention (PC-Control) or continued intervention (PC-Active) for 30 more months; 218 SD participants were also followed. RESULTS: During Phase 3, weight increased 1.0 kg in PC-Active and 0.5 kg in PC-Control (mean difference 0.6 kg; 95% CI:-1.4 to 2.7; P = 0.54). Mean weight change over the entire study was -3.2 kg in those originally assigned to PC (PC-Combined) and -1.6 kg in SD (mean difference -1.6 kg; 95% CI:-3.0 to -0.1; P = 0.04). CONCLUSIONS: After 30 months of the PC maintenance intervention, continuation for another 30 months provided no additional benefit. However, across the entire study, weight loss was slightly greater in those originally assigned to PC.

Developing a data infrastructure for a learning health system: the PORTAL network.

The Kaiser Permanente & Strategic Partners Patient Outcomes Research To Advance Learning (PORTAL) network engages four healthcare delivery systems (Kaiser Permanente, Group Health Cooperative, HealthPartners, and Denver Health) and their affiliated research centers to create a new national network infrastructure that builds on existing relationships among these institutions. PORTAL is enhancing its current capabilities by expanding the scope of the common data model, paying particular attention to incorporating patient-reported data more systematically, implementing new multi-site data governance procedures, and integrating the PCORnet PopMedNet platform across our research centers. PORTAL is partnering with clinical research and patient experts to create cohorts of patients with a common diagnosis (colorectal cancer), a rare diagnosis (adolescents and adults with severe congenital heart disease), and adults who are overweight or obese, including those with pre-diabetes or diabetes, to conduct large-scale observational comparative effectiveness research and pragmatic clinical trials across diverse clinical care settings.

Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH Health Care Systems Collaboratory.

Widespread sharing of data from electronic health records and patient-reported outcomes can strengthen the national capacity for conducting cost-effective clinical trials and allow research to be embedded within routine care delivery. While pragmatic clinical trials (PCTs) have been performed for decades, they now can draw on rich sources of clinical and operational data that are continuously fed back to inform research and practice. The Health Care Systems Collaboratory program, initiated by the NIH Common Fund in 2012, engages healthcare systems as partners in discussing and promoting activities, tools, and strategies for supporting active participation in PCTs. The NIH Collaboratory consists of seven demonstration projects, and seven problem-specific working group 'Cores', aimed at leveraging the data captured in heterogeneous 'real-world' environments for research, thereby improving the efficiency, relevance, and generalizability of trials. Here, we introduce the Collaboratory, focusing on its Phenotype, Data Standards, and Data Quality Core, and present early observations from researchers implementing PCTs within large healthcare systems. We also identify gaps in knowledge and present an informatics research agenda that includes identifying methods for the definition and appropriate application of phenotypes in diverse healthcare settings, and methods for validating both the definition and execution of electronic health records based phenotypes.

Development and Implementation of a Tailored Self-assessment Tool in an Internet-based Weight Loss Maintenance Program.

BACKGROUND: Using the Internet to replicate client/counselor interactions provides a tremendous opportunity to disseminate interventions at relatively low cost per participant. However, there are substantial challenges with this approach. The Weight Loss Maintenance Trial (WLM) compared two long-term weight-maintenance interventions: (1) a personal contact arm and (2) an Internet arm, to a third self-directed control arm. The Internet arm focused on use of an interactive website for support of long-term weight maintenance. This paper describes a highly interactive self-assessment tool developed for use in the WLM trial Internet intervention arm. METHODS: The Tailored Self-Assessment (TSA) website tool was an interactive resource for those WLM participants assigned to the Internet arm to review their personal weight-management progress and make choices about future weight-management actions. The TSA was highly tailored and ended with a suggested list of personalized action plans. While the participant could complete the TSA at any time, criteria-based reminder messages prompted participation. RESULTS: The TSA was one of 27 interactive tools on the WLM website. Over the course of the 28 months, the TSA was completed 800 times by the 348 randomized participants. Fifty-three percent of the participants (185/348) used the TSA at least once (range: 0, 110) and 72% of the 185 participants who did complete the TSA at least once, completed it more than once. CONCLUSION: The Internet has great potential to impact health behavior by attempting to replicate personal counseling. We learned that while development is complex and appears costly, tailored strategies based on client feedback are likely worthwhile and should be formally tested.

Self-reported health and functional status information improves prediction of inpatient admissions and costs.

OBJECTIVES: To determine whether adding selfreported health and functional status data to a diagnostic risk-score model explains additional variance in predicting inpatient admissions and costs. STUDY DESIGN: Retrospective observational analysis. METHODS: We used data from a Health Status Questionnaire (HSQ), completed by 6407 Kaiser Permanente Northwest Medicare patients between December 2006 and October 2008. We used answers from 3 items on the HSQ: (1) General Self-rated Health score, (2) needing help with 1 or more activities of daily living, and (3) having a bothersome health condition. We calculated a DxCG relative risk score from utilization information in the year prior to the survey, using electronic medical records. We compared: (1) DxCG as the sole independent variable and (2) DxCG plus the 3 items as independent variables. We estimated area under the curve (AUC) for each model. Any inpatient admission (yes/no) and being in the top 10% of costs (in the year after survey) were the dependent variables for the first and second logistic regression models, respectively. RESULTS: The 3 items explained an additional 2.8% and 4.0% of variance for inpatient admissions and top 10% of costs,respectively, in addition to the variance explained by the DxCG score alone. For DxCG alone, the AUC was 0.686 (95% confidence interval [CI] 0.663-0.710) and 0.741 (95% CI 0.719- 0.764), respectively, for inpatient admissions and top 10% of costs and improved to 0.709 (95% CI 0.687-0.730) and 0.770 (95% CI 0.749-0.790) when the 3 self-reported items were added. CONCLUSIONS: Using self-reported health information improved the predictive power of a DxCG model to forecast inpatient admissions and patient cost-tier.

Associations of internet website use with weight change in a long-term weight loss maintenance program.

BACKGROUND: The Weight Loss Maintenance Trial (WLM) compared two long-term weight-maintenance interventions, a personal contact arm and an Internet arm, with a no-treatment control after an initial six-month Phase I weight loss program. The Internet arm focused on use of an interactive website for support of long-term weight maintenance. There is limited information about patterns of website use and specific components of an interactive website that might help promote maintenance of weight loss. OBJECTIVE: This paper presents a secondary analysis of the subset of participants in the Internet arm and focuses on website use patterns and features associated with long-term weight maintenance. METHODS: Adults at risk for cardiovascular disease (CVD) who lost at least 4 kilograms in an initial 20-week group-based, behavioral weight-loss program were trained to use an interactive website for weight loss maintenance. Of the 348 participants, 37% were male and 38% were African American. Mean weight loss was 8.6 kilograms. Participants were encouraged to log in at least weekly and enter a current weight for the 30-month study period. The website contained features that encouraged setting short-term goals, creating action plans, and reinforcing self-management habits. The website also included motivational modules, daily tips, and tailored messages. Based on log-in and weight-entry frequency, we divided participants into three website use categories: consistent, some, and minimal. RESULTS: Participants in the consistent user group (n = 212) were more likely to be older (P = .002), other than African American (P = .02), and more educated (P = .01). While there was no significant difference between website use categories in the amount of Phase I change in body weight (P = .45) or income (P = .78), minimal website users (n = 75) were significantly more likely to have attended fewer Phase I sessions (P = .001) and had a higher initial body mass index (BMI) (P < .001). After adjusting for baseline characteristics including initial BMI, variables most associated with less weight regain included: number of log-ins (P = .001), minutes on the website (P < .001), number of weight entries (P = .002), number of exercise entries (P < .001), and sessions with additional use of website features after weight entry (P = .002). CONCLUSION: Participants defined as consistent website users of an interactive behavioral website designed to promote maintenance of weight loss were more successful at maintaining long-term weight loss. TRIAL REGISTRATION: NCT00054925; http://clinicaltrials.gov/ct2/show/NCT00054925 (Archived by WebCite at http://www.webcitation.org/5rC7523ue).

Development and implementation cost analysis of telephone- and Internet-based interventions for the maintenance of weight loss.

OBJECTIVES: The Weight Loss Maintenance Trial (WLM) was a multicenter, randomized trial comparing two weight loss maintenance interventions, a personal contact (PC) program with primarily telephone-based monthly contacts, and an Internet-based program (interactive technology, IT), to a self-directed control group, among overweight or obese individuals at high cardiovascular risk. This study describes implementation costs of both interventions as well as IT development costs. METHODS: Resources were micro-costed in 2006 dollars from the primary perspective of a sponsoring healthcare system considering adopting an extant intervention, rather than developing its own. Costs were discounted at 3 percent annually. Length of trial participation was 30 months (randomization during February-November 2004). IT development costs were assessed over 36 months. Univariate and multivariate, including probabilistic, sensitivity analyses were performed. RESULTS: Total discounted IT development costs over 36 months were $839,949 ($2,414 per IT participant). Discounted 30-month implementation costs for 342 PC participants were $537,242 ($1,571 per participant), and for 348 IT participants, were $214,879 ($617 per participant). Under all plausible scenarios, PC implementation costs exceeded IT implementation costs. CONCLUSIONS: Costs of implementing and operating an Internet-based intervention for weight loss maintenance were substantially less than analogous costs of an intervention using standard phone and in-person contacts and are of a magnitude that would be attractive to many health systems, subject to demonstration of cost-effectiveness.

Comparison of strategies for sustaining weight loss: the weight loss maintenance randomized controlled trial.

CONTEXT: Behavioral weight loss interventions achieve short-term success, but re-gain is common. OBJECTIVE: To compare 2 weight loss maintenance interventions with a self-directed control group. DESIGN, SETTING, AND PARTICIPANTS: Two-phase trial in which 1032 overweight or obese adults (38% African American, 63% women) with hypertension, dyslipidemia, or both who had lost at least 4 kg during a 6-month weight loss program (phase 1) were randomized to a weight-loss maintenance intervention (phase 2). Enrollment at 4 academic centers occurred August 2003-July 2004 and randomization, February-December 2004. Data collection was completed in June 2007. INTERVENTIONS: After the phase 1 weight-loss program, participants were randomized to one of the following groups for 30 months: monthly personal contact, unlimited access to an interactive technology-based intervention, or self-directed control. Main Outcome Changes in weight from randomization. RESULTS: Mean entry weight was 96.7 kg. During the initial 6-month program, mean weight loss was 8.5 kg. After randomization, weight regain occurred. Participants in the personal-contact group regained less weight (4.0 kg) than those in the self-directed group (5.5 kg; mean difference at 30 months, -1.5 kg; 95% confidence interval [CI], -2.4 to -0.6 kg; P = .001). At 30 months, weight regain did not differ between the interactive technology-based (5.2 kg) and self-directed groups (5.5 kg; mean difference -0.3 kg; 95% CI, -1.2 to 0.6 kg; P = .51); however, weight regain was lower in the interactive technology-based than in the self-directed group at 18 months (mean difference, -1.1 kg; 95% CI, -1.9 to -0.4 kg; P = .003) and at 24 months (mean difference, -0.9 kg; 95% CI, -1.7 to -0.02 kg; P = .04). At 30 months, the difference between the personal-contact and interactive technology-based group was -1.2 kg (95% CI -2.1 to -0.3; P = .008). Effects did not differ significantly by sex, race, age, and body mass index subgroups. Overall, 71% of study participants remained below entry weight. CONCLUSIONS: The majority of individuals who successfully completed an initial behavioral weight loss program maintained a weight below their initial level. Monthly brief personal contact provided modest benefit in sustaining weight loss, whereas an interactive technology-based intervention provided early but transient benefit. TRIAL REGISTRATION: clinicaltrials.gov Identifier: NCT00054925.

Design and implementation of an interactive website to support long-term maintenance of weight loss.

BACKGROUND: For most individuals, long-term maintenance of weight loss requires long-term, supportive intervention. Internet-based weight loss maintenance programs offer considerable potential for meeting this need. Careful design processes are required to maximize adherence and minimize attrition. OBJECTIVE: This paper describes the development, implementation and use of a Web-based intervention program designed to help those who have recently lost weight sustain their weight loss over 1 year. METHODS: The weight loss maintenance website was developed over a 1-year period by an interdisciplinary team of public health researchers, behavior change intervention experts, applications developers, and interface designers. Key interactive features of the final site include social support, self-monitoring, written guidelines for diet and physical activity, links to appropriate websites, supportive tools for behavior change, check-in accountability, tailored reinforcement messages, and problem solving and relapse prevention training. The weight loss maintenance program included a reminder system (automated email and telephone messages) that prompted participants to return to the website if they missed their check-in date. If there was no log-in response to the email and telephone automated prompts, a staff member called the participant. We tracked the proportion of participants with at least one log-in per month, and analyzed log-ins as a result of automated prompts. RESULTS: The mean age of the 348 participants enrolled in an ongoing randomized trial and assigned to use the website was 56 years; 63% were female, and 38% were African American. While weight loss data will not be available until mid-2008, website use remained high during the first year with over 80% of the participants still using the website during month 12. During the first 52 weeks, participants averaged 35 weeks with at least one log-in. Email and telephone prompts appear to be very effective at helping participants sustain ongoing website use. CONCLUSIONS: Developing interactive websites is expensive, complex, and time consuming. We found that extensive paper prototyping well in advance of programming and a versatile product manager who could work with project staff at all levels of detail were essential to keeping the development process efficient. TRIAL REGISTRATION: clinicaltrials.gov NCT00054925.