Training satisfaction for subspecialty fellows in internal medicine: findings from the Veterans Affairs (VA) Learners' Perceptions Survey.

BACKGROUND: Learner satisfaction assessment is critical in the design and improvement of training programs. However, little is known about what influences satisfaction and whether trainee specialty is correlated. A national comparison of satisfaction among internal medicine subspecialty fellows in the Department of Veterans Affairs (VA) provides a unique opportunity to examine educational factors associated with learner satisfaction. We compared satisfaction across internal medicine fellows by subspecialty and compared factors associated with satisfaction between procedural versus non-procedural subspecialty fellows, using data from the Learners' Perceptions Survey (LPS), a validated survey tool. METHODS: We surveyed 2,221 internal medicine subspecialty fellows rotating through VA between 2001 and 2008. Learners rated their overall training satisfaction on a 100-point scale, and on a five-point Likert scale ranked satisfaction with items within six educational domains: learning, clinical, working and physical environments; personal experience; and clinical faculty/preceptor. RESULTS: Procedural and non-procedural fellows reported similar overall satisfaction scores (81.2 and 81.6). Non-procedural fellows reported higher satisfaction with 79 of 81 items within the 6 domains and with the domain of physical environment (4.06 vs. 3.85, p <0.001). Satisfaction with clinical faculty/preceptor and personal experience had the strongest impact on overall satisfaction for both. Procedural fellows reported lower satisfaction with physical environment. CONCLUSIONS: Internal medicine fellows are highly satisfied with their VA training. Nonprocedural fellows reported higher satisfaction with most items. For both procedural and non-procedural fellows, clinical faculty/preceptor and personal experience have the strongest impact on overall satisfaction.

Measuring use and cost of care for patients with mood disorders: the utilization and cost inventory.

BACKGROUND: Researchers conducting cost-outcome studies must account for all materially relevant care that subjects receive from their care providers. However, access to provider records is often limited. This article describes and tests the Utilization and Cost Inventory (UAC-I), a structured patient interview designed to measure costs of care when access to provider records is limited. METHODS: UAC-I was tested on 212 consenting adult veterans with mood disorder attending a VA medical center. Counts (inpatient days and outpatient encounters) and costs (dollars) computed from survey responses were compared with estimates from medical records and an alternative structured questionnaire. RESULTS: The agreement between inpatient costs computed from provider records and from UAC-I responses, assessed using the intraclass correlation coefficient (ICC), was 0.66, 95% confidence interval (CI), 0.30-0.84; the bias was -3.7%, 95% CI, -48 to 41. The ICC for the service data (inpatient days) was 0.97, 95% CI, 0.95-0.99; the bias was <1%, 95% CI, -14 to 15. The ICC for outpatient costs computed from provider records and from UAC-I responses was 0.53 95% CI, 0.38-0.65; the bias was <1%, 95% CI, -27 to 27. The ICC for outpatient encounters was 0.74, 95% CI, 0.65-0.80; the bias was <1%, 95% CI, -16 to 18. CONCLUSIONS: These results indicate that it may be feasible for cost-outcome studies to compare patient groups for inpatient and outpatient costs computed from patient self-reports.

Impact of clinical training on recruiting graduating health professionals.

OBJECTIVES: Recruiting professional staff is an important business reason for hospitals allowing health trainees to engage in supervised patient care. Whereas prior studies have focused on educational institutions, this study focuses on teaching hospitals and whether trainees' clinical experiences affect their willingness to work (ie, recruitability) for the type of healthcare center where they trained. STUDY DESIGN: A pre-post, observational study based on Learners' Perceptions Survey data in which respondents served as their own controls. METHODS: Convenience sample of 15,207 physician, 11,844 nursing, and 13,012 associated health trainees who rotated through 1 of 169 US Department of Veterans Affairs (VA) medical centers between July 1, 2014, and June 30, 2017. Generalized estimating equations computed how clinical, learning, working, and cultural experiences influenced pre-post differences in willingness to consider VA for future employment. RESULTS: VA recruitability increased dramatically from 55% pretraining to 75% post training (adjusted odds ratio [OR], 2.1; 95% CI, 2.0-2.1; P <.001) in all 3 cohorts: physician (from 39% to 59%; OR, 1.6; 95% CI, 1.5-1.6; P <.001), nursing (from 61% to 84%; OR, 2.5; 95% CI, 2.4-2.6; P <.001), and associated health trainees (from 68% to 87%; OR, 2.7; 95% CI, 2.6-2.9; P <.001). For all trainees, changes in recruitability (P <.001) were associated with how trainees rated their clinical learning environment, personal experiences, and culture of psychological safety. Satisfaction ratings with faculty and preceptors (P <.001) were associated with positive changes in recruitability among nursing and associated health students but not physician residents, whereas nursing students who gave higher ratings for interprofessional team culture became less recruitable. CONCLUSIONS: Academic medical centers can attract their health trainees for future employment if they provide positive clinical, working, learning, and cultural experiences.

Has Interprofessional Education Changed Learning Preferences? A National Perspective.

OBJECTIVE: To assess how changes in curriculum, accreditation standards, and certification and licensure competencies impacted how medical students and physician residents value interprofessional team and patient-centered care. PRIMARY DATA SOURCE: The Department of Veterans Affairs Learners' Perceptions Survey (2003-2013). The nationally administered survey asked a representative sample of 56,569 U.S. medical students and physician residents, with a comparison group of 78,038 nonphysician trainees, to rate satisfaction with 28 elements, in two overall domains, describing their clinical learning experiences at VA medical centers. STUDY DESIGN: Value preferences were scored as independent adjusted associations between an element (interprofessional team, patient-centered preceptor) and the respective overall domain (clinical learning environment, faculty, and preceptors) relative to a referent element (quality of clinical care, quality of preceptor). PRINCIPAL FINDINGS: Physician trainees valued interprofessional (14 percent vs. 37 percent, p < .001) and patient-centered learning (21 percent vs. 36 percent, p < .001) less than their nonphysician counterparts. Physician preferences for interprofessional learning showed modest increases over time (2.5 percent/year, p < .001), driven mostly by internal medicine and surgery residents. Preferences did not increase with trainees' academic progress. CONCLUSIONS: Despite changes in medical education, physician trainees continue to lag behind their nonphysician counterparts in valuing experience with interprofessional team and patient-centered care.

An empirical analysis of cost outcomes of the Texas Medication Algorithm Project.

OBJECTIVE: Disease management systems that incorporate medication algorithms have been proposed as cost-effective means to offer optimal treatment for patients with severe and chronic mental illnesses. The Texas Medication Algorithm Project was designed to compare health care costs and clinical outcomes between patients who received algorithm-guided medication management or usual care in 19 public mental health clinics. METHODS: This longitudinal cohort study for patients with major depression (N=350), bipolar disorder (N=267), and schizophrenia (N=309) applied a multi-part declining-effects cost model. Outcomes were assessed by the Inventory of Depressive Symptomatology and the Brief Psychiatric Rating Scale. RESULTS: Compared with patients in usual care, patients in algorithm-based care incurred higher medication costs and had more frequent physician visits, although these differences often became smaller with time. For major depression, algorithm-based care achieved better outcomes sustainable with time but at higher agency and non-agency costs (mixed cost-effective). For bipolar disorder, patients in algorithm-based management achieved better outcomes at lower agency costs (cost-effective). For schizophrenia, patients in algorithm-based care achieved better outcomes that diminished with time, with no detectable difference in health care costs (cost-effective). CONCLUSIONS: Cost outcomes of algorithm-based care and usual care varied by disorder and over time. For bipolar disorder and schizophrenia, algorithm-based care improved outcomes without higher costs for health care services. For major depression, substantively better and sustained outcomes were obtained but at greater costs.

Does Psychological Safety Impact the Clinical Learning Environment for Resident Physicians? Results From the VA's Learners' Perceptions Survey.

BACKGROUND: Psychological safety (PS) is the perception that it is safe to take interpersonal risks in the work environment. In teaching hospitals, PS may influence the clinical learning environment for trainees. OBJECTIVE: We assessed whether resident physicians believe they are psychologically safe, and if PS is associated with how they rate satisfaction with their clinical learning experience. METHODS: Data were extracted from the Learners' Perceptions Survey (LPS) of residents who rotated through a Department of Veterans Affairs health care facility for academic years 2011-2014. Predictors of PS and its association with resident satisfaction were adjusted to account for confounding and response rate biases using generalized linear models. RESULTS: The 13 044 respondents who completed the LPS (30% response rate) were comparable to nonpediatric, non-obstetrics-gynecology residents enrolled in US residency programs. Among respondents, 11 599 (89%) agreed that ". . . members of the clinical team of which I was part are able to bring up problems and tough issues." Residents were more likely to report PS if they were male, were in a less complex clinical facility, in an other medicine or psychiatry specialty, or cared for patients who were aged, had multiple illnesses, or had social supports. Nonpsychiatric residents felt safer when treating patients with no concurrent mental health diagnoses. PS was strongly associated with how residents rated their satisfaction across 4 domains of their clinical learning experience (P < .001). CONCLUSIONS: PS appears to be an important factor in resident satisfaction across 4 domains that evaluators of graduate medical education programs should consider when assessing clinical learning experiences.

Impact of work therapy on health status among homeless, substance-dependent veterans: a randomized controlled trial.

BACKGROUND: Little is known about the health outcomes of clinician-supervised, performance-based, abstinence-contingent therapeutic work programs on homeless persons with addiction disorders. This study examined the effect of the Department of Veterans Affairs compensated work therapy program (CWT) on nonvocational outcomes. With mandatory urine screenings and adherence to addiction treatment schedules, CWT provided work opportunities (wages, hours, and responsibilities) with jobs created from Veterans Affairs contracts competitively obtained from private industry. METHODS: Homeless, substance-dependent veterans (N = 142) from 4 Department of Veterans Affairs medical centers were randomized, assessed at baseline, and reassessed at 3-month intervals for 1 year. Both CWT and control groups had access to comprehensive rehabilitation, addictions, psychiatric, and medical services. Data were analyzed to determine an immediate CWT effect after treatment and rates of change during 1 year. RESULTS: Compared with control subjects, patients in the CWT program were more likely to (1) initiate outpatient addictions treatment, (2) experience fewer drug and alcohol problems, (3) report fewer physical symptoms related to substance use, (4) avoid further loss of physical functioning, and (5) have fewer episodes of homelessness and incarceration. No effect on psychiatric outcomes was found. CONCLUSION: Work therapy can enhance nonvocational outcomes of addiction treatment for homeless persons, although long-term gains remain unknown.

Studying the effects of ACGME duty hours limits on resident satisfaction: results from VA learners' perceptions survey.

BACKGROUND: As the Accreditation Council on Graduate Medical Education (ACGME) deliberates over further limiting duty hours of graduate medical education (GME) trainees, few large-scale studies have shown residents to be satisfied with the effect the 2003 standards have had on clinical care, education outcomes, or working environments. This study measures the effect of the 2003 duty hours limits on resident-reported satisfaction with GME training during their rotations through the Department of Veterans Affairs (VA) medical centers from 2001 through 2007. METHOD: Self-reported satisfaction with clinical care and education environments were assessed by comparing responses to VA's annual Learners' Perceptions Survey administered before 2003 with responses administered after 2003. To measure duty hours effects on satisfaction, before-after differences were adjusted for covariate biases modeled after an exhaustive covariate search with 10-fold cross-validation. Because nonteaching controls are not available in satisfaction studies, we used a robust differencing variable technique to control before-after differences for trend biases in the simultaneous presence of missing data and possible model misspecification. RESULTS: There were 19,605 responders. Adjusting for covariate and trend biases, after the 2003 ACGME standards, 25% more residents in medicine specialties reported satisfaction with VA clinical environment and 11% more with VA preceptors and faculty. For surgery, 33% more residents reported satisfaction with VA clinical environment and 12% more with VA preceptors and faculty. Satisfaction with working environment was mixed. CONCLUSIONS: The 2003 ACGME duty hours standards were associated with improved satisfaction for resident clinical training and learning environments.

Measuring the intensity of resident supervision in the department of veterans affairs: the resident supervision index.

PURPOSE: To develop a survey instrument designed to quantify supervision by attending physicians in nonprocedural care and to assess the instrument's feasibility and reliability. METHOD: In 2008, the Department of Veterans Affairs (VA) Office of Academic Affiliations convened an expert panel to adopt a working definition of attending supervision in nonprocedural patient care and to construct a survey to quantify it. Feasibility was field-tested on residents and their supervising attending physicians at primary care internal medicine clinics at the VA Loma Linda Healthcare System in their encounters with randomly selected outpatients diagnosed with either major depressive disorder or diabetes. The authors assessed both interrater concurrent reliability and test-retest reliability. RESULTS: The expert panel adopted the VA's definition of resident supervision and developed the Resident Supervision Index (RSI) to measure supervision in terms of residents' case understanding, attending physicians' contributions to patient care through feedback to the resident, and attending physicians' time (minutes). The RSI was field-tested on 60 residents and 37 attending physicians for 148 supervision episodes from 143 patient encounters. Consent rates were 94% for residents and 97% for attending physicians; test-retest reliability intraclass correlations (ICCs) were 0.93 and 0.88, respectively. Concurrent reliability between residents' and attending physicians' reported time was an ICC of 0.69. CONCLUSIONS: The RSI is a feasible and reliable measure of resident supervision that is intended for research studies in graduate medical education focusing on education outcomes, as well as studies assessing quality of care, patient health outcomes, care costs, and clinical workload.

Measuring progressive independence with the resident supervision index: theoretical approach.

BACKGROUND: Graduate medical education is based on an on-the-job training model in which residents provide clinical care under supervision. The traditional method is to offer residents graduated levels of responsibility that will prepare them for independent practice. However, if progressive independence from supervision exceeds residents' progressive professional development, patient outcomes may be at risk. Leaders in graduate medical education have called for "optimal" supervision, yet few studies have conceptually defined what optimal supervision means and whether optimal care is theoretically compatible with progressive independence, nor have they developed a test for progressive independence. OBJECTIVE: This research develops theory and analytic models as part of the Resident Supervision Index to quantify the intensity of supervision. METHODS: We introduce an explicit set of assumptions for an ideal patient-centered theory of optimal supervision of resident-provided care. A critical assumption is that informed attending staff will use available resources to optimize patient outcomes first and foremost, with residents gaining clinical competencies by contributing to optimal care. Next, we derive mathematically the consequences of these assumptions as theoretical results. RESULTS: Under optimal supervision, (1) patient outcome is expected to be no worse than if residents were not involved, (2) supervisors will avoid undersupervising residents (when patients are at increased risk for poor outcomes) or oversupervising residents (when residents miss clinical opportunities to practice care), (3) optimal patient outcomes will be compatible with progressive independence, (4) progressive development can be inferred from progressive independence whenever residents contribute to patient care, and (5) analytic models that test for progressive independence will emphasize adjusting the association between length of graduate medical education training and supervision for case complexity and clinic workload, but not patient health outcomes. CONCLUSION: An explicit theoretical framework is critical to measure scientifically progressive independence from supervision using graduate medical education data.

Measuring progressive independence with the resident supervision index: empirical approach.

BACKGROUND: A Resident Supervision Index (RSI) developed by our research team quantifies the intensity of resident supervision in graduate medical education, with the goal of testing for progressive independence. The 4-part RSI method includes a survey instrument for staff and residents (RSI Inventory), a strategy to score survey responses, a theoretical framework (patient centered optimal supervision), and a statistical model that accounts for the presence or absence of supervision and the intensity of patient care. METHODS: The RSI Inventory data came from 140 outpatient encounters involving 57 residents and 37 attending physicians during a 3-month period at a Department of Veterans Affairs outpatient clinic. Responses are scored to quantitatively measure the intensity of resident supervision across 10 levels of patient services (staff is absent, is present, participated, or provided care with or without a resident), case discussion (resident-staff interaction), and oversight (staff reviewed case, reviewed medical chart, consulted with staff, or assessed patient). Scores are analyzed by level and for patient care using a 2-part model (supervision initiated [yes or no] versus intensity once supervision was initiated). RESULTS: All resident encounters had patient care supervision, resident oversight, or both. Consistent with the progressive independence hypothesis, residents were 1.72 (P  =  .019) times more likely to be fully responsible for patient care with each additional postgraduate year. Decreasing case complexity, increasing clinic workload, and advanced nonmedical degrees among attending staff were negatively associated with supervision intensity, although associations varied by supervision level. CONCLUSIONS: These data are consistent with the progressive independence hypothesis in graduate medical education and offer empirical support for the 4-part RSI method to quantify the intensity of resident supervision for research, program evaluation, and resident assessment purposes. Before informing policy, however, more scientific research in actual teaching settings is needed to better understand the relationships among patient outcomes, clinic workload, case complexity, and graduate medical education experience in resident supervision and professional development.

The impact of nonclinical factors on care use for patients with depression: a STAR*D report.

INTRODUCTION: This article presents baseline findings that describe how nonclinical factors were associated with patient use of psychiatric and general medical care and how those relationships changed after patients enrolled in the 41-site Sequenced Treatment Alternatives to Relieve Depression study (STAR*D). AIMS: STAR*D offered adult outpatients with major depression diligently delivered, measurement-based care. To achieve full remission within a tolerable medication dose, recommendations for treatment based on routine symptom and side-effect measurements were discussed with patients by clinical research coordinators and offered to clinicians who could flexibly tailor that guidance to accommodate individual patient needs. Medications were provided gratis. Pre- and post-enrollment data came from provider records and from patient face-to-face, telephone, and computer-assisted surveys. Two-part nested mixed models assessed patient likelihood and volume of mental and general medical care services. RESULTS: Prior to enrollment, predisposing (gender, race, education, and care attitude), affordability (private insurance), and clinical factors (depressive symptoms and mental and physical functioning) were found to be important drivers of patient use of psychiatric and general medical care. After STAR*D enrollment, however, predisposing factors were less important drivers of psychiatric service use but remained important drivers of general medical care. CONCLUSIONS: Data suggest diligent, measurement-based mental health programs may reduce race, gender, and education disparities in the use of needed mental health care.

Release bias in accessing medical records in clinical trials: a STAR*D report.

Clinical trials often require subjects to sign medical record releases to allow investigators to measure treatment fidelity, off-protocol care use, and care costs. Little is known, however, if limiting samples to those willing to sign releases impacts external validity. Data came from outpatients with non-psychotic major depressive disorder who enrolled in the multisite Sequenced Treatment Alternatives to Relieve Depression (STAR*D) study. Differences between those who signed (n = 3116) and who did not sign (n = 925) releases were assessed using logistic regression and two-part, three-level log-transformed regression models, corrected for site clustering and repeated measures. Patients who released records tended to believe care was helpful, were younger, and married. However, release status had little material or consistent associations with patient health outcomes or use of care. With appropriate adjustments to data, requiring patient medical records may pose only minimal challenges to external validity in cost-outcome studies.

Voice response system to measure healthcare costs: a STAR*D report.

OBJECTIVE: To evaluate a telephone-operated, interactive voice response (IVR) system designed to collect use-of-care data from patients with major depression (UAC-IVR). STUDY DESIGN: Patient self-reports from repeated IVR surveys were compared with provider records for 3789 patients with major depression at 41 clinical sites participating in the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) trial. METHODS: UAC-IVR responses were examined for consistency and compared with provider records to compute reporting biases and intraclass correlation coefficients. Predictors of inconsistent responses and reporting biases were based on mixed logistic and regression models adjusted for need and predisposing and enabling covariates, and corrected for nesting and repeated measures. RESULTS: Inconsistent responses were found for 10% of calls and 21% of patients. Underreporting biases (-20%) and moderate agreement (intraclass correlation of 68%) were found when UAC-IVR responses were compared with medical records. IVR reporting biases were less for patients after 3 calls or more (experience), for patients with severe baseline symptoms (motivation), and for patients who gave consistent IVR responses (reliability). Bias was unrelated to treatment outcomes or demographic factors. CONCLUSION: Clinical managers should use IVR systems to collect service histories only after patients are properly trained and responses monitored for consistency and reporting biases.

Factors determining medical students' and residents' satisfaction during VA-based training: findings from the VA Learners' Perceptions Survey.

PURPOSE: To compare medical students' and physician residents' satisfaction with Veterans Affairs (VA) training to determine the factors that were most strongly associated with trainee satisfaction ratings. METHOD: Each year from 2001 to 2006, all medical students and residents in VA teaching facilities were invited to complete the Learners' Perceptions Survey. Participants rated their overall training satisfaction on a 100-point scale and ranked specific satisfaction in four separate educational domains (learning environment, clinical faculty, working environment, and physical environment) on a five-point Likert scale. Each domain was composed of unique items. RESULTS: A total of 6,527 medical students and 16,583 physician residents responded to the survey. The overall training satisfaction scores for medical students and physician residents were 84 and 79, respectively (P < .001), with significant differences in satisfaction reported across the training continuum. For both medical students and residents, the rating of each of the four educational domains was statistically significantly associated with the overall training satisfaction score (P < .001). The learning environment domain had the strongest association with overall training satisfaction score, followed by the clinical preceptor, working environment, and physical environment domains; no significant differences were found between medical students and physician residents in the rank order. Satisfaction with quality of care and faculty teaching contributed significantly to training satisfaction. CONCLUSIONS: Factors that influence training satisfaction were similar for residents and medical students. The domain with the highest association was the learning environment; quality of care was a key item within this domain.