Construct validation of judgement-based assessments of medical trainees' competency in the workplace using a "Kanesian" approach to validation.

BACKGROUND: Evaluations of clinical assessments that use judgement-based methods have frequently shown them to have sub-optimal reliability and internal validity evidence for their interpretation and intended use. The aim of this study was to enhance that validity evidence by an evaluation of the internal validity and reliability of competency constructs from supervisors' end-of-term summative assessments for prevocational medical trainees. METHODS: The populations were medical trainees preparing for full registration as a medical practitioner (74) and supervisors who undertook ≥2 end-of-term summative assessments (n = 349) from a single institution. Confirmatory Factor Analysis was used to evaluate assessment internal construct validity. The hypothesised competency construct model to be tested, identified by exploratory factor analysis, had a theoretical basis established in workplace-psychology literature. Comparisons were made with competing models of potential competency constructs including the competency construct model of the original assessment. The optimal model for the competency constructs was identified using model fit and measurement invariance analysis. Construct homogeneity was assessed by Cronbach's α. Reliability measures were variance components of individual competency items and the identified competency constructs, and the number of assessments needed to achieve adequate reliability of R > 0.80. RESULTS: The hypothesised competency constructs of "general professional job performance", "clinical skills" and "professional abilities" provides a good model-fit to the data, and a better fit than all alternative models. Model fit indices were χ2/df = 2.8; RMSEA = 0.073 (CI 0.057-0.088); CFI = 0.93; TLI = 0.95; SRMR = 0.039; WRMR = 0.93; AIC = 3879; and BIC = 4018). The optimal model had adequate measurement invariance with nested analysis of important population subgroups supporting the presence of full metric invariance. Reliability estimates for the competency construct "general professional job performance" indicated a resource efficient and reliable assessment for such a construct (6 assessments for an R > 0.80). Item homogeneity was good (Cronbach's alpha = 0.899). Other competency constructs are resource intensive requiring ≥11 assessments for a reliable assessment score. CONCLUSION: Internal validity and reliability of clinical competence assessments using judgement-based methods are acceptable when actual competency constructs used by assessors are adequately identified. Validation for interpretation and use of supervisors' assessment in local training schemes is feasible using standard methods for gathering validity evidence.

A critical evaluation of the validity and the reliability of global competency constructs for supervisor assessment of junior medical trainees.

Supervisor assessments are critical for both formative and summative assessment in the workplace. Supervisor ratings remain an important source of such assessment in many educational jurisdictions even though there is ambiguity about their validity and reliability. The aims of this evaluation is to explore the: (1) construct validity of ward-based supervisor competency assessments; (2) reliability of supervisors for observing any overarching domain constructs identified (factors); (3) stability of factors across subgroups of contexts, supervisors and trainees; and (4) position of the observations compared to the established literature. Evaluated assessments were all those used to judge intern (trainee) suitability to become an unconditionally registered medical practitioner in the Australian Capital Territory, Australia in 2007-2008. Initial construct identification is by traditional exploratory factor analysis (EFA) using Principal component analysis with Varimax rotation. Factor stability is explored by EFA of subgroups by different contexts such as hospital type, and different types of supervisors and trainees. The unit of analysis is each assessment, and includes all available assessments without aggregation of any scores to obtain the factors. Reliability of identified constructs is by variance components analysis of the summed trainee scores for each factor and the number of assessments needed to provide an acceptably reliable assessment using the construct, the reliability unit of analysis being the score for each factor for every assessment. For the 374 assessments from 74 trainees and 73 supervisors, the EFA resulted in 3 factors identified from the scree plot, accounting for only 68 % of the variance with factor 1 having features of a "general professional job performance" competency (eigenvalue 7.630; variance 54.5 %); factor 2 "clinical skills" (eigenvalue 1.036; variance 7.4 %); and factor 3 "professional and personal" competency (eigenvalue 0.867; variance 6.2 %). The percent trainee score variance for the summed competency item scores for factors 1, 2 and 3 were 40.4, 27.4 and 22.9 % respectively. The number of assessments needed to give a reliability coefficient of 0.80 was 6, 11 and 13 respectively. The factor structure remained stable for subgroups of female trainees, Australian graduate trainees, the central hospital, surgeons, staff specialist, visiting medical officers and the separation into single years. Physicians as supervisors, male trainees, and male supervisors all had a different grouping of items within 3 factors which all had competency items that collapsed into the predefined "face value" constructs of competence. These observations add new insights compared to the established literature. For the setting, most supervisors appear to be assessing a dominant construct domain which is similar to a general professional job performance competency. This global construct consists of individual competency items that supervisors spontaneously align and has acceptable assessment reliability. However, factor structure instability between different populations of supervisors and trainees means that subpopulations of trainees may be assessed differently and that some subpopulations of supervisors are assessing the same trainees with different constructs than other supervisors. The lack of competency criterion standardisation of supervisors' assessments brings into question the validity of this assessment method as currently used.

Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review.

Even though rater-based judgements of clinical competence are widely used, they are context sensitive and vary between individuals and institutions. To deal adequately with rater-judgement unreliability, evaluating the reliability of workplace rater-based assessments in the local context is essential. Using such an approach, the primary intention of this study was to identify the trainee score variation around supervisor ratings, identify sampling number needs of workplace assessments for certification of competence and position the findings within the known literature. This reliability study of workplace-based supervisors' assessments of trainees has a rater-nested-within-trainee design. Score variation attributable to the trainee for each competency item assessed (variance component) were estimated by the minimum-norm quadratic unbiased estimator. Score variance was used to estimate the number needed for a reliability value of 0.80. The trainee score variance for each of 14 competency items varied between 2.3% for emergency skills to 35.6% for communication skills, with an average for all competency items of 20.3%; the "Overall rating" competency item trainee variance was 28.8%. These variance components translated into 169, 7, 17 and 28 assessments needed for a reliability of 0.80, respectively. Most variation in assessment scores was due to measurement error, ranging from 97.7% for emergency skills to 63.4% for communication skills. Similar results have been demonstrated in previously published studies. In summary, overall supervisors' workplace based assessments have poor reliability and are not suitable for use in certification processes in their current form. The marked variation in the supervisors' reliability in assessing different competencies indicates that supervisors may be able to assess some with acceptable reproducibility; in this case communication and possibly overall competence. However, any continued use of this format for assessment of trainee competencies necessitates the identification of what supervisors in different institutions can reliably assess rather than continuing to impose false expectations from unreliable assessments.

Prognostic efficacy of cardiac biomarkers for mortality in dialysis patients.

BACKGROUND: The high prevalence of cardiovascular mortality in the end-stage renal disease population is well established. The aim of this current study was to document the relative prognostic significance of established cardiac biomarkers troponin T (TnT), troponin I (TnI), B-type natriuretic peptide (BNP) and N-terminal pro-BNP (NT-pro-BNP) in this population. METHODS: A prospective cohort study of dialysis patients undertaken in a single tertiary centre in Australia. Relevant clinical and biochemical information was collected at entry and all patients followed up prospectively without any loss to follow up. End-point of interest was all-cause mortality. Statistical analysis using Cox proportional hazards was used to study relationship between competing covariates and outcome. A total of 143 patients with a mean age of 59.67 +/- 15.49 years was followed up for a median duration of 30 months. Of these patients, 89.3% were white Australians of European ancestry. Twenty-seven per cent had an established diagnosis of diabetes mellitus. The mean concentrations (+/-SD) of TnT, TnI, BNP and N-terminal peptide pro-BNP (NT-pro-BNP) were 0.08 +/- 0.04 microg/L, 0.09 +/- 0.2 microg/L, 270 +/- 117 ng/L and 1434 +/- 591 ng/L respectively. RESULTS: Twenty-eight subjects died during the period of follow up. By univariate analysis, all cardiac markers (TnT, TnI, BNP, NT-pro-BNP and C-reactive protein) were significantly associated with an increase in mortality. On Cox proportionate hazards analysis, only albumin and NT-pro-BNP showed a significant association with mortality, with hazard ratios of 0.834, 95% confidence interval (CI) 0.779-0.893, P < 0.001, and 1.585, 95%CI 1.160-20165, P = 0.004 respectively. CONCLUSION: In patients with end-stage renal failure on dialysis NT-pro-BNP provides greater prognostic information compared with TnT and TnI.

Abnormal platelet reactivity in men with premature coronary heart disease.

BACKGROUND: Continuous platelet aggregation does not occur in patients with stable coronary heart disease (CHD). However, there may be a latent potential for increased aggregation given appropriate stimuli, since increased in-vitro platelet aggregation appears to be predictive of cardiac events. This study evaluates the relationship between in-vitro platelet aggregation and coronary artery disease (CAD) as defined by angiography. METHODS: In-vitro platelet aggregation was assessed in a case-control study of 53 men with CHD younger than 50 years, and in 48 control subjects without CHD who were matched for age, sex, and socioeconomic status. All major risk factors were evaluated. Semi-quantitative composite scores of the extent of arterial wall involvement and composite scores for the severity of discrete lesions were documented from standard coronary angiography. Measures of platelet aggregation in response to adrenaline, adenosine diphoshate (ADP), and collagen included: (1) the maximum slope of the aggregation curve (rate); (2) lag time to 50% maximum aggregation (LT50%); and (3) the threshold concentration of each agonist to cause maximal aggregation. RESULTS: The CHD patients had larger platelets than patients in the control group (mean platelet volume 9.40 +/- 0.73 versus 8.88 +/- 0.87; P < 0.01). The aggregation rate was significantly faster with adrenaline in the group with CHD than patients in the control group (rate of aggregation 13.9 +/- 8.8 versus 8.6 +/- 4.0 cm/s respectively, mean +/- SD; LT50% for adrenaline 130 +/- 70 versus 230 +/- 10 s respectively, mean +/- SD). Fewer CHD patients had no aggregation response to adrenaline than in the control group (8 versus 31%, P < 0.05). Adrenaline-induced platelet aggregation, as measured by LT50% for adrenaline, weakly but significantly correlated with the number and severity of discrete obstructive coronary lesions (r = -0.28, P < 0.05). This association remained significant after multivariate regression analysis. No association was found between any measure of platelet reactivity and the extent of disease as measured by a semi-quantitative composite score; this measured the extent of disease of the coronary artery walls visible by angiography. CONCLUSIONS: These findings indicate that platelets from men with premature CHD are larger and aggregate more rapidly in response to platelet agonists, particularly adrenaline. An increased rate of aggregability with adrenaline predicted the severity of CAD lesion but not the extent of the disease, and this suggests that platelets play a role in the formation of localized obstructive lesions in coronary arteries, and therefore acute coronary events.

Relationship of blood cholesterol and apoprotein B levels to angiographically defined coronary artery disease in young males.

BACKGROUND: Coronary heart disease is mainly caused by the effects of obstruction to blood flow in the coronary arteries from discrete mural lesions that encroach into the lumen and usually occur in arteries that are involved by atherosclerosis. Even though the level of certain lipoproteins is indisputably related to the degree of this atherosclerotic involvement of the coronary arteries, the question of whether lipoproteins are also associated with the obstructive lesions remains uncertain. METHODS: This study addressed the question in 53 males (age, 44.6 +/- 4.9 y) with premature coronary heart disease and angiographically proven coronary artery disease. The cholesterol, triglyceride, high-density lipoprotein, apoprotein B, and apoprotein A-I levels were compared by linear correlation to semiquantitative angiographic measures of coronary artery disease severity (coronary stenosis score and mean coronary stenosis score), the extent of mural involvement (coronary atheromatous score and mean coronary atheromatous score), and also the number of normal coronary artery segments, an alternative severity score (Jenkins), and left ventricular function score. RESULTS: Age, past and present cholesterol, low-density lipoprotein cholesterol, and apoprotein B levels correlated directly with the extent of disease (r = 0.27, 0.46, 0.29, 0.26, 0.35, respectively, P < 0.05 with coronary atheromatous score; r = 0.29, 0.44, 0.33, 0.30, 0.35, respectively, P < 0.05 with mean coronary atheromatous score). Age and New York Heart Association angina functional class correlated directly with disease severity (r = 0.25, 0.31, respectively, P < 0.05 with coronary stenosis score; r = 0.25, 0.34, respectively, P < 0.05 with mean coronary stenosis score). There was an inverse association between age, past and present cholesterol, low-density lipoprotein cholesterol, and apoprotein B levels with the number of normal segments (r = -0.23, -0.46, -0.38, -0.35, -0.39, respectively, P < 0.05). Multiple regression analysis was undertaken with lipoproteins and age as independent variables and angiographic scores as dependent variables; the apoprotein B level was the most predictive of the extent of coronary artery disease (P < 0.02) and inversely predicted the number of normal segments (P < 0.002). Of those variables entered into the regression model, only age was independently predictive of the severity of angiographic coronary artery disease. CONCLUSION: Apoprotein B levels are not predictive of coronary artery disease severity but do predict independently the extent of involvement of coronary atherosclerosis defined angiographically.

The acute haemodynamic effects of oral nicardipine.

The haemodynamic effects of oral nicardipine at two different doses were assessed in fifteen patients at rest following diagnostic cardiac catheterisation. Six patients received four doses of 20 mg nicardipine hydrochloride orally every 8 h, and nine patients received four doses of 30 mg orally every 8 h. No side effects were encountered with either dose. At both dose levels, there was a significant fall in systemic vascular resistance, significant increases in heart rate and cardiac output, and no change in stroke volume index. No changes were seen in pulmonary vascular haemodynamics, and there were no significant changes in mean arterial pressure. Plasma levels of nicardipine reached a peak within 1 h. In parallel with the haemodynamic effects, the plasma concentrations achieved after the third and fourth doses were higher than after the first dose. These haemodynamic changes are consistent with a vasodilator effect, which produces a decrease in peripheral vascular resistance. It would appear that the 30 mg dose has a more potent vasodilator action than the 20 mg dose and, in the patients studied, this larger dose was not associated with any side effects.