• Construct validation of judgement-based assessments of medical trainees' competency in the workplace using a "Kanesian" approach to validation.

    29 December 2017

    BACKGROUND: Evaluations of clinical assessments that use judgement-based methods have frequently shown them to have sub-optimal reliability and internal validity evidence for their interpretation and intended use. The aim of this study was to enhance that validity evidence by an evaluation of the internal validity and reliability of competency constructs from supervisors' end-of-term summative assessments for prevocational medical trainees. METHODS: The populations were medical trainees preparing for full registration as a medical practitioner (74) and supervisors who undertook ≥2 end-of-term summative assessments (n = 349) from a single institution. Confirmatory Factor Analysis was used to evaluate assessment internal construct validity. The hypothesised competency construct model to be tested, identified by exploratory factor analysis, had a theoretical basis established in workplace-psychology literature. Comparisons were made with competing models of potential competency constructs including the competency construct model of the original assessment. The optimal model for the competency constructs was identified using model fit and measurement invariance analysis. Construct homogeneity was assessed by Cronbach's α. Reliability measures were variance components of individual competency items and the identified competency constructs, and the number of assessments needed to achieve adequate reliability of R > 0.80. RESULTS: The hypothesised competency constructs of "general professional job performance", "clinical skills" and "professional abilities" provides a good model-fit to the data, and a better fit than all alternative models. Model fit indices were χ2/df = 2.8; RMSEA = 0.073 (CI 0.057-0.088); CFI = 0.93; TLI = 0.95; SRMR = 0.039; WRMR = 0.93; AIC = 3879; and BIC = 4018). The optimal model had adequate measurement invariance with nested analysis of important population subgroups supporting the presence of full metric invariance. Reliability estimates for the competency construct "general professional job performance" indicated a resource efficient and reliable assessment for such a construct (6 assessments for an R > 0.80). Item homogeneity was good (Cronbach's alpha = 0.899). Other competency constructs are resource intensive requiring ≥11 assessments for a reliable assessment score. CONCLUSION: Internal validity and reliability of clinical competence assessments using judgement-based methods are acceptable when actual competency constructs used by assessors are adequately identified. Validation for interpretation and use of supervisors' assessment in local training schemes is feasible using standard methods for gathering validity evidence.

  • Prognostic models in acute pulmonary embolism: a systematic review and meta-analysis.

    9 January 2018

    OBJECTIVE: To review the evidence for existing prognostic models in acute pulmonary embolism (PE) and determine how valid and useful they are for predicting patient outcomes. DESIGN: Systematic review and meta-analysis. DATA SOURCES: OVID MEDLINE and EMBASE, and The Cochrane Library from inception to July 2014, and sources of grey literature. ELIGIBILITY CRITERIA: Studies aiming at constructing, validating, updating or studying the impact of prognostic models to predict all-cause death, PE-related death or venous thromboembolic events up to a 3-month follow-up in patients with an acute symptomatic PE. DATA EXTRACTION: Study characteristics and study quality using prognostic criteria. Studies were selected and data extracted by 2 reviewers. DATA ANALYSIS: Summary estimates (95% CI) for proportion of risk groups and event rates within risk groups, and accuracy. RESULTS: We included 71 studies (44,298 patients). Among them, 17 were model construction studies specific to PE prognosis. The most validated models were the PE Severity Index (PESI) and its simplified version (sPESI). The overall 30-day mortality rate was 2.3% (1.7% to 2.9%) in the low-risk group and 11.4% (9.9% to 13.1%) in the high-risk group for PESI (9 studies), and 1.5% (0.9% to 2.5%) in the low-risk group and 10.7% (8.8% to12.9%) in the high-risk group for sPESI (11 studies). PESI has proved clinically useful in an impact study. Shifting the cut-off or using novel and updated models specifically developed for normotensive PE improves the ability for identifying patients at lower risk for early death or adverse outcome (0.5-1%) and those at higher risk (up to 20-29% of event rate). CONCLUSIONS: We provide evidence-based information about the validity and utility of the existing prognostic models in acute PE that may be helpful for identifying patients at low risk. Novel models seem attractive for the high-risk normotensive PE but need to be externally validated then be assessed in impact studies.

  • Poisoned by oxalic acid.

    27 December 2017

  • International Subarachnoid Aneurysm Trial (ISAT) of neurosurgical clipping versus endovascular coiling in 2143 patients with ruptured intracranial aneurysms: a randomised trial.

    18 January 2018

    BACKGROUND: Endovascular detachable coil treatment is being increasingly used as an alternative to craniotomy and clipping for some ruptured intracranial aneurysms, although the relative benefits of these two approaches have yet to be established. We undertook a randomised, multicentre trial to compare the safety and efficacy of endovascular coiling with standard neurosurgical clipping for such aneurysms judged to be suitable for both treatments. METHODS: We enrolled 2143 patients with ruptured intracranial aneurysms and randomly assigned them to neurosurgical clipping (n=1070) or endovascular treatment by detachable platinum coils (n=1073). Clinical outcomes were assessed at 2 months and at 1 year with interim ascertainment of rebleeds and death. The primary outcome was the proportion of patients with a modified Rankin scale score of 3-6 (dependency or death) at 1 year. Trial recruitment was stopped by the steering committee after a planned interim analysis. Analysis was per protocol. FINDINGS: 190 of 801 (23.7%) patients allocated endovascular treatment were dependent or dead at 1 year compared with 243 of 793 (30.6%) allocated neurosurgical treatment (p=0.0019). The relative and absolute risk reductions in dependency or death after allocation to an endovascular versus neurosurgical treatment were 22.6% (95% CI 8.9-34.2) and 6.9% (2.5-11.3), respectively. The risk of rebleeding from the ruptured aneurysm after 1 year was two per 1276 and zero per 1081 patient-years for patients allocated endovascular and neurosurgical treatment, respectively. INTERPRETATION: In patients with a ruptured intracranial aneurysm, for which endovascular coiling and neurosurgical clipping are therapeutic options, the outcome in terms of survival free of disability at 1 year is significantly better with endovascular coiling. The data available to date suggest that the long-term risks of further bleeding from the treated aneurysm are low with either therapy, although somewhat more frequent with endovascular coiling.