Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

BACKGROUND: To critically appraise the development and measurement properties, including sensibility, reliability, and validity of the Non-Technical Skills of Surgeons (NOTSS) system. DATA SOURCES: Articles that described development process of the NOTSS system were identified. Relevant primary studies that presented evidence of reliability and validity were identified through a comprehensive literature review. RESULTS: NOTSS was developed through robust item generation and reduction strategies. It was shown to have good content validity, acceptability, and feasibility. Inter-rater reliability increased with greater expertise and number of assessors. Studies demonstrated evidence of cross-sectional construct validity, in that the tool was able to differentiate known groups of varied non-technical skill levels. Evidence of longitudinal construct validity also existed to demonstrate that NOTSS detected changes in non-technical skills before and after targeted training. CONCLUSION: In populations and settings presented in our critical appraisal, NOTSS provided reliable and valid measurements of intraoperative non-technical skills of surgeons.

Original publication




Journal article


Am J Surg

Publication Date





990 - 997


Critical appraisal, Measurement property, Non-technical skill, Structured feedback, Surgeon evaluation, Educational Measurement, Humans, Reproducibility of Results, Surgeons