© 2021, The Author(s). Background: Accurately quantifying the risk of osteoporotic fracture is important for directing appropriate clinical interventions. While skeletal measures such as heel quantitative speed of sound (SOS) and dual-energy X-ray absorptiometry bone mineral density are able to predict the risk of osteoporotic fracture, the utility of such measurements is subject to the availability of equipment and human resources. Using data from 341,449 individuals of white British ancestry, we previously developed a genome-wide polygenic risk score (PRS), called gSOS, that captured 25.0% of the total variance in SOS. Here, we test whether gSOS can improve fracture risk prediction. Methods: We examined the predictive power of gSOS in five genome-wide genotyped cohorts, including 90,172 individuals of European ancestry and 25,034 individuals of Asian ancestry. We calculated gSOS for each individual and tested for the association between gSOS and incident major osteoporotic fracture and hip fracture. We tested whether adding gSOS to the risk prediction models had added value over models using other commonly used clinical risk factors. Results: A standard deviation decrease in gSOS was associated with an increased odds of incident major osteoporotic fracture in populations of European ancestry, with odds ratios ranging from 1.35 to 1.46 in four cohorts. It was also associated with a 1.26-fold (95% confidence interval (CI) 1.13–1.41) increased odds of incident major osteoporotic fracture in the Asian population. We demonstrated that gSOS was more predictive of incident major osteoporotic fracture (area under the receiver operating characteristic curve (AUROC) = 0.734; 95% CI 0.727–0.740) and incident hip fracture (AUROC = 0.798; 95% CI 0.791–0.805) than most traditional clinical risk factors, including prior fracture, use of corticosteroids, rheumatoid arthritis, and smoking. We also showed that adding gSOS to the Fracture Risk Assessment Tool (FRAX) could refine the risk prediction with a positive net reclassification index ranging from 0.024 to 0.072. Conclusions: We generated and validated a PRS for SOS which was associated with the risk of fracture. This score was more strongly associated with the risk of fracture than many clinical risk factors and provided an improvement in risk prediction. gSOS should be explored as a tool to improve risk stratification to identify individuals at high risk of fracture.