Generalizability of deep learning models for dental image analysis

Abstract

We assessed the generalizability of deep learning models and how to improve it. Our exemplary use-case was the detection of apical lesions on panoramic radiographs. We employed two datasets of panoramic radiographs from two centers, one in Germany (Charite, Berlin, n=650) and one in India (KGMU, Lucknow, n=650): First, U-Net type models were trained on images from Charite (n=500) and assessed on test sets from Charite and KGMU (each n=150). Second, the relevance of image characteristics was explored using pixel-value transformations, aligning the image characteristics in the datasets. Third, cross-center training effects on generalizability were evaluated by stepwise replacing Charite with KGMU images. Last, we assessed the impact of the dental status (presence of root-canal fillings or restorations). Models trained only on Charite images showed a (mean +/- SD) F1-score of 54.1 +/- 0.8% on Charite and 32.7 +/- 0.8% on KGMU data (p<0.001/t-test). Alignment of image data characteristics between the centers did not improve generalizability. However, by gradually increasing the fraction of KGMU images in the training set (from 0 to 100%) the F1-score on KGMU images improved (46.1 +/- 0.9%) at a moderate decrease on Charite images (50.9 +/- 0.9%, p<0.01). Model performance was good on KGMU images showing root-canal fillings and/or restorations, but much lower on KGMU images without root-canal fillings and/or restorations. Our deep learning models were not generalizable across centers. Cross-center training improved generalizability. Noteworthy, the dental status, but not image characteristics were relevant. Understanding the reasons behind limits in generalizability helps to mitigate generalizability problems.