In this study, Dr. Julia Owen and her co-authors developed a novel method for automated detection of abnormal optical coherence tomography (OCT) B-scan ("B-scan of interest") images of the retina. Commercial OCT devices do not routinely provide automated diagnoses, so disease detection requires expert interpretation of the images by an ophthalmologist. Reliable automated detection of potentially abnormal retinal scans could save time, as the machine could flag abnormal scans that require further review by a clinician. But there have been challenges with developing such models, including the lack of large training datasets, standardized methods for image acquisition and processing, agreed-upon evaluation metrics, and limitations in computing power.
Dr. Owne developed an approach that overcomes some of these limitations, by using an efficient method for training lightweight machine learning models, or models with fewer parameters and operations, using unlabeled and routinely-available data. Lightweight models are useful in clinical settings because they can perform quickly and are more compatible with portable devices.
Lightweight models can require specialized approaches to achieve comparable accuracy to large, state-of-the-art models. In this study, unlabeled images were used to distill knowledge from the larger "teacher" network to the lightweight student networks through soft-target learning. First, the pre-trained "teacher" model was used to create labels for unlabeled images. These were then pooled with expert-labeled images and used to train the lightweight networks. The authors varied the relative amount of unlabeled images available for the student-teacher training and compared the resulting performance.
Inference time in seconds is plotted against the maximum validation accuracy across epochs and five repeated training runs. The model family is depicted by the color of the marker with the ResNet50 shown in gray. The size of the marker is scaled by the number of parameters in each model. The lightweight models are abbreviated to SqueezeNet (SN), SqueeseResNet (SRN), MobileNet (M), Fast MobileNet (FM), SqueezeNext (SQN), and ShuffleNet (SFN), see Supplementary Table 1 for more details on abbreviations used.
The authors found that four general families of LWMs were successful at perform B-scan of interest identification using the student-teacher framework. All of the lightweight student networks had improved validation accuracy with the addition of teacher-generated labeled data, and several models were able to perform comparably to the state-of-the-art, large teacher model while running in a fraction of the time. This ability makes the lightweight models ideal candidates to run in clinical settings where runtime is important and computing resources are often limited. The student-teacher framework is also appealing because it successfully takes advantage of automated labeling by the teacher machine learning model, so there is less need for clinicians to label images by hand, a time consuming process.
Owen JP, Blazes M, Manivannan N, Lee GC, Yu S, Durbin MK, Nair A, Singh RP, Talcott KE, Melo AG, Greenlee T, Chen ER, Conti TF, Lee CS, Lee AY. Student becomes teacher: training faster deep learning lightweight networks for automated identification of optical coherence tomography B-scans of interest using a student-teacher framework. Biomedical Optics Express. 2021. Vol. 12, Issue 9, pp. 5387-5399. doi.org/10.1364/BOE.433432