In this study published in the Journal of Ophthalmology, the authors experimented with distributing the work of performing a large complex image labeling task (identifying layers of the retina on multiple optical coherence tomography images) using a popular crowdsourcing marketplace. Optical coherence tomography is an important noninvasive diagnostic tool in the field of ophthalmology, especially for the management of age-related macular degeneration, a common cause of blindness in the developed world. However, the automated measurements provided by the optical coherence tomography software result in frequent errors for critical parameters such as macular thickness and volume. Human eyes have the ability to identify and complete areas on the image where there is poor signal to noise ratio and are better at recognizing complex pathology. The authors sought to evaluate the feasibility of using Mechanical Turk as a parallel platform to allow multiple people to simultaneously and rapidly perform manual segmentations (identification of retinal layer boundaries) of the images.
The idea for many people working on small parts of a large task in parallel is based on the MapReduce framework. Recently in computer science, the MapReduce programming model has caused a paradigm shift in the way that large data sets are distributed in parallel within a computing cluster, and the framework has become popularized as a generic framework to solve big data problems in multicore cluster systems. In this study, the automated analysis pipeline (using a MapReduce framework) was able to create, submit, collect, collate, process, and analyze a total of over 92,500 data points from the 61 ocular coherence tomograpy images. Each image was segmented twice by Amazon Mechanical Turk users. It took 3 days to complete the process, with 75% of tasks finished within the first 24 hours. A total of 22 individual Mechanical Turk users provided segmentations. After all the segmentations were performed, the data was collected and image processing was performed to enhance the accuracy of the manual segmentations.
This study provides a proof of concept that Mechanical Turk provides a cost-effective, scalable, high-availability method for manual segmentation of ocular coherence tomography images that are difficult for automated algorithms to handle. The authors note that it will be important to compare the the accuracy of the segmentation performed by Mechanical Turk users to the ones performed by trained experts, but previous studies using crowdsourcing for similar medical applications have demonstrated high level of diagnostic accuracy. Rapid, reliable, cost-effective methods of interpreting large data will be crucial in the future, and this approach may prove useful for the analysis of high volumes of images in clinical studies or for training of future automated segmentation algorithms.
Lee AY, Lee CS, Keane PA, Tufail A. Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation. J Ophthalmol. 2016;2016:6571547. doi: 10.1155/2016/6571547. Epub 2016 May 11. PubMed PMID: 27293877; PubMed Central PMCID: PMC4879255.