Study 1, Ophthalmology Science, 2024: Can the model tell a small melanoma from a harmless lesion?
A retrospective multicentre study of 802 photographs from 688 patients, all examined at St. Erik Eye Hospital. The model was built on one part of them and tested on two sets it had never seen: 86 photographs held back from the same collection, and 53 taken at other clinics across Sweden before the patients came to St. Erik. Twelve ophthalmologists, from residents to ocular oncologists, read the same 86 photographs without knowing the answers.
- The model, from the photograph alone
- Checklists, also scored with ultrasound or OCT
Internal test set86 photographs, held back from the same collection
Both checklists were scored by ophthalmologists who also had ultrasound or OCT from the records. The model had the photograph only.
External test set53 photographs, other clinics in Sweden
The ophthalmologists read the internal set only, and the checklists were compared with the model on the internal set only.
AUC
| Test set | Method | AUC | 95 % confidence interval |
|---|---|---|---|
| Internal test set, 86 photographs, held back from the same collection | The model, from the photograph alone | 0.88 | 0.82 to 0.95 |
| Internal test set, 86 photographs, held back from the same collection | MOLES, Checklists, also scored with ultrasound or OCT | 0.77 | 0.66 to 0.88 |
| Internal test set, 86 photographs, held back from the same collection | TFSOM-UHHD, Checklists, also scored with ultrasound or OCT | 0.67 | 0.54 to 0.81 |
| Internal test set, 86 photographs, held back from the same collection | Both checklists were scored by ophthalmologists who also had ultrasound or OCT from the records. The model had the photograph only. | ||
| External test set, 53 photographs, other clinics in Sweden | The model, from the photograph alone | 0.88 | 0.74 to 1.00 |
| External test set, 53 photographs, other clinics in Sweden | The ophthalmologists read the internal set only, and the checklists were compared with the model on the internal set only. | ||
At the 0.63 threshold, set before testing, the model had 100 % sensitivity and 74 % specificity on the internal set, and 80 % sensitivity and 81 % specificity on the external set. On the internal set, the one the twelve ophthalmologists also read, its sensitivity was significantly higher than the residents' and the consultants' and not significantly different from the ocular oncologists'; its specificity did not differ significantly from any group's. Ophthalmology Science, 2024
The authors' conclusion
“This study develops and validates a deep learning algorithm for differentiating small choroidal melanomas from nevi, matching or surpassing the discriminatory performance of experienced human ophthalmologists.”
The authors' caveat
“Further research will aim to validate its utility in clinical settings.”
- Sensitivity
- The share of melanomas correctly classed as melanoma.
- Specificity
- The share of harmless lesions correctly classed as harmless.
| Reader | Sensitivity | Specificity | Dunn P against the model, sensitivity and specificity |
|---|---|---|---|
| Model | 100 % | 74 % | |
| Resident ophthalmologists, 6 | 85 % | 63 % | 0.04 and 0.08 |
| Consultant ophthalmologists, 3 | 83 % | 78 % | 0.006 and 0.36 |
| Ocular oncologists, 3 | 98 % | 70 % | > 0.99 and > 0.99 |
Dunn post hoc tests with Bonferroni correction. Across the four groups, sensitivity differed (Kruskal-Wallis P < 0.001) and specificity did not (P = 0.26). All twelve ophthalmologists pooled against the model (Mann-Whitney U): sensitivity P = 0.006, specificity P = 0.54.
| Method | Scored from | AUC (95 % CI) | Against the model |
|---|---|---|---|
| Model | The photograph | 0.88 (0.82 to 0.95) | |
| MOLES | The photograph plus ultrasound and/or OCT from the records | 0.77 (0.66 to 0.88) | DeLong P < 0.001 |
| TFSOM-UHHD | The photograph plus ultrasound and/or OCT from the records | 0.67 (0.54 to 0.81) | DeLong P < 0.001 |
| Camera | Photographs | AUC (95 % CI) | Sensitivity | Specificity |
|---|---|---|---|---|
| Ultra-widefield, Optos, 200° | 31 | 0.89 (0.76 to 1.00) | 100 % | 62 % |
| Standard field, Canon, 45° | 55 | 0.89 (0.81 to 0.97) | 100 % | 77 % |
The external set gave AUC 0.88 (0.74 to 1.00), with 80 % sensitivity and 81 % specificity. The ophthalmologists did not read it.
- Readers
- Six resident ophthalmologists, three consultant ophthalmologists and three ocular oncologists, blinded to diagnosis and follow-up. The three oncologists “account for over half of the country's ocular oncologists”, the authors note, adding that “caution is warranted when generalizing their assessments”.
- MOLES
- Five signs, each scored 0, 1 or 2: Mushroom shape, Orange pigment, Large size, Enlarging tumour and Subretinal fluid. A total above 2 reads as probable melanoma.
- TFSOM-UHHD
- Risk factors for growth, whose capital letters spell the name: Thickness over 2 mm, subretinal Fluid, Symptoms, Orange pigment, a Margin within 3 mm of the optic disc, Ultrasonographic Hollowness, and the absence of a Halo and of Drusen.
- What each had
- The checklists were scored by ophthalmologists who also had ultrasound and/or OCT from the records. The twelve readers and the model had the photograph only.



