Lossless Float16 Quantization for Multi-Class Skin Lesion Classification: Lesion-Level Stratified Evaluation on ISIC 2019

Abstract

Image-level random data partitioning, which continues to plague the field through intra-lesion data leakage training and test sets may both contain multiple images of the same lesion is still the dominant methodological limitation in dermoscopic AI research that produces falsely high but unacceptable metrics. However, the impacts of this leakage on Post-Training Quantization (PTQ) benchmarking have not yet been investigated. This work proposes a leakage-free, precise lesion-level stratified evaluation framework for multi-class skin lesion classifier evaluating PTQ. A total of 23,247 images from eight diagnostic classes of ISIC 2019 benchmark were divided based on 11,847 unique lesions. Our two-stage EfficientNetB0 transfer learning pipeline with the focal loss which can down weight easy-to-classify samples achieves 86.98% macro-AUC and 84.05% top-2 accuracy. Then, utilizing Float16 PTQ for further model size reduction (49.6% compression ratio all models performance degradation is not statistically significant), a lossless compression through vigorous lesion-level evaluation is demonstrated. Variance analysis on three independent splits shows that data partitioning alone can explain up to ±4.28% top-1 accuracy variability, as opposed to quantization precision which matter barely less.