| Knowledge distillation has emerged as a key technique for transferring knowledge from large, high-capacity neural networks to compact student models, enabling efficient deployment in resource-constrained environments while maintaining strong predictive performance. Despite its widespread adoption, most existing distillation approaches primarily focus on improving classification accuracy, often overlooking the modeling and preservation of predictive uncertainty, which is critical for robust and trustworthy decision-making models. In parallel, uncertainty estimation has become an essential component of modern deep learning systems, especially for out-of-distribution (OOD) detection and the mitigation of overconfident predictions. However, integrating uncertainty-aware mechanisms into knowledge distillation remains challenging, since compression reduces model capacity and alters the induced predictive distribution. In this work, we propose a bidirectional distillation framework that combines ensemble teachers with lightweight students, and uses the student’s inherent predictive uncertainty to adaptively soften the teacher’s targets. This uncertainty-guided softening yields smoother optimization updates, mitigating feature collapse and improving OOD detection. We evaluate the proposed approach on standard image classification benchmarks, including CIFAR-10 and a Human-Detection dataset, assessing both in-distribution accuracy and out-of-distribution detection. Experimental results demonstrate consistent improvements in OOD detection performance while maintaining competitive in-distribution accuracy, highlighting the importance of uncertainty-aware distillation for reliable deep learning systems. |
*** Title, author list and abstract as submitted during Camera-Ready version delivery. Small changes that may have occurred during processing by Springer may not appear in this window.