Hybrid YOLO11 Framework for Automated Pill Detection and Identification using Deep Learning and OCR

Abstract

Medication errors caused by incorrect pill identification remain a significant concern in healthcare and can adversely affect patient safety. Although existing automated pill recognition methods utilize visual characteristics such as color, shape, size, and imprint, their performance often decreases when images are captured under varying lighting conditions, with partial occlusions, or when pill markings are damaged. This paper presents a hybrid framework for automated pill detection and identification by integrating the YOLO11 object detection model with Optical Character Recognition (OCR) and multi-feature analysis. In the proposed approach, YOLO11 is employed to accurately detect and localize pills from input images, while OCR extracts the imprint information present on the pill surface. The extracted imprint is combined with visual features, including color, shape, size, and texture, to improve the reliability of pill identification. A confidence-based matching module compares these features with a pill database containing medication names, dosage details, therapeutic uses, manufacturer information, and safety guidelines to determine the final prediction. In addition, data augmentation techniques are applied during training to improve the model's ability to handle variations in image quality and environmental conditions. The proposed framework is evaluated using standard performance metrics such as precision, recall, F1-score, Intersection over Union (IoU), mean Average Precision (mAP), inference time, and overall accuracy. Its performance is compared with several widely used deep learning models, including CNN, SSD, Faster R-CNN, YOLOv5, YOLOv8, and YOLOv10. The experimental analysis demonstrates that the proposed framework achieves more accurate localization, improved identification performance, and reduced inference time, making it suitable for deployment in hospitals, pharmacies, and other healthcare environments where reliable medication identification is essential.