International Journal of Innovative Research in Computer and Communication Engineering
ISSN Approved Journal | Impact factor: 8.771 | ESTD: 2013 | Follows UGC CARE Journal Norms and Guidelines
| Monthly, Peer-Reviewed, Refereed, Scholarly, Multidisciplinary and Open Access Journal | High Impact Factor 8.771 (Calculated by Google Scholar and Semantic Scholar | AI-Powered Research Tool | Indexing in all Major Database & Metadata, Citation Generator | Digital Object Identifier (DOI) |
| TITLE | YOLO-Based GUI Element Detection for User Interaction Analysis |
|---|---|
| ABSTRACT | Understanding how users interact with software interfaces increasingly depends on the ability to automatically locate and classify the on-screen elements - buttons, text inputs, checkboxes, sliders, dropdowns and menu items - that a user can act upon. Manual annotation of graphical user interface (GUI) elements does not scale to the pace at which modern applications change, motivating an automated, vision-based detection approach. This paper applies a You Only Look Once (YOLO) single-stage object detector to the task of GUI element detection, framing each screenshot as an object-detection problem in which every interactive widget is localized with a bounding box and assigned an element-type label. The detector is trained on a custom UI-element dataset spanning thirteen widget categories, and is evaluated using standard detection metrics - precision, recall, mean Average Precision at IoU 0.5 ([email protected]) and across IoU thresholds 0.5:0.95 ([email protected]:0.95) - together with per-image inference latency. Training logs show the box, classification and distribution-focal losses converging steadily, while validation results reveal a strong class-imbalance effect: frequent classes such as button and link are detected with useful precision and mAP, whereas rare classes with only a handful of annotated instances (checkbox, slider, radio, clickable) are not yet reliably detected. Despite the modest overall mAP obtained from this first-pass, imbalanced dataset, the model achieves a per-image inference time of close to twenty milliseconds, confirming that a YOLO-based pipeline is fast enough to support real-time downstream user-interaction analysis such as automated UI testing, accessibility tooling and usability analytics. The paper closes with a concrete plan - dataset rebalancing, stronger augmentation and transfer learning - for closing the accuracy gap on the minority element classes. |
| AUTHOR | G. SNEHA, P. ALEKHYA Department of Computer Science and Engineering, St. Mary's Women's Engineering College, Budampadu, Guntur, Andhra Pradesh, India |
| VOLUME | 186 |
| DOI | DOI: 10.15680/IJIRCCE.2026.1407019 |
| pdf/19_YOLO-Based GUI Element Detection for User Interaction Analysis.pdf | |
| KEYWORDS | |
| References | [1] A. A. Rahmadi and A. Sudaryanto, "Visual Recognition of Graphical User Interface Components Using Deep Learning Technique," J. Ilmu Komputer dan Informasi, vol. 13, no. 1, pp. 35-42, 2020. [2] S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," Proc. of NeurIPS, 2015. [3] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," Proc. of CVPR, pp. 770-778, 2016. [4] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," Proc. of CVPR, 2016. [5] J. Redmon and A. Farhadi, "YOLOv3: An Incremental Improvement," arXiv:1804.02767, 2018. [6] G. Jocher et al., "YOLOv5," https://github.com/ultralytics/yolov5, 2020. [7] C. Wang et al., "YOLOv8: A Real-Time Object Detection Framework with Distribution Focal Loss," arXiv:2304.08344, 2023. [8] J. Deng et al., "ImageNet: A Large-Scale Hierarchical Image Database," Proc. of CVPR, pp. 248-255, 2009. [9] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, "The Pascal Visual Object Classes (VOC) Challenge," Int. Journal of Computer Vision, vol. 88, no. 2, pp. 303-338, 2010. [10] UI-Elements-Detection-Dataset. Available: https://github.com/YashJain0412/UI-Elements-Detection-Dataset [11] A. Shorten and T. M. Khoshgoftaar, "A Survey on Image Data Augmentation for Deep Learning," Journal of Big Data, vol. 6, no. 60, 2019. [12] A. Vaswani et al., "Attention Is All You Need," Proc. of NeurIPS, 2017. |
