Sketch_based Image Retrieval Using Convolutional Neural Network with Multi_step Training
Subject Areas : Image Processing
Azita Gheitasi
1
,
Hassan Farsi
2
,
Sajad Mohamadzadeh
3
*
1 - Birjand, Iran
2 - University of Birjand
3 - University of Birjand
Keywords: Sketch-Based Image Retrieval (SBIR), Deep Learning, Multi-step training, contrastive loss, triplet loss,
Abstract :
The expansion of touch-screen devices has provided the possibility of human-machine interactions in the form of free-hand drawings. In sketch-based image retrieval (SBIR) systems, the query image is a simple binary design that represents the mental image of a person with the rough shape of an object. A simple sketch is convenient and efficient for recording ideas visually, and can outdo hundreds of words. The objective is to retrieve a natural image with the same label as the query sketch. This article presents a multi-step training method. Regression functions are used in the deep network structure to improve system performance, and various loss functions are employed for a better convergence of the retrieval system. The convolutional neural network used has two branches, one related to the sketch and the other related to the image, and these two branches can have the same or different architecture. In the proposed method, the data augmentation is used to increase the data. Data augmentation is essential in preventing overfitting, especially when the training data is limited. The proposed networks were evaluated using three well-known datasets (Tu-Berlin, Quick-Draw, Sketchy). After four training steps, a 56.48% MAP, 78.38% Precision was achieved, indicating the desirable performance of the network.
[1] Birari, D., D. Hiran, and V. Narawade, Survey on Sketch Based Image and Data Retrieval, in ICCCE 2019. 2020, Springer. p. 285-290.
[2] Li, Y. and W. Li, A survey of sketch-based image retrieval. Machine Vision and Applications, 2018. 29(7): p. 1083-1100.
[3] Liu, L., et al. Deep sketch hashing: Fast free-hand sketch-based image retrieval. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[4] Zhang, J., et al. Generative domain-migration hashing for sketch-to-image retrieval. in Proceedings of the European conference on computer vision (ECCV). 2018.
[5] Sain, A., et al. Stylemeup: Towards style-agnostic sketch-based image retrieval. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[6] Dorrani, Z., H. Farsi, and S. Mohamadzadeh, Image edge detection with fuzzy ant colony optimization algorithm. International Journal of Engineering, 2020. 33(12): p. 2464-2470.
[7] Bui, T., et al., Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regression. Computers & Graphics, 2018. 71: p. 77-87.
[8] Farsi, H. and S. Mohamadzadeh, Combining Hadamard matrix, discrete wavelet transform and DCT features based on PCA and KNN for image retrieval. Majlesi Journal of Electrical Engineering, 2013. 7(1).
[9] Del Bimbo, A. and P. Pala, Visual image retrieval by elastic matching of user sketches. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997. 19(2): p. 121-132.
[10] Chans, Y., et al. Feature-based approach for image retrieval by sketch. in Multimedia Storage and Archiving Systems II. 1997. SPIE.
[11] Rajendran, R.K. and S.-F. Chang. Image retrieval with sketches and compositions. in 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No. 00TH8532). 2000. IEEE.
[12] Bhunia, A.K., et al. More photos are all you need: Semi-supervised learning for fine-grained sketch based image retrieval. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[13] Dey, S., et al. Doodle to search: Practical zero-shot sketch-based image retrieval. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
[14] Gheitasi, A., H. Farsi, and S. Mohamadzadeh, Estimation of hand skeletal postures by using deep convolutional neural networks. International Journal of Engineering, 2020. 33(4): p. 552-559.
[15] Szegedy, C., et al. Going deeper with convolutions. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
[16] Yu, Q., et al. Sketch me that shoe. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.
[17] Bhattacharjee, S.D., et al. Query adaptive instance search using object sketches. in Proceedings of the 24th ACM international conference on Multimedia. 2016.
[18] Wang, F., L. Kang, and Y. Li. Sketch-based 3d shape retrieval using convolutional neural networks. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
[19] Qi, Y., et al. Sketch-based image retrieval via siamese convolutional neural network. in 2016 IEEE International Conference on Image Processing (ICIP). 2016. IEEE.
[20] Fuentes, A. and J.M. Saavedra. Sketch-qnet: A quadruplet convnet for color sketch-based image retrieval. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[21] V. Vakili_Zare, K.R., and H. Rezaei, K_Nearest Neighbor Classification Using Data and Deep and Deep Neural Networks 3rd Internattional Conference on Soft Computing 2019: p. 1034-1040.
[22] Schroff, F., D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
[23] Wang, X. and A. Gupta. Unsupervised learning of visual representations using videos. in Proceedings of the IEEE international conference on computer vision. 2015.
[24] Gordo, A., et al. Deep image retrieval: Learning global representations for image search. in European conference on computer vision. 2016. Springer.
[25] Sangkloy, P., et al., The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG), 2016. 35(4): p. 1-12.
[26] Bui, T., et al., Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network. Computer Vision and Image Understanding, 2017. 164: p. 27-37.
[27] Torres, P. and J.M. Saavedra. Compact and effective representations for sketch-based image retrieval. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[28] Saavedra, J.M., Rst-shelo: sketch-based image retrieval using sketch tokens and square root normalization. Multimedia Tools and Applications, 2017. 76(1): p. 931-951.
[29] Chavhan, R., Zero-Shot Sketch Based Image Retrieval. 2021, INDIAN INSTITUTE OF TECHNOLOGY BOMBAY.
[30] Deng, C., et al., Progressive cross-modal semantic network for zero-shot sketch-based image retrieval. IEEE Transactions on Image Processing, 2020. 29: p. 8892-8902.
[31] Tursun, O., et al., An efficient framework for zero-shot sketch-based image retrieval. Pattern Recognition, 2022. 126: p. 108528.
[32] Zhang, X., et al., A hybrid convolutional neural network for sketch recognition. Pattern Recognition Letters, 2020. 130: p. 73-82.
[33] Bhunia, A.K., et al. Sketch less for more: On-the-fly fine-grained sketch-based image retrieval. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.