Users' photos of items can reveal their tastes in a recommender system

  1. Pablo Pérez-Núñez 1
  2. Jorge Díez 1
  3. Oscar Luaces 1
  4. Beatriz Remeseiro 1
  5. Antonio Bahamonde 1
  1. 1 Universidad de Oviedo
    info

    Universidad de Oviedo

    Oviedo, España

    ROR https://ror.org/006gksa02

Revista:
Information Sciences

ISSN: 1872-6291,  0020-0255

Año de publicación: 2023

Volumen: 642

Páginas: 119227

Tipo: Artículo

DOI: 10.1016/J.INS.2023.119227 GOOGLE SCHOLAR lock_openAcceso abierto editor

Otras publicaciones en: Information Sciences

Resumen

Recommender Systems (RS) are based on the generalization of the observed interactions of a population of users with a collection of items. Collaborative Filters (CF) give good results, but they degrade when there are few interactions to learn from. The alternative would be to observe some features of the users that could be linked to their tastes. However, specific information on users or items is often not available. In this research work, we explore how to exploit the photos of items taken by users. Our aim is to assign similar meanings to the photos of items with which the same group of users interacted. For this purpose, we define a multi-label classification task from images to sets of users. The classifier uses a general-purpose convolutional neural network to extract the basic visual features, followed by additional layers necessary to accomplish the learning task. To evaluate our proposal we compared it with CFs, using two tourism datasets that include: restaurants of six cities and points of interest of three locations. According to the experimentation carried out, the poor results achieved by CFs are outperformed by our proposal, which takes into account the visual and taste semantics of the available photos.

€ Ver financiación

Información de financiación

This work was funded under grants PID2019-109238GB-C21 and TIN2015-65069-C2-2-R from the Spanish Ministry of Science and Innovation, and IDI-2018-000176 from the Principado de Asturias Regional Government, partially supported with ERDF funds. Pablo Pérez-Núñez acknowledges the support of the Principado de Asturias Regional Government under Severo Ochoa predoctoral program (ref. BP19-012). We are grateful to NVIDIA Corporation for the donation of the Titan Xp GPUs used in this research.

Financiadores

Referencias bibliográficas

  • C. Alonso, P. Pérez-Núñez, O. Luaces, J. Díez, B. Remeseiro, A. Bahamonde, TripAdvisor points of interest, https://doi .org /10 .5281 /zenodo .5749846, 2021.
  • G. Amis, Improving TripAdvisor photo selection with deep learning, https://www .tripadvisor .com /engineering /improving -tripadvisor -photo -selection -deep learning/, 2017.
  • K. Chaudhari, A. Thakkar, A comprehensive survey on travel recommender systems, Arch. Comput. Methods Eng. 27 (2020) 1545–1571.
  • W.T. Chu, Y.L. Tsai, A hybrid recommendation system considering visual information for predicting favorite restaurants, World Wide Web 20(6) (2017) 1313–1331.
  • Y. Deldjoo, T. Di Noia, D. Malitesta, F.A. Merra, A study on the relative importance of convolutional neural networks in visually-aware recommender systems, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp.3961–3967.
  • Y. Deldjoo, M. Schedl, P. Cremonesi, G. Pasi, Recommender systems leveraging multimedia content, ACM Comput. Surv. 53(5) (2020) 1–38.
  • J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, L. Fei-Fei, ImageNet: a large-scale hierarchical image database, in: IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp.248–255.
  • V. Dominguez, P. Messina, I. Donoso-Guzmán, D. Parra, The effect of explanations and algorithmic accuracy on visual recommender systems of artistic images, in: 24th International Conference on Intelligent User Interfaces, 2019, pp.408–416.
  • O.J. Dunn, Multiple comparisons among means, J.Am. Stat. Assoc. 56(293) (1961) 52–64.
  • J. Díez, P. Pérez-Núnez, O. Luaces, B. Remeseiro, A. Bahamonde, Towards explainable personalized recommendations by learning from users’ photos, Inf. Sci. 520 (2020) 416–430.
  • B. Ferwerda, M. Tkalcic, Predicting users’ personality from Instagrampictures: using visual and/or content features?, in: 26th Conference on User Modeling, Adaptation and Personalization, 2018, pp.157–161.
  • M. Figueredo, J. Ribeiro, N. Cacho, A. Thome, A. Cacho, F. Lopes, V. Araujo, From photos to travel itinerary: a tourism recommender system for smart tourism destination, in: IEEE 4th International Conference on Big Data Computing Service and Applications, 2018, pp.85–92.
  • G. Guo, Y. Meng, Y. Zhang, C. Han, Y. Li, Visual semantic image recommendation, IEEE Access 7 (2019) 33424–33433.
  • K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp.770–778.
  • R. He, J. McAuley, VBPR: visual Bayesian personalized ranking from implicit feedback, in: 30th AAAI Conference on Artificial Intelligence, 2016, pp.144–150.
  • X. He, L. Liao, H. Zhang, L. Nie, X. Hu, T.S. Chua, Neural collaborative filtering, in: Proceedings of the 26th International Conference on World Wide Web, 2017, pp.173–182.
  • G. Huang, Z. Liu, L. Van Der Maaten, K.Q. Weinberger, Densely connected convolutional networks, in: IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp.4700–4708.
  • S. Ioffe, C. Szegedy, Batch normalization: accelerating deep network training by reducing internal covariate shift, in: International Conference on Machine Learning, 2015, pp.448–456.
  • W.C. Kang, C. Fang, Z. Wang, J. McAuley, Visually-aware fashion recommendation and design with generative image models, in: IEEE International Conference on Data Mining, 2017, pp.207–216.
  • K. Kawattikul, Product recommendation using image and text processing, in: International Conference on Information Technology, 2018, pp.1–4.
  • D.P. Kingma, J. Ba, Adam: a method for stochastic optimization, in: 3rd International Conference on Learning Representations, 2015, pp.1–15.
  • V. Nair, G.E. Hinton, Rectified linear units improve restricted Boltzmann machines, in: 27th International Conference on Machine Learning, 2010, pp.807–814.
  • L. Nanni, S. Ghidoni, S. Brahnam, Handcrafted vs. non-handcrafted features for computer vision classification, Pattern Recognit. 71 (2017) 158–172.
  • J. Neve, R. McConville, ImRec: learning reciprocal preferences using images, in: Fourteenth ACM Conference on Recommender Systems, 2020, pp.170–179.
  • P. Pérez-Núñez, O. Luaces, J. Díez, B. Remeseiro, A. Bahamonde, TripAdvisor restaurant reviews, https://doi .org /10 .5281 /zenodo .5644892, 2021.
  • M. Sertkan, J. Neidhardt, H. Werthner, Eliciting touristic profiles: a user study on picture collections, in: 28th ACM Conference on User Modeling, Adaptation and Personalization, 2020, pp.230–238.
  • N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, J.Mach. Learn. Res. 15(1) (2014) 1929–1958.
  • L. Yang, Y. Cui, F. Zhang, J.P. Pollak, S. Belongie, D. Estrin, Plateclick: bootstrapping food preferences through an adaptive visual interface, in: 24th ACM International on Conference on Information and Knowledge Management, 2015, pp.183–192.