Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Re-Enrichment Learning: Metadata Saliency for the Evolutive Personalization of a Recommender System

Re-Enrichment Learning: Metadata Saliency for the Evolutive Personalization of a Recommender System applied sciences Article Re-Enrichment Learning: Metadata Saliency for the Evolutive Personalization of a Recommender System 1,† 2, Yuseok Ban and Kyungjae Lee * Agency for Defense Development, Daejeon 34068, Korea; ban@add.re.kr Department of Computer Science, Yong In University, Yong In 17092, Korea * Correspondence: kjlee@yiu.ac.kr; Tel.: +82-31-8020-3696 † First author: Tel.: +82-44-998-8148. Abstract: Many studies have been conducted on recommender systems in both the academic and industrial fields, as they are currently broadly used in various digital platforms to make personalized suggestions. Despite the improvement in the accuracy of recommenders, the diversity of interest areas recommended to a user tends to be reduced, and the sparsity of explicit feedback from users has been an important issue for making progress in recommender systems. In this paper, we introduce a novel approach, namely re-enrichment learning, which effectively leverages the implicit logged feedback from users to enhance user retention in a platform by enriching their interest areas. The approach consists of (i) graph-based domain transfer and (ii) metadata saliency, which (i) find an adaptive and collaborative domain representing the relations among many users’ metadata and (ii) extract attentional features from a user ’s implicit logged feedback, respectively. The experimental results show that our proposed approach has a better capacity to enrich the diversity of interests of a user by means of implicit feedback and to help recommender systems achieve more balanced personalization. Our approach, finally, helps recommenders improve user retention, i.e., encouraging users to click more items or dwell longer on the platform. Citation: Ban, Y.; Lee, K. Keywords: recommender system; personalization; diversity; sparsity; re-enrichment learning; graph- Re-Enrichment Learning: Metadata based domain transfer; metadata saliency Saliency for the Evolutive Personalization of a Recommender System. Appl. Sci. 2021, 11, 1733. https://doi.org/10.3390/app11041733 1. Introduction Academic Editors: Kwan-Hee Yoo, Online contents and services have been rapidly growing in recent decades. To help Carson K. Leung and Nakhoon Baek users make decisions when faced with overwhelming options and to achieve commercial Received: 17 January 2021 success in making profit, both academic and industrial research on finding specific sets Accepted: 7 February 2021 of items that meet the personalized interests of users has been intensely conducted. For Published: 16 February 2021 this purpose, recommender systems have been suggested and designed that are based on a user ’s history, including whether the user has bought an item, which ratings the user Publisher’s Note: MDPI stays neu- has given to items, etc. Previous studies have reported that recommendations account for tral with regard to jurisdictional clai- 60% of the clicks on the main screen of Youtube [1], 75% of what people watch through ms in published maps and institutio- Netflix [2], and 35% of the sales through Amazon [3]. nal affiliations. Recommendation is obviously not an easy task. A recommender system needs to provide a personalized user experience with long-term satisfaction under the condition that users’ preferences can change over time [4]. Behaviors based on the preferences of users could be either positively or negatively influenced by recommendation results. Tradi- Copyright: © 2021 by the authors. Li- censee MDPI, Basel, Switzerland. tionally, the recommendation problem has been considered as equivalent to the problem This article is an open access article of precisely predicting the rating that a user would leave on an item. However, recent distributed under the terms and con- academic studies and industrial providers have both emphasized that one of the main ditions of the Creative Commons At- measurement targets is improved user retention [5,6]. In spite of the gains in theoretical tribution (CC BY) license (https:// accuracy, it was unclear that the winning strategy in accuracy would result always in creativecommons.org/licenses/by/ increasing business value [7]. 4.0/). Appl. Sci. 2021, 11, 1733. https://doi.org/10.3390/app11041733 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, 1733 2 of 17 Still, recommender systems are mostly optimized toward the accuracy of predictions of item ratings, and are likely to gradually narrow down the interest area of a user—the so-called the diversity problem [8–11]. That is to say, recommendations where the lesser interests get squeezed out by the main interest ironically help the system minimize prediction errors. This not only impedes the diversity of experiences of users, but also causes filter bubbles [12,13], e.g., a reduced spectrum of user consumption and the political bias, which limit discovery or neglect the potential for promoting new items from the long tail [14,15]. Moreover, as the training of a recommender system mostly relies on explicit feedback (typically the ratings of items), it is inevitable to face the fact that only a small portion of users leave their ratings—the so-called the sparsity problem [16,17]. In this paper, we propose a novel data interpretation approach, which includes graph- based domain transfer and metadata saliency, for recommender systems, and show its effective application for a personalized recommender, namely re-enrichment learning. To address the problems of diversity and sparsity, the goal of re-enrichment learning is to find conspicuous features of the implicit logged feedback [18,19] that tell the behavior of a user, e.g., how frequently the user clicks on a certain item or how long the user stays on the platform, and to promote immanently correlated recommendations. Unlike explicit feedback, implicit logged feedback is relatively easier to spontaneously collect from users. Metadata saliency describes the confidence of a recommendation in the transferred domain, which reflects the graph-based relationships among items. It is generally known that the visual saliency represents the position-based visual preference [20]. As such, methods for predicting eye fixation maps on websites or mobile interfaces [21–23] have been studied, and the relation between the probability of the movement of a mouse pointer and the eye fixation has been discussed [24]. Inspired by those works, metadata saliency is a novel approach to interest fixation for distinguishing a certain set of items from the others. Based on the establishment of links between one group of items and their closely related ones, re-enrichment learning is used to train a recommender system based on the metadata saliency, thus enriching the potential interest areas of a user rather than narrowing them down. The experimental results show that our approach is effective in interpreting implicit logged feedback and helpful in persuading a user to click more or stay longer on a commercial platform. The organization of the rest of this paper is as follows: Section 2 presents a review of the related work, Section 3 describes our proposed approach in detail, and Section 4 provides the comparative experiments and the discussion. Finally, our conclusion is given in Section 5. 2. Background Recommender systems can be largely classified into three categories: the collaborative approach, content-based approach, and hybrid approach [16,17,25]. Firstly, the collabora- tive approach is based on the idea that people with similar preferences are likely to agree on an evaluation of an item. This approach consists of neighborhood-based methods, which directly use the users’ item ratings to estimate the classification of a new item, and model- based methods, which use the ratings to acquire knowledge and teach a predictive model. The advantage of this approach is that new data can easily be added in an incremental manner, whereas its weakness to the cold-start problem of new users and the sparsity problem are its disadvantages. Koren et al. (SVD) [26] suggested a singular value decompo- sition (SVD)-based matrix factorization algorithm that uses stochastic gradient descent for optimization. Rendle et al. (Bayesian personalized ranking (BPR)) [27] introduced a generic optimization criterion and learning algorithm for personalizing a recommender system. The optimization criterion is based on the maximum posterior estimator derived from Bayesian analysis, while the learning algorithm is based on stochastic gradient descent with bootstrap sampling. He et al. (L-GCN) [28] adopted the neighborhood aggregation of a graph convolution network (GCN) [29] and proposed learning user and item embeddings by linearly propagating them with respect to the interaction graph of the user and item. Appl. Sci. 2021, 11, 1733 3 of 17 They used the weighted summation of the embeddings, which are learned at all layers, as in the final stage. Then, the content-based approach is based on the combination of a user ’s profile and an item description. In other words, the similarity between the items that the user has liked in the past and the detailed content of an item play important roles. This approach shows strength when recommending an item that is not yet rated by any user, whereas it suffers from the overspecialization problem. Wang et al. (deep knowledge-aware network (DKN)) [30] proposed a multi-channel and word-entity-aligned knowledge-aware convo- lutional neural network for semantic and knowledge-based representation of news as well as an attention module to dynamically compute the aggregated historical representation of a user. Wu et al. (neural news recommendation with personalized attention (NPA)) [31] suggested personalized convolutional neural networks that adopt the embedding of user IDs as the queries of the correlations between words and news. The meta-context dimen- sion tree (Meta CDT) was proposed for the selection of the most suitable contents and services for a user in a certain context, and was used in a practical context-aware applica- tion by Colace et al. [32] and Casillo et al. [33]. The approach has strength in tailoring the information domain according to the user ’s needs as well as in analyzing relevant features of context models. Lastly, there also are hybrid approaches that combine the collaborative and content- based approaches to exploit the advantages of the individual approaches [16,34]. Many types have been proposed, such as by aggregating the predictions of collaborative and content-based approaches or integrating one approach’s characteristics into the other. Kula (light factorization machine (L-FM)) [35] proposed a hybrid matrix factorization model that represents users and items as linear combinations of the latent factors of their content features. Cheng et al. (wide and deep (W&D)) [36] introduced a framework for jointly training feed-forward neural networks with embedding and a linear model based on feature transformation for generic recommenders with sparse inputs. Many of the recent studies have discussed the problem that unbalanced or biased recommendations can shrink the diversity of interests of a user. Practical evaluation scenarios and techniques have been introduced to produce unbiased estimators in spite of biased or missing data [37,38]. Research has been carried out on adopting the relationships among items or entities as well as among users [39]. For instance, each user has an individual graph-based representation for their explicit feedback of ratings, and the graph can provide insights into how good use of the hidden information can be made in terms of personalized recommendations. Still, most of the previous approaches have difficulty with the aforementioned sparsity and diversity problems. For the sparsity problem, we need to further study how to incorporate the naturally obtainable feedback information from users. In addition, there must be many alternatives to alleviate the diversity problem resulting from unbalanced recommendations. 3. Re-Enrichment Learning The two main objectives of re-enrichment learning are literally to enrich the interest areas recommended to a user and to achieve them by extracting meaningful features from naturally acquirable feedback data. In other words, re-enrichment learning is directly designed to tackle the diversity and sparsity problems of a recommender system. Our proposed approach is composed of two cores: graph-based domain transfer and metadata saliency. The domain of metadata saliency is determined by the graph-based domain transfer, while the graph itself can be changed recursively by the metadata saliency from new feedback. To be more concrete, every implicit logged feedback of a user causes a change in the user ’s metadata saliency; the update in the metadata saliency is promptly reflected in the user ’s graph description, as well as in the universal domain determined from a large set of users; the new environment of graph again recursively influences the next feedback of the user. Appl. Sci. 2021, 11, 1733 4 of 17 3.1. Graph-Based Domain Transfer An undirected weighted graph is assigned to every user in order to build a data structure that consists of item categories as the finite set of nodes and the similarities between categories as the finite set of edges. G = (V , E ) s.t. k2f1, ..., Ng and t  0 (1) k,t k,t k,t V = fv (1), v (2), ..., v ( M)g (2) k,t k,t k,t k,t E = fe (1, 2), e (1, 3), ..., e ( M 1, M)g (3) k,t k,t k,t k,t th As noted in Equations (1)–(3), the graph of the k user among N users at time t, G , is k,t made up of M nodes V with their attributes v and C edges E with their attributes k,t k,t 2 k,t e . The edge attribute between the two nodes v (i) and v (j) is e (i, j), which can be k,t k,t k,t k,t computed as in Equation (4), where i, j 2 f1, ..., Mg. The attribute of an edge represents the combination of the influences and similarity of two nodes. A node is determined to be influential when its attribute is large, and two nodes are determined to be similar when they share similar levels of attributes to that of the same user. Here, the attribute of a node is the metadata saliency, which represents the attentional prominence of the node from the entire set of nodes. Equation (4) is composed of two terms: (i) the likelihood for the influence of a node and (ii) the weight for the similarity between two nodes. A softmax function has been adopted in the first term to produce the influence of a node as a probabilistic value within [0, 1] (see Equations (5) and (6)), where h 2 f1, ..., Mg, and a is the influence factor, which determines the degree of importance of an individual feedback (s.t. a  1). In the second term, the inversed substitution between two nodes’ attributes is adopted to produce the similarity as a weight value (see Equations (7) and (8)). The attributes of nodes have been normalized to remove scale dependency. (4) e (i, j) = s a v (i)  1 v (i, j) k,t k,t k,t av (i) k,t s(a v (i)) = (5) k,t av (h) k,t å e h=1 v (i) min v (h) k,t k,t 8h v (i) = (6) k,t max v (h) min v (h) k,t k,t 8h 8h 4 4 v (i, j) min v (i, h) k,t k,t 8h v (i, j) = (7) k,t 4 4 max v (i, h) min v (i, h) k,t k,t 8h 8h (8) v (i, j) = jv (i) v (j)j k,t k,t k,t Then, the edges can be denoted by an jV j × jV j matrix, i.e., adjacency matrix k,t k,t E = e (i, j) [40], where each element indicates the attribute of an edge, and the elements k,t k,t can be stored in a triangular matrix as e (i, j) = e (j, i) (as shown in Figure 1). Finally, k,t k,t the attributes of nodes and edges, as well as the adjacent matrix that allows their sequential representation, are able to respond to every feedback event. Appl. Sci. 2021, 11, 1733 5 of 17 𝑛𝑛𝑛𝑛 𝑟𝑟𝑛𝑛 𝑠𝑠 𝑡 1 2 3 ⋯ 𝑀𝑀 ⋯ 0 1 𝑒𝑒 (1,1) 𝑘𝑘 ,𝑠𝑠 𝑛𝑛𝑛𝑛 𝑒𝑒 (2,1) 𝑒𝑒 (2,2) ⋰ ⋮ 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 𝑟𝑟𝑛𝑛 𝑒𝑒 (3,1) 𝑒𝑒 (3,2) 𝑒𝑒 (3,3) 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 ⋮ ⋮ ⋮ ⋮ ⋱ 𝑠𝑠 𝑡 𝑒𝑒 (𝑀𝑀 , 1) 𝑒𝑒 (𝑀𝑀 , 2) 𝑒𝑒 (𝑀𝑀 , 3) 𝑒𝑒 (𝑀𝑀 ,𝑀𝑀 ) 𝑀𝑀 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 Figure 1. Adjacency matrix E . k,t In the end, a universal domain D at time t can be determined based on the graphs of th N users, where D (h) is the h domain in which M nodes are sorted by the level of the th attribute of the edge connected to the h node (see Equation (9)). That is to say, in D (h), the th h node itself appears first, the node with the second largest attribute of the edge connected th to the h node appears second, and so on. Understandably, the domain changes over time as the individual graphs vary—e.g., with fluctuating attributes, newly inserted nodes starting with zero-level attributes, etc. Empirically, two nodes that have large attributes and share the edge of a large attribute become adjacent neighbors in the transferred domain. Here, being neighbors has dire consequences with respect to metadata saliency. D = fD (1), D (2), ..., D ( M)g (9) t t t t 3.2. Metadata Saliency A user ’s behavioral history can be used not only as primary cumulative data, but also as information from which secondary features are extracted. Metadata saliency is one of the secondary features that can play an important role, especially in leveraging implicit logged feedback data. A few interesting items usually account for the majority of interest fixation, and they can be quantified by extracting metadata saliency from the logged feedback. Among the various types of feedback, we focus on the ones that contain the retention data of a user, and we start with the simplest type: the clicks on recommended th items. The metadata saliency of the i category in the transferred domain at time t, v (i ), k,t d th is computed based on the i clicked category, as in Equation (10). A Gaussian fixation model has been adopted [21,24,41,42], of which the average and standard deviation are i th and 1, respectively, in order to reinforce the metadata saliency of the i category, but also to enrich its nearest neighbors with respect to the transferred domain. The Gaussian fixation th is also scaled based on the metadata saliency of the i category at time t 1, v (i ), k,t1 d and the influence factor a (see Equation (11)). The very first metadata saliency, v , can k,0 be set by a preference survey for new users for the cold start (discussed in Section 4.2.1). Otherwise, for users who refuse the survey, the average of the metadata saliencies of many random users can be an alternative for v (discussed in Section 4.2.2). k,0 i 1  2 d (i i ) (10) 2 d v (i ) = v (i ) + p  e k,t d k,t1 d 2p A = (11) v (i )+a k,t1 d 𝑠𝑠𝑠𝑠 𝑠𝑠𝑠𝑠 Appl. Sci. 2021, 11, 1733 6 of 17 Suppose that a graph has been built to represent the metadata saliencies of movie genres for a user (see Figure 2). A node is presented as a circle, of which the radius indicates its attribute level. An edge is drawn as a line, of which the thickness is proportional to its attribute level. The user must mostly have preferences for Action movies, as well as Sci-Fi movies, because the two nodes corresponding to those genres are the two biggest circles, as shown in Figure 2. Provided that the user clicked an item that belongs to the Action category at time t, not only does the metadata saliency of Action node get larger at time t + 1, but that of the Sci-Fi node also benefits from it to a certain extent due to the connection of the edge to the two nodes and the Gaussian fixation effect. Moreover, the Fantasy and Adventure nodes also deserve a small benefit from the feedback. Because of the new feedback, the attributes of the nodes and edges change simultaneously and incrementally create a new condition in the graph representation. As such, the metadata saliency naturally changes over time by means of the newly collected feedback. The change in metadata saliency is promptly reflected in a user ’s graph and, by extension, also in the universal domain, which is built upon many users’ graphs. Consequently, the evolutive characteristic based on the mutual interaction in the graph makes re-enrichment learning prompt and flexible. Mystery Thriller Animation Fantasy Action Sci-Fi Crime Adventure War Documentary Figure 2. Conceptual visualization of a graph (node: circle radius, edge: line thickness). 3.3. Summary In summary, we provide Figure 3 to depict a diagram that connects all the steps of the process, and the process can be listed as follows: • Step 1: Obtain the user ’s implicit logged feedback i at time t from the recommender. • Step 2: Update nodes V ! V : Calculate node attribute v (i ) at time t using k,t1 k,t k,t d the universal domain D and the node attribute v (i ) at time t 1 (as shown in t1 k,t1 d Equation (10)). • Step 3: Update edges E ! E : Calculate the edge attribute e (i, j) at time t k,t1 k,t k,t using the node attribute v (i ) at time t (as in Equation (4), which consists of the two k,t d following terms). – Compute the likelihood of influence of the node: s a v (i) (see Equation (5)). k,t – Compute the weight for the similarity between two nodes: 1 v (i, j) (see k,t Equation (7)). • Step 4: Update the universal domain D ! D (as in Equation (9)): Sort domain t1 t D (h) by edge attribute. • Step 5: Apply the node attribute v (i ) at time t, i.e., the metadata saliency, to the k,t d recommender. Appl. Sci. 2021, 11, 1733 7 of 17 Furthermore, we provide a pseudocode to effectively present how the different steps of our proposed approach are organized (see Table 1). Obtain Update Update Update Feedback Nodes Edges Universal domain 9 4 7 3 + + 6 8 2 1. Action 1 9 7 4 5 3 8 6 1 2. Adventure + + 3 5 4 7 9 3. Animation 8 6 1 4 7 5 3 4. Crime 2 8 6 1 5 3 4 7 9 2 8 6 5. Documentary 1 9 4 7 5 3 √ 2 𝒊𝒊 6 8 1 6. Fantasy + + + + 𝒅𝒅 → 7 4 9 5 3 7. Mystery 8 6 1 9 4 7 5 3 + + 8 6 1 8. Sci-Fi 7 4 5 3 2 8 6 9. Thriller + 1 9 7 4 5 3 + 2 8 6 10. War 1 Figure 3. The process of re-enrichment learning. Table 1. The pseudocode for re-enrichment learning. Pseudocode Input : Implicit logged feedback i , Nodes V , Edges E , Universal domain D k,t1 k,t1 t1 Output : Nodes V , Edges E , Universal domain D k,t k,t Parameter : Influence factor a / Update Nodes V ! V / k,t1 k,t for k = 1 : N do for i = 1 : M do 1 : Calculate Metadata saliency v (i ) k,t d end for : Construct Nodes V = fv (1), v (2), ..., v ( M)g k,t k,t k,t k,t end for / Update Edges E ! E / k,t1 k,t for k = 1 : N do for i = 1 : M do for j = i : M do 1 : Calculate Likelihood for influence of node s a v (i) k,t 2 : Calculate Weight for similarity between two nodes 1 v (i, j) k,t 3 : Calculate Edge attribute e (i, j) = s a v (i)  1 v (i, j) k,t k,t k,t end for : Construct Edges E = fe (1, 2), e (1, 2), ..., e ( M 1, M)g k,t k,t k,t k,t end for end for / Update Universal domain D ! D / t1 t for h = 1 : M do 1 : Calculate Domain D (h) by sorting nodes w.r.t. edge attribute end for : Construct Universal domain D = fD (1), D (2), ..., D ( M)g t t t t Appl. Sci. 2021, 11, 1733 8 of 17 4. Experiment and Discussion 4.1. Experimental Setup 4.1.1. Dataset • MovieLens dataset [43]: The MovieLens (100k) dataset was collected by the Grou- pLens Research Project at the University of Minnesota (see Figure 4). It consists of 100,000 ratings, which range from 1 to 5, obtained from 943 users on 1682 movies as items. The data were acquired through the MovieLens website during the seven- month collection period. The dataset was cleaned up by excluding users who wrote less than 20 ratings. A movie can belong to more than one genre; there are 18 genres, defined as: (i) Action, (ii) Adventure, (iii) Animation, (iv) Children’s, (v) Comedy, (vi) Crime, (vii) Documentary, (viii) Drama, (ix) Fantasy, (x) Film-Noir, (xi) Horror, (xii) Musical, (xiii) Mystery, (xiv) Romance, (xv) Sci-Fi, (xvi) Thriller, (xvii) War, and (xviii) Western. • Amazon dataset [44,45]: The Amazon review dataset contains the reviews of items and their metadata collected from the Amazon platform (see Figure 5). The dataset sampled for the experiment included around 70 million ratings of 745,018 products, ranging from 1 to 5. In the experiment, 20 categories were included: (i) Beauty, (ii) Fashion, (iii) Appliances, (iv) Arts, crafts, and sewing, (v) CDs and vinyl, (vi) Cell phones and accessories, (vii) Digital music, (viii) Gift cards, (ix) Grocery and gourmet food, (x) Industrial and scientific, (xi) Luxury beauty, (xii) Magazine subscriptions, (xiii) Movies and TV, (xiv) Musical instruments, (xv) Office products, (xvi) Patio, lawn, and garden, (xvii) Pet supplies, (xviii) Prime pantry, (xix) Software, and (xx) Sports and outdoors. Figure 4. Examples of movies from different genres (MovieLens dataset). Figure 5. Examples of goods from different categories (Amazon dataset). 4.1.2. Evaluation Learning from implicit user feedback, e.g., click and dwell time, has been an important factor in improving recommender systems [18]. The common implicit retention metric [19], the click-through rate (CTR) [6,46–49], was adopted to intuitively and directly evaluate the effectiveness of the methods for user satisfaction or retention on digital platforms. Specifically, the metric was used to measure the ratio of clicks to recommendations, as in Equation (12). Appl. Sci. 2021, 11, 1733 9 of 17 Number of click-throughs (12) CT R =  100 (%) Number of recommendations Furthermore, the metric of mean average precision [50], M AP@K, supports the consis- tency of an experiment. This metric indicates how many items a user engages with among those recommended to them, as CTR does, but also uses the sequence of feedback (see Equation (13), where S indicates the number of samples). To begin with, average precision, AP@K, is defined as the summation of the engaged precision values divided by the number of engagements, m, where P(i) is the precision at i and d(i) is a bivariate function for engagement (as in Equations (14)–(16)). M AP@K = ( AP@K) (13) n=1 AP@K = P(i) d(i) (14) i=1 Number of relevant recommendations P(i) = (15) Number of recommendations th 1 if i recommendation is engaged, d(i) = (16) 0 otherwise. A/B experiments [5,19,51] were conducted to verify the difference between applying and not applying re-enrichment learning to baseline methods. Five baseline methods are used for comparison: SVD [26], BPR [27], L-GCN [28], L-FM [35], and W&D [36]. The baseline methods are either collaborative or hybrid approaches because our proposed method treats the category preferences of users, e.g., movie genre and product category, rather than the internal content of an item, e.g., the words in a news article. We collected data from 32 participants, who clicked on all the items that seemed to be interesting. For fairness, the participants were informed of which method was used in each experiment. The time required to finish one sequence of collecting implicit feedback ranged from 10 to 30 min depending on the participant. The sequence of collecting feedback was repeated four times for every participant. A total of 4532 feedbacks were collected for the MovieLens dataset and 4830 feedbacks were collected for Amazon dataset were collected. Lastly, a non-parametric statistical hypothesis test, Wilcoxon test [52], was performed to check the significance of improvements. We adopted a significance level a of 0.05, which indicates a 5% risk of concluding that a difference exists when there is no actual difference. The interpretation of a result in the Wilcoxon test is that the null hypothesis, “The population median (h) equals the hypothesized median (h )”, is wrong when the p-value is smaller than the significance level (i.e., the difference is significant). 4.1.3. Configuration The baseline methods were applied with the settings of default parameters given in their open sources. Based on the scaling property of signal processing, the influence factor a in Equation (4), which operates as the scaling factor of the softmax function, was identically set to 4 in all cases. This parameter determines the agility of re-enrichment learning in adapting to recent short-term feedback. 4.2. Results and Discussion 4.2.1. Recommending Movies At the beginning, users were asked to respond to a survey on their preferred categories as a cold start. Otherwise, users could also skip it to simply use the given averaged preference model built upon the feedback collected from other previous users. When Appl. Sci. 2021, 11, 1733 10 of 17 responding to the survey, the selected categories directly created an initial metadata saliency. Then, a set of 10 movies, with their images and titles, was recommended to a user at a time, and the user was allowed to click on any items they were interested in. The user could also skip the entire set and go on to the next set. Table 2 shows our A/B experiments, and it compares the results of all baseline comparison methods with respect to CTR and MAP@K for the MovieLens dataset. As demonstrated in Figure 6, our proposed re-enrichment learning helps baseline methods improve the resulting retention of users by providing a feature that enriches the interest areas of the user instead of overspecializing them. Compared to the intuitive metric, CTR, MAP@K additionally reflects the order in which the user engages with items within a recommended set, so we can infer that our approach has nothing to do with recommending more attractive items in the front. Still, the results of CTR and MAP@K show a consistent tendency for improvement. In addition, Table 3 presents the results of the Wilcoxon test which essentially calculates the difference between sets of paired samples and analyzes these differences to establish if they are statistically significantly different from one another. The resulting p-values in all cases were much smaller than the significance level a, which allows the conclusion that the differences between the population distribution and the hypothesized distribution are statistically significant. A : w / o R e - e n r i c h m e n t l e a r n i n g A : w / o R e - e n r i c h m e n t l e a r n i n g B : w / R e - e n r i c h m e n t l e a r n i n g B : w / R e - e n r i c h m e n t l e a r n i n g 3 0 0 . 4 0 0 . 3 5 0 . 3 0 2 0 K 0 . 2 5 R @ 0 . 2 0 T P C A 0 . 1 5 1 0 0 . 1 0 0 . 0 5 0 0 . 0 0 A B A B A B A B A B A B A B A B A B A B N N D D D R M D R M C C F F P & P & V V G - G - - - S B S B L L W W L L (a) Average click-through rate (b) Mean average precision @ K Figure 6. Visualization of A/B experiments on the MovieLens dataset. Table 2. The results of A/B experiments on the MovieLens dataset. CTR MAP@K Method A B B A (B A)/A A B B A (B A)/A SVD 14.76% 18.52% + 3.76% " 25.47% 0.306 0.339 + 0.033 " 10.91% BPR 17.10% 20.90% + 3.80% " 22.22% 0.318 0.368 + 0.049 " 15.51% L-GCN 17.40% 20.37% + 2.97% " 17.07% 0.341 0.363 + 0.022 " 6.54% L-FM 12.89% 16.91% + 4.02% " 31.19% 0.280 0.329 + 0.049 " 17.56% W&D 14.28% 17.18% + 2.90% " 20.31% 0.290 0.324 + 0.034 " 11.77% This observation is supported by Figure 7, which shows an example comparing the initial metadata saliency of a user to the metadata saliency variations over 30 implicit feedbacks. The x-axis presents the node of a graph (i.e., movie genre), while the y-axis shows the attribute of a node (i.e., the level of metadata saliency). As can be observed in Figure 7a, the user started with two large saliencies at Action and Sci-Fi. Despite the fact that the Action and Sci-Fi genres accounted for the majority of the 30 feedbacks and the two were still ranked in the two largest saliencies, the metadata saliencies of other genres (Adventure and Thriller) prominently increased because they had relatively large Appl. Sci. 2021, 11, 1733 11 of 17 edge attributes with Action and Sci-Fi (see the thicker lines in Figure 7a). We depict all the edges connected to Action as red-colored lines and all the edges connected to Sci-Fi as blue-colored lines, which show that Adventure and Thriller are fairly interrelated with Action and Sci-Fi with respect to the thickness of the lines. It is worth observing that Adventure and Thriller were encouraged to be recommended, and were consequently selected quite a few times. Table 3. The Wilcoxon test of the A/B experiments on the MovieLens dataset. CTR MAP@K Method p-Value a p-Value a SVD 0.000001220 < 0.05 0.003038835 < 0.05 BPR 0.000001947 < 0.05 0.000007521 < 0.05 L-GCN 0.000001472 < 0.05 0.002098150 < 0.05 L-FM 0.000001110 < 0.05 0.000256538 < 0.05 W&D 0.000002135 < 0.05 0.000070817 < 0.05 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 (a) Initial metadata saliency (v ) k,0 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 (b) Metadata saliency after 30 feedbacks (v ) k,30 Figure 7. The effect of implicit feedback on metadata saliency (MovieLens dataset). Collaborative approaches find new items in which a particular user is most likely to be interested by modeling the task as a regression or classification using the rating data. By being provided with a recommendation of an item that satisfies them, the user is likely Action Adventure Animation Children's Comedy Crime Documentary Drama Fantasy Action Film-Noir Adventure Horror Animation Musical Children's Mystery Comedy Romance Crime Sci-Fi Documentary Thriller Drama War Fantasy Western Film-Noir Horror Musical Mystery Romance Sci-Fi Thriller War Western Appl. Sci. 2021, 11, 1733 12 of 17 to engage more with the platform. By replacing the rating with metadata saliency, the retention results of collaborative approaches were increased. This observation directly shows that users spent more time exploring the enriched interest areas. In addition, only small subsets of movies from the available set of movies were rated by users [16], which made it difficult to successively collect explicit cues for training. In contrast, metadata saliency is based on implicit logged feedback, which can be naturally and continuously obtained. Moreover, hybrid approaches already merge different techniques or features to avoid limitations, but they also can be further assisted by re-enrichment learning. Based on our graph-based domain transfer, the relations between the categories were effectively incorporated, and that allowed the metadata saliency to emphasize the salient interest areas of the user, as well as the interest areas implicitly linked to them. As a result, applying re-enrichment learning led to significantly improved click-through rates for the hybrid approaches. L-FM is based on the latent representation approach, which aims to learn user and item representations from the interaction data [35], while the wide linear model of W&D aims to memorize sparse feature interactions based on cross-product feature transformation [36]. These characteristics allow L-FM and W&D to be run together with graph-based domain transfer and metadata saliency. 4.2.2. Recommending Goods Likewise, the preference survey was followed by a set of 10 recommendations, and users could click on any items they wished. Experiments were carried out using the Amazon dataset [44,45], which contains Amazon products’ review data as well as the products’ metadata. Table 4 reports the results of our A/B experiments, and Figure 8 demonstrates their visual comparisons. We observed that the methods with re-enrichment learning outperformed their baselines, generally producing increased CTR and MAP@K. The results of the Wilcoxon test on the Amazon dataset show more or less the same tendency as those of the MovieLens dataset. As shown in Table 5, the p-values were fairly smaller in all cases than the significance level a, which means that the differences between A and B were significant. A : w / o R e - e n r i c h m e n t l e a r n i n g A : w / o R e - e n r i c h m e n t l e a r n i n g B : w / R e - e n r i c h m e n t l e a r n i n g B : w / R e - e n r i c h m e n t l e a r n i n g 3 0 0 . 4 0 0 . 3 5 2 5 0 . 3 0 2 0 0 . 2 5 1 5 0 . 2 0 T P C A 0 . 1 5 1 0 0 . 1 0 0 . 0 5 0 0 . 0 0 A B A B A B A B A B A B A B A B A B A B N N D D D R M D R M C C P F & P F & V V G - G - S B - S B - L W L W L L (a) Average click-through rate (b) Mean average precision @ K Figure 8. Visualization of A/B experiments on the Amazon dataset. Appl. Sci. 2021, 11, 1733 13 of 17 Table 4. The results of A/B experiments on the Amazon dataset. CTR MAP@K Method A B B A (B A)/A A B B A (B A)/A SVD 15.43% 19.21% + 3.78% " 24.50% 0.303 0.340 + 0.037 " 12.21% BPR 16.16% 19.03% + 2.87% " 17.76% 0.313 0.337 + 0.024 " 7.61% L-GCN 16.46% 19.56% + 3.10% " 18.83% 0.315 0.341 + 0.027 " 8.46% L-FM 11.79% 15.01% + 3.22% " 27.31% 0.247 0.296 + 0.049 " 19.75% W&D 11.52% 15.34% + 3.82% " 33.16% 0.242 0.285 + 0.043 " 17.71% Table 5. The Wilcoxon test of the A/B experiments on the Amazon dataset. CTR MAP@K Method p-Value a p-Value a SVD 0.000000834 < 0.05 0.000029252 < 0.05 BPR 0.000001340 < 0.05 0.000600795 < 0.05 L-GCN 0.000000833 < 0.05 0.000643566 < 0.05 L-FM 0.000000916 < 0.05 0.000043941 < 0.05 W&D 0.000001110 < 0.05 0.000104444 < 0.05 Figure 9 demonstrates an example of the variations in metadata saliency over 30 feedbacks by a user. This example is interesting because the user did not participate in the preference survey at the beginning. As such, the average metadata saliency from 70M previous feedbacks was used for the initial metadata saliency (see Figure 9a). The two categories, Digital music and Gift cards, initially obtained relatively large metadata saliencies. On the other hand, the user mostly selected Appliances, Cell phones and accessories, Industrial and scientific, and especially Sports and outdoors. Consequently, the metadata saliency levels of those categories increased during the latter feedback (see Figure 9b). The rest of the categories moved opposite to the most selected ones, yet the saliency levels, of which the edge attribute was large (see the thicker lines in Figure 9a), were rather slightly raised. Re-enrichment learning noticeably raised the click-through rates of the collaborative approaches. The advantage of collaborative approaches is that adding new data in an in- cremental manner is relatively easy, and re-enrichment learning is quite well matched with this advantage. It is easy to add a new node and its edges, starting with initial attributes set to zero. Furthermore, our proposed approach supplements the weakness of collaborative approaches when very few explicit feedbacks, i.e., ratings, are limitedly collectable from users. Furthermore, we observed that hybrid approaches with re-enrichment learning also outperformed their baselines. Re-enrichment learning effectively exploits the implicit logged feedback information and led to better click-through rate results. The observation from Figure 9 intuitively conveys that the interest areas that have close relations in the graph-based transferred domain to the areas selected by a user were nourished due to re-enrichment learning. In consequence, the recommendations based on re-enrichment learning encouraged users to consume more content. Appl. Sci. 2021, 11, 1733 14 of 17 0.08 0.06 0.04 0.02 (a) Initial metadata saliency (v ) k,0 0.08 0.06 0.04 0.02 (b) Metadata saliency after 30 feedbacks (v ) k,30 Figure 9. The effect of implicit feedback on metadata saliency (Amazon dataset). 5. Conclusions In this paper, we introduced a novel approach that effectively exploits the implicit logged feedback from users to enrich their potential interest areas and to increase user retention in recommender systems for digital platforms. Our approach includes graph- based domain transfer and metadata saliency, which are incorporated to operate as re- enrichment learning. A universal domain was built based on the graph representation of item categories and their interrelations as nodes and edges, respectively. While capturing the attentional prominence of a node, metadata saliency also confers benefits on the nearest neighbors of the node in the universal domain. Every implicit logged feedback of a user causes a change in the user ’s metadata saliency; the update in the metadata saliency is promptly reflected in the user ’s graph description, as well as in the universal domain determined by a number of users; the new environment of graph again recursively influences the next feedback of the user. The eventual goal is the improvement of user retention, rather than accurately predicting ratings, as there are many better ways to help people find interesting items than focusing only on those with high predictions of ratings. It should be pointed out that re-enrichment learning shows a constant tendency of internal and external advantages. Internally, it has a better capacity to enrich the diversity of possible interest areas of users and to help recommender systems achieve more balanced personalization. In addition, to address the issue of sparsity of explicit feedback, it extracts ALL BEAUTY AMAZON FASHION APPLIANCES ARTS CRAFTS AND SEWING CDS AND VINYL CELL PHONES AND ACCESSORIES DIGITAL MUSIC GIFT CARDS GROCERY AND GOURMET FOOD ALL BEAUTY INDUSTRIAL AND SCIENTIFIC AMAZON FASHION LUXURY BEAUTY APPLIANCES MAGAZINE SUBSCRIPTIONS ARTS CRAFTS AND SEWING MOVIES AND TV CDS AND VINYL MUSICAL INSTRUMENTS CELL PHONES AND ACCESSORIES OFFICE PRODUCTS DIGITAL MUSIC PATIO LAWN AND GARDEN GIFT CARDS PET SUPPLIES GROCERY AND GOURMET FOOD PRIME PANTRY INDUSTRIAL AND SCIENTIFIC SOFTWARE LUXURY BEAUTY SPORTS AND OUTDOORS MAGAZINE SUBSCRIPTIONS MOVIES AND TV MUSICAL INSTRUMENTS OFFICE PRODUCTS PATIO LAWN AND GARDEN PET SUPPLIES PRIME PANTRY SOFTWARE SPORTS AND OUTDOORS Appl. Sci. 2021, 11, 1733 15 of 17 the immanent features from implicit logged feedback, which is much more naturally collectable. Externally, it derives an intuitive interpretation of the relation between the recommender system and profitability. In other words, it helps find an industrial value as well as a better solution that makes users click on more items or dwell longer on platforms. In future work, we plan to extend this approach both to create other types of saliency in metadata and to fertilize the graph representation by incorporating various user informa- tion, e.g., gender, age, and occupation. We believe re-enrichment learning to be promising for industrial applications. Author Contributions: Y.B. conceived, designed, and performed the experiments; Y.B. and K.L. analyzed the data and wrote the paper. All authors have read and agreed to the published version of the manuscript. Funding: This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2020R1G1A1102041). Conflicts of Interest: The authors declare no conflict of interest. Abbreviations The following abbreviations are used in this manuscript: SVD Singular value decomposition BPR Bayesian personalized ranking GCN Graph convolution network DKN Deep knowledge-aware network NPA Neural news recommendation with personalized attention CDT Context dimension tree L-FM Light factorization machines W&D Wide and deep CTR Click-through rate References 1. Davidson, J.; Liebald, B.; Liu, J.; Nandy, P.; Van Vleet, T.; Gargi, U.; Gupta, S.; He, Y.; Lambert, M.; Livingston, B.; et al. The YouTube video recommendation system. In Proceedings of the Fourth ACM Conference on Recommender Systems, Barcelona Spain, 26–30 September 2010; pp. 293–296. 2. Amatriain, X.; Basilico, J. Netflix recommendations: Beyond the 5 stars (part 1). Netflix Tech. Blog. 2012, 6. 3. MacKenzie, I.; Meyer, C.; Noble, S. How Retailers Can Keep up with Consumers; McKinsey & Company: New York, NY, USA, 2013; Volume 18. 4. Jagerman, R.; Markov, I.; de Rijke, M. When people change their mind: Off-policy evaluation in non-stationary recommendation environments. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, Melbourne Australia, 11–15 February 2019; pp. 447–455. 5. Gomez-Uribe, C.A.; Hunt, N. The netflix recommender system: Algorithms, business value, and innovation. ACM Trans. Manag. Inf. Syst. (TMIS) 2015, 6, 1–19. [CrossRef] 6. Jannach, D.; Jugovac, M. Measuring the business value of recommender systems. ACM Trans. Manag. Inf. Syst. (TMIS) 2019, 10, 1–23. [CrossRef] 7. Rabiu, I.; Salim, N.; Da’u, A.; Osman, A. Recommender System Based on Temporal Models: A Systematic Review. Appl. Sci. 2020, 10, 2204. [CrossRef] 8. Zehlike, M.; Bonchi, F.; Castillo, C.; Hajian, S.; Megahed, M.; Baeza-Yates, R. Fa* ir: A fair top-k ranking algorithm. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore, 6–10 November 2017; pp. 1569–1578. 9. Kleinberg, J.; Raghavan, M. Selection problems in the presence of implicit bias. arXiv 2018, arXiv:1801.03533. 10. Mehrotra, R.; McInerney, J.; Bouchard, H.; Lalmas, M.; Diaz, F. Towards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fairness & satisfaction in recommendation systems. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, Torino, Italy, 22–26 October 2018; pp. 2243–2251. 11. Stoyanovich, J.; Yang, K.; Jagadish, H. Online set selection with fairness and diversity constraints. In Proceedings of the EDBT Conference, Vienna, Austria, 26–29 March 2018; pp. 241–252. 12. Zuiderveen Borgesius, F.; Trilling, D.; Möller, J.; Bodó, B.; De Vreese, C.H.; Helberger, N. Should we worry about filter bubbles? Internet Policy Rev. J. Internet Regul. 2016, 5, 1–16. [CrossRef] 13. Steck, H. Calibrated recommendations. In Proceedings of the 12th ACM Conference on Recommender Systems, Vancouver, BC, Canada, 2–7 October 2018; pp. 154–162. Appl. Sci. 2021, 11, 1733 16 of 17 14. Jannach, D.; Lerche, L.; Kamehkhosh, I.; Jugovac, M. What recommenders recommend: An analysis of recommendation biases and possible countermeasures. User Model. User Adapt. Interact. 2015, 25, 427–491. [CrossRef] 15. Lee, D.; Hosanagar, K. How do recommender systems affect sales diversity? A cross-category investigation via randomized field experiment. Inf. Syst. Res. 2019, 30, 239–259. [CrossRef] 16. Shah, K.; Salunke, A.; Dongare, S.; Antala, K. Recommender systems: An overview of different approaches to recommendations. In Proceedings of the 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), Coimbatore, India, 17–18 March 2017; pp. 1–4. 17. Patel, B.; Desai, P.; Panchal, U. Methods of recommender system: A review. In Proceedings of the 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), Coimbatore, India, 17–18 March 2017; pp. 1–4. 18. Hu, Y.; Koren, Y.; Volinsky, C. Collaborative filtering for implicit feedback datasets. In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, Pisa, Italy, 15–19 December 2008; pp. 263–272. 19. Chen, M.; Beutel, A.; Covington, P.; Jain, S.; Belletti, F.; Chi, E.H. Top-k off-policy correction for a REINFORCE recommender system. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, Melbourne, Australia, 11–15 February 2019; pp. 456–464. 20. Gu, Y.; Chang, J.; Zhang, Y.; Wang, Y. An element sensitive saliency model with position prior learning for web pages. In Proceedings of the 2019 3rd International Conference on Innovation in Artificial Intelligence, Suzhou, China, 15–18 March 2019; pp. 157–161. 21. Shen, C.; Huang, X.; Zhao, Q. Predicting eye fixations on webpage with an ensemble of early features and high-level representa- tions from deep network. IEEE Trans. Multimed. 2015, 17, 2084–2093. [CrossRef] 22. Gupta, P.; Gupta, S.; Jayagopal, A.; Pal, S.; Sinha, R. Saliency prediction for mobile user interfaces. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 1529–1538. 23. Li, J.; Su, L.; Wu, B.; Pang, J.; Wang, C.; Wu, Z.; Huang, Q. Webpage saliency prediction with multi-features fusion. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 674–678. 24. Jiang, M.; Huang, S.; Duan, J.; Zhao, Q. Salicon: Saliency in context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1072–1080. 25. Bobadilla, J.; Ortega, F.; Hernando, A.; Gutiérrez, A. Recommender systems survey. Knowl.-Based Syst. 2013, 46, 109–132. [CrossRef] 26. Koren, Y.; Bell, R.; Volinsky, C. Matrix factorization techniques for recommender systems. Computer 2009, 42, 30–37. [CrossRef] 27. Rendle, S.; Freudenthaler, C.; Gantner, Z.; Schmidt-Thieme, L. BPR: Bayesian personalized ranking from implicit feedback. arXiv 2012, arXiv:1205.2618. 28. He, X.; Deng, K.; Wang, X.; Li, Y.; Zhang, Y.; Wang, M. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. arXiv 2020, arXiv:2002.02126. 29. Kipf, T.N.; Welling, M. Semi-supervised classification with graph convolutional networks. arXiv 2016, arXiv:1609.02907. 30. Wang, H.C.; Jhou, H.T.; Tsai, Y.S. Adapting topic map and social influence to the personalized hybrid recommender system. Inf. Sci. 2018. [CrossRef] 31. Wu, C.; Wu, F.; An, M.; Huang, J.; Huang, Y.; Xie, X. NPA: Neural news recommendation with personalized attention. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA, 4–8 August 2019; pp. 2576–2584. 32. Colace, F.; Lemma, S.; Lombardi, M.; Pascale, F. A Context Aware Approach for Promoting Tourism Events: The Case of Artist’s Lights in Salerno. ICEIS 2017, 2, 752–759. 33. Casillo, M.; Colace, F.; Lemma, S.; Lombardi, M.; Pascale, F. A Tailor made System for Providing Personalized Services. In Proceedings of the Twenty-Ninth International Conference on Software Engineering and Knowledge Engineering (SEKE 2017), Pittsburgh, PA, USA, 5–7 July 2017; pp. 495–500. 34. Singh, R.; Rani, A.; Chuchra, K. A Survey on the Generation of Recommender Systems. Int. J. Inf. Eng. Electron. Bus. 2017, 9, 26–35. [CrossRef] 35. Kula, M.; Bogers, T.; Koolen, M. (Eds.) Metadata Embeddings for User and Item Cold-start Recommendations. In Proceedings of the 2nd Workshop on New Trends on Content-Based Recommender Systems co-located with 9th ACM Conference on Recommender Systems (RecSys 2015), Vienna, Austria, 16–20 September 2015; Volume 1448, pp. 14–21. 36. Cheng, H.T.; Koc, L.; Harmsen, J.; Shaked, T.; Chandra, T.; Aradhye, H.; Anderson, G.; Corrado, G.; Chai, W.; Ispir, M.; et al. Wide & deep learning for recommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, Boston, MA, USA, 15 September 2016; pp. 7–10. 37. Schnabel, T.; Swaminathan, A.; Frazier, P.I.; Joachims, T. Unbiased comparative evaluation of ranking functions. In Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval, Newark, DE, USA, 12–16 September 2016; pp. 109–118. 38. Bonner, S.; Vasile, F. Causal embeddings for recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems, Vancouver, BC, Canada, 2–7 October 2018; pp. 104–112. 39. Chaudhari, S.; Azaria, A.; Mitchell, T. An entity graph based recommender system. AI Commun. 2017, 30, 141–149. [CrossRef] 40. Chiaselotti, G.; Ciucci, D.; Gentile, T. Simple undirected graphs as formal contexts. In International Conference on Formal Concept Analysis; Springer: Nerja, Spain, 23–26 January 2015; pp. 287–302. Appl. Sci. 2021, 11, 1733 17 of 17 41. Zhao, Q.; Koch, C. Learning a saliency map using fixated locations in natural scenes. J. Vis. 2011, 11, 9. [CrossRef] 42. Godil, A.; Wagan, A.I. Salient local 3D features for 3D shape retrieval. In Three-Dimensional Imaging, Interaction, and Measurement; International Society for Optics and Photonics: San Francisco, CA, USA, 4–27 January 2011; Volume 7864, p. 78640S. 43. Harper, F.M.; Konstan, J.A. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. (Tiis) 2015, 5, 1–19. [CrossRef] 44. McAuley, J.; Targett, C.; Shi, Q.; Van Den Hengel, A. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, 9–13 August 2015; pp. 43–52. 45. He, R.; McAuley, J. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, Montréal, QC, Canada, 11–15 April 2016; pp. 507–517. 46. Haruna, K.; Akmar Ismail, M.; Suhendroyono, S.; Damiasih, D.; Pierewan, A.C.; Chiroma, H.; Herawan, T. Context-aware recommender system: A review of recent developmental process and future research direction. Appl. Sci. 2017, 7, 1211. [CrossRef] 47. Feng, Y.; Lv, F.; Hu, B.; Sun, F.; Kuang, K.; Liu, Y.; Liu, Q.; Ou, W. MTBRN: Multiplex Target-Behavior Relation Enhanced Network for Click-Through Rate Prediction. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, Online, 19–23 October 2020; pp. 2421–2428. 48. Zhang, D.; Liu, L.; Wei, Q.; Yang, Y.; Yang, P.; Liu, Q. Neighborhood Aggregation Collaborative Filtering Based on Knowledge Graph. Appl. Sci. 2020, 10, 3818. [CrossRef] 49. Huang, R.; McIntyre, S.; Song, M.; Ou, Z. An Attention-Based Latent Information Extraction Network (ALIEN) for High-Order Feature Interactions. Appl. Sci. 2020, 10, 5468. [CrossRef] 50. Schütze, H.; Manning, C.D.; Raghavan, P. Introduction to Information Retrieval; Cambridge University Press: Cambridge, UK, 2008; Volume 39. 51. Kohavi, R.; Longbotham, R. Online Controlled Experiments and A/B Testing. Encycl. Mach. Learn. Data Min. 2017, 7, 922–929. 52. Wilcoxon, F. Individual comparisons by ranking methods. In Breakthroughs in Statistics; Springer: New York, NY, USA, 1992; pp. 196–202. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Sciences Multidisciplinary Digital Publishing Institute

Re-Enrichment Learning: Metadata Saliency for the Evolutive Personalization of a Recommender System

Applied Sciences , Volume 11 (4) – Feb 16, 2021

Loading next page...
 
/lp/multidisciplinary-digital-publishing-institute/re-enrichment-learning-metadata-saliency-for-the-evolutive-aAVisp3dr7
Publisher
Multidisciplinary Digital Publishing Institute
Copyright
© 1996-2021 MDPI (Basel, Switzerland) unless otherwise stated Disclaimer The statements, opinions and data contained in the journals are solely those of the individual authors and contributors and not of the publisher and the editor(s). MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Terms and Conditions Privacy Policy
ISSN
2076-3417
DOI
10.3390/app11041733
Publisher site
See Article on Publisher Site

Abstract

applied sciences Article Re-Enrichment Learning: Metadata Saliency for the Evolutive Personalization of a Recommender System 1,† 2, Yuseok Ban and Kyungjae Lee * Agency for Defense Development, Daejeon 34068, Korea; ban@add.re.kr Department of Computer Science, Yong In University, Yong In 17092, Korea * Correspondence: kjlee@yiu.ac.kr; Tel.: +82-31-8020-3696 † First author: Tel.: +82-44-998-8148. Abstract: Many studies have been conducted on recommender systems in both the academic and industrial fields, as they are currently broadly used in various digital platforms to make personalized suggestions. Despite the improvement in the accuracy of recommenders, the diversity of interest areas recommended to a user tends to be reduced, and the sparsity of explicit feedback from users has been an important issue for making progress in recommender systems. In this paper, we introduce a novel approach, namely re-enrichment learning, which effectively leverages the implicit logged feedback from users to enhance user retention in a platform by enriching their interest areas. The approach consists of (i) graph-based domain transfer and (ii) metadata saliency, which (i) find an adaptive and collaborative domain representing the relations among many users’ metadata and (ii) extract attentional features from a user ’s implicit logged feedback, respectively. The experimental results show that our proposed approach has a better capacity to enrich the diversity of interests of a user by means of implicit feedback and to help recommender systems achieve more balanced personalization. Our approach, finally, helps recommenders improve user retention, i.e., encouraging users to click more items or dwell longer on the platform. Citation: Ban, Y.; Lee, K. Keywords: recommender system; personalization; diversity; sparsity; re-enrichment learning; graph- Re-Enrichment Learning: Metadata based domain transfer; metadata saliency Saliency for the Evolutive Personalization of a Recommender System. Appl. Sci. 2021, 11, 1733. https://doi.org/10.3390/app11041733 1. Introduction Academic Editors: Kwan-Hee Yoo, Online contents and services have been rapidly growing in recent decades. To help Carson K. Leung and Nakhoon Baek users make decisions when faced with overwhelming options and to achieve commercial Received: 17 January 2021 success in making profit, both academic and industrial research on finding specific sets Accepted: 7 February 2021 of items that meet the personalized interests of users has been intensely conducted. For Published: 16 February 2021 this purpose, recommender systems have been suggested and designed that are based on a user ’s history, including whether the user has bought an item, which ratings the user Publisher’s Note: MDPI stays neu- has given to items, etc. Previous studies have reported that recommendations account for tral with regard to jurisdictional clai- 60% of the clicks on the main screen of Youtube [1], 75% of what people watch through ms in published maps and institutio- Netflix [2], and 35% of the sales through Amazon [3]. nal affiliations. Recommendation is obviously not an easy task. A recommender system needs to provide a personalized user experience with long-term satisfaction under the condition that users’ preferences can change over time [4]. Behaviors based on the preferences of users could be either positively or negatively influenced by recommendation results. Tradi- Copyright: © 2021 by the authors. Li- censee MDPI, Basel, Switzerland. tionally, the recommendation problem has been considered as equivalent to the problem This article is an open access article of precisely predicting the rating that a user would leave on an item. However, recent distributed under the terms and con- academic studies and industrial providers have both emphasized that one of the main ditions of the Creative Commons At- measurement targets is improved user retention [5,6]. In spite of the gains in theoretical tribution (CC BY) license (https:// accuracy, it was unclear that the winning strategy in accuracy would result always in creativecommons.org/licenses/by/ increasing business value [7]. 4.0/). Appl. Sci. 2021, 11, 1733. https://doi.org/10.3390/app11041733 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, 1733 2 of 17 Still, recommender systems are mostly optimized toward the accuracy of predictions of item ratings, and are likely to gradually narrow down the interest area of a user—the so-called the diversity problem [8–11]. That is to say, recommendations where the lesser interests get squeezed out by the main interest ironically help the system minimize prediction errors. This not only impedes the diversity of experiences of users, but also causes filter bubbles [12,13], e.g., a reduced spectrum of user consumption and the political bias, which limit discovery or neglect the potential for promoting new items from the long tail [14,15]. Moreover, as the training of a recommender system mostly relies on explicit feedback (typically the ratings of items), it is inevitable to face the fact that only a small portion of users leave their ratings—the so-called the sparsity problem [16,17]. In this paper, we propose a novel data interpretation approach, which includes graph- based domain transfer and metadata saliency, for recommender systems, and show its effective application for a personalized recommender, namely re-enrichment learning. To address the problems of diversity and sparsity, the goal of re-enrichment learning is to find conspicuous features of the implicit logged feedback [18,19] that tell the behavior of a user, e.g., how frequently the user clicks on a certain item or how long the user stays on the platform, and to promote immanently correlated recommendations. Unlike explicit feedback, implicit logged feedback is relatively easier to spontaneously collect from users. Metadata saliency describes the confidence of a recommendation in the transferred domain, which reflects the graph-based relationships among items. It is generally known that the visual saliency represents the position-based visual preference [20]. As such, methods for predicting eye fixation maps on websites or mobile interfaces [21–23] have been studied, and the relation between the probability of the movement of a mouse pointer and the eye fixation has been discussed [24]. Inspired by those works, metadata saliency is a novel approach to interest fixation for distinguishing a certain set of items from the others. Based on the establishment of links between one group of items and their closely related ones, re-enrichment learning is used to train a recommender system based on the metadata saliency, thus enriching the potential interest areas of a user rather than narrowing them down. The experimental results show that our approach is effective in interpreting implicit logged feedback and helpful in persuading a user to click more or stay longer on a commercial platform. The organization of the rest of this paper is as follows: Section 2 presents a review of the related work, Section 3 describes our proposed approach in detail, and Section 4 provides the comparative experiments and the discussion. Finally, our conclusion is given in Section 5. 2. Background Recommender systems can be largely classified into three categories: the collaborative approach, content-based approach, and hybrid approach [16,17,25]. Firstly, the collabora- tive approach is based on the idea that people with similar preferences are likely to agree on an evaluation of an item. This approach consists of neighborhood-based methods, which directly use the users’ item ratings to estimate the classification of a new item, and model- based methods, which use the ratings to acquire knowledge and teach a predictive model. The advantage of this approach is that new data can easily be added in an incremental manner, whereas its weakness to the cold-start problem of new users and the sparsity problem are its disadvantages. Koren et al. (SVD) [26] suggested a singular value decompo- sition (SVD)-based matrix factorization algorithm that uses stochastic gradient descent for optimization. Rendle et al. (Bayesian personalized ranking (BPR)) [27] introduced a generic optimization criterion and learning algorithm for personalizing a recommender system. The optimization criterion is based on the maximum posterior estimator derived from Bayesian analysis, while the learning algorithm is based on stochastic gradient descent with bootstrap sampling. He et al. (L-GCN) [28] adopted the neighborhood aggregation of a graph convolution network (GCN) [29] and proposed learning user and item embeddings by linearly propagating them with respect to the interaction graph of the user and item. Appl. Sci. 2021, 11, 1733 3 of 17 They used the weighted summation of the embeddings, which are learned at all layers, as in the final stage. Then, the content-based approach is based on the combination of a user ’s profile and an item description. In other words, the similarity between the items that the user has liked in the past and the detailed content of an item play important roles. This approach shows strength when recommending an item that is not yet rated by any user, whereas it suffers from the overspecialization problem. Wang et al. (deep knowledge-aware network (DKN)) [30] proposed a multi-channel and word-entity-aligned knowledge-aware convo- lutional neural network for semantic and knowledge-based representation of news as well as an attention module to dynamically compute the aggregated historical representation of a user. Wu et al. (neural news recommendation with personalized attention (NPA)) [31] suggested personalized convolutional neural networks that adopt the embedding of user IDs as the queries of the correlations between words and news. The meta-context dimen- sion tree (Meta CDT) was proposed for the selection of the most suitable contents and services for a user in a certain context, and was used in a practical context-aware applica- tion by Colace et al. [32] and Casillo et al. [33]. The approach has strength in tailoring the information domain according to the user ’s needs as well as in analyzing relevant features of context models. Lastly, there also are hybrid approaches that combine the collaborative and content- based approaches to exploit the advantages of the individual approaches [16,34]. Many types have been proposed, such as by aggregating the predictions of collaborative and content-based approaches or integrating one approach’s characteristics into the other. Kula (light factorization machine (L-FM)) [35] proposed a hybrid matrix factorization model that represents users and items as linear combinations of the latent factors of their content features. Cheng et al. (wide and deep (W&D)) [36] introduced a framework for jointly training feed-forward neural networks with embedding and a linear model based on feature transformation for generic recommenders with sparse inputs. Many of the recent studies have discussed the problem that unbalanced or biased recommendations can shrink the diversity of interests of a user. Practical evaluation scenarios and techniques have been introduced to produce unbiased estimators in spite of biased or missing data [37,38]. Research has been carried out on adopting the relationships among items or entities as well as among users [39]. For instance, each user has an individual graph-based representation for their explicit feedback of ratings, and the graph can provide insights into how good use of the hidden information can be made in terms of personalized recommendations. Still, most of the previous approaches have difficulty with the aforementioned sparsity and diversity problems. For the sparsity problem, we need to further study how to incorporate the naturally obtainable feedback information from users. In addition, there must be many alternatives to alleviate the diversity problem resulting from unbalanced recommendations. 3. Re-Enrichment Learning The two main objectives of re-enrichment learning are literally to enrich the interest areas recommended to a user and to achieve them by extracting meaningful features from naturally acquirable feedback data. In other words, re-enrichment learning is directly designed to tackle the diversity and sparsity problems of a recommender system. Our proposed approach is composed of two cores: graph-based domain transfer and metadata saliency. The domain of metadata saliency is determined by the graph-based domain transfer, while the graph itself can be changed recursively by the metadata saliency from new feedback. To be more concrete, every implicit logged feedback of a user causes a change in the user ’s metadata saliency; the update in the metadata saliency is promptly reflected in the user ’s graph description, as well as in the universal domain determined from a large set of users; the new environment of graph again recursively influences the next feedback of the user. Appl. Sci. 2021, 11, 1733 4 of 17 3.1. Graph-Based Domain Transfer An undirected weighted graph is assigned to every user in order to build a data structure that consists of item categories as the finite set of nodes and the similarities between categories as the finite set of edges. G = (V , E ) s.t. k2f1, ..., Ng and t  0 (1) k,t k,t k,t V = fv (1), v (2), ..., v ( M)g (2) k,t k,t k,t k,t E = fe (1, 2), e (1, 3), ..., e ( M 1, M)g (3) k,t k,t k,t k,t th As noted in Equations (1)–(3), the graph of the k user among N users at time t, G , is k,t made up of M nodes V with their attributes v and C edges E with their attributes k,t k,t 2 k,t e . The edge attribute between the two nodes v (i) and v (j) is e (i, j), which can be k,t k,t k,t k,t computed as in Equation (4), where i, j 2 f1, ..., Mg. The attribute of an edge represents the combination of the influences and similarity of two nodes. A node is determined to be influential when its attribute is large, and two nodes are determined to be similar when they share similar levels of attributes to that of the same user. Here, the attribute of a node is the metadata saliency, which represents the attentional prominence of the node from the entire set of nodes. Equation (4) is composed of two terms: (i) the likelihood for the influence of a node and (ii) the weight for the similarity between two nodes. A softmax function has been adopted in the first term to produce the influence of a node as a probabilistic value within [0, 1] (see Equations (5) and (6)), where h 2 f1, ..., Mg, and a is the influence factor, which determines the degree of importance of an individual feedback (s.t. a  1). In the second term, the inversed substitution between two nodes’ attributes is adopted to produce the similarity as a weight value (see Equations (7) and (8)). The attributes of nodes have been normalized to remove scale dependency. (4) e (i, j) = s a v (i)  1 v (i, j) k,t k,t k,t av (i) k,t s(a v (i)) = (5) k,t av (h) k,t å e h=1 v (i) min v (h) k,t k,t 8h v (i) = (6) k,t max v (h) min v (h) k,t k,t 8h 8h 4 4 v (i, j) min v (i, h) k,t k,t 8h v (i, j) = (7) k,t 4 4 max v (i, h) min v (i, h) k,t k,t 8h 8h (8) v (i, j) = jv (i) v (j)j k,t k,t k,t Then, the edges can be denoted by an jV j × jV j matrix, i.e., adjacency matrix k,t k,t E = e (i, j) [40], where each element indicates the attribute of an edge, and the elements k,t k,t can be stored in a triangular matrix as e (i, j) = e (j, i) (as shown in Figure 1). Finally, k,t k,t the attributes of nodes and edges, as well as the adjacent matrix that allows their sequential representation, are able to respond to every feedback event. Appl. Sci. 2021, 11, 1733 5 of 17 𝑛𝑛𝑛𝑛 𝑟𝑟𝑛𝑛 𝑠𝑠 𝑡 1 2 3 ⋯ 𝑀𝑀 ⋯ 0 1 𝑒𝑒 (1,1) 𝑘𝑘 ,𝑠𝑠 𝑛𝑛𝑛𝑛 𝑒𝑒 (2,1) 𝑒𝑒 (2,2) ⋰ ⋮ 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 𝑟𝑟𝑛𝑛 𝑒𝑒 (3,1) 𝑒𝑒 (3,2) 𝑒𝑒 (3,3) 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 ⋮ ⋮ ⋮ ⋮ ⋱ 𝑠𝑠 𝑡 𝑒𝑒 (𝑀𝑀 , 1) 𝑒𝑒 (𝑀𝑀 , 2) 𝑒𝑒 (𝑀𝑀 , 3) 𝑒𝑒 (𝑀𝑀 ,𝑀𝑀 ) 𝑀𝑀 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 𝑘𝑘 ,𝑠𝑠 Figure 1. Adjacency matrix E . k,t In the end, a universal domain D at time t can be determined based on the graphs of th N users, where D (h) is the h domain in which M nodes are sorted by the level of the th attribute of the edge connected to the h node (see Equation (9)). That is to say, in D (h), the th h node itself appears first, the node with the second largest attribute of the edge connected th to the h node appears second, and so on. Understandably, the domain changes over time as the individual graphs vary—e.g., with fluctuating attributes, newly inserted nodes starting with zero-level attributes, etc. Empirically, two nodes that have large attributes and share the edge of a large attribute become adjacent neighbors in the transferred domain. Here, being neighbors has dire consequences with respect to metadata saliency. D = fD (1), D (2), ..., D ( M)g (9) t t t t 3.2. Metadata Saliency A user ’s behavioral history can be used not only as primary cumulative data, but also as information from which secondary features are extracted. Metadata saliency is one of the secondary features that can play an important role, especially in leveraging implicit logged feedback data. A few interesting items usually account for the majority of interest fixation, and they can be quantified by extracting metadata saliency from the logged feedback. Among the various types of feedback, we focus on the ones that contain the retention data of a user, and we start with the simplest type: the clicks on recommended th items. The metadata saliency of the i category in the transferred domain at time t, v (i ), k,t d th is computed based on the i clicked category, as in Equation (10). A Gaussian fixation model has been adopted [21,24,41,42], of which the average and standard deviation are i th and 1, respectively, in order to reinforce the metadata saliency of the i category, but also to enrich its nearest neighbors with respect to the transferred domain. The Gaussian fixation th is also scaled based on the metadata saliency of the i category at time t 1, v (i ), k,t1 d and the influence factor a (see Equation (11)). The very first metadata saliency, v , can k,0 be set by a preference survey for new users for the cold start (discussed in Section 4.2.1). Otherwise, for users who refuse the survey, the average of the metadata saliencies of many random users can be an alternative for v (discussed in Section 4.2.2). k,0 i 1  2 d (i i ) (10) 2 d v (i ) = v (i ) + p  e k,t d k,t1 d 2p A = (11) v (i )+a k,t1 d 𝑠𝑠𝑠𝑠 𝑠𝑠𝑠𝑠 Appl. Sci. 2021, 11, 1733 6 of 17 Suppose that a graph has been built to represent the metadata saliencies of movie genres for a user (see Figure 2). A node is presented as a circle, of which the radius indicates its attribute level. An edge is drawn as a line, of which the thickness is proportional to its attribute level. The user must mostly have preferences for Action movies, as well as Sci-Fi movies, because the two nodes corresponding to those genres are the two biggest circles, as shown in Figure 2. Provided that the user clicked an item that belongs to the Action category at time t, not only does the metadata saliency of Action node get larger at time t + 1, but that of the Sci-Fi node also benefits from it to a certain extent due to the connection of the edge to the two nodes and the Gaussian fixation effect. Moreover, the Fantasy and Adventure nodes also deserve a small benefit from the feedback. Because of the new feedback, the attributes of the nodes and edges change simultaneously and incrementally create a new condition in the graph representation. As such, the metadata saliency naturally changes over time by means of the newly collected feedback. The change in metadata saliency is promptly reflected in a user ’s graph and, by extension, also in the universal domain, which is built upon many users’ graphs. Consequently, the evolutive characteristic based on the mutual interaction in the graph makes re-enrichment learning prompt and flexible. Mystery Thriller Animation Fantasy Action Sci-Fi Crime Adventure War Documentary Figure 2. Conceptual visualization of a graph (node: circle radius, edge: line thickness). 3.3. Summary In summary, we provide Figure 3 to depict a diagram that connects all the steps of the process, and the process can be listed as follows: • Step 1: Obtain the user ’s implicit logged feedback i at time t from the recommender. • Step 2: Update nodes V ! V : Calculate node attribute v (i ) at time t using k,t1 k,t k,t d the universal domain D and the node attribute v (i ) at time t 1 (as shown in t1 k,t1 d Equation (10)). • Step 3: Update edges E ! E : Calculate the edge attribute e (i, j) at time t k,t1 k,t k,t using the node attribute v (i ) at time t (as in Equation (4), which consists of the two k,t d following terms). – Compute the likelihood of influence of the node: s a v (i) (see Equation (5)). k,t – Compute the weight for the similarity between two nodes: 1 v (i, j) (see k,t Equation (7)). • Step 4: Update the universal domain D ! D (as in Equation (9)): Sort domain t1 t D (h) by edge attribute. • Step 5: Apply the node attribute v (i ) at time t, i.e., the metadata saliency, to the k,t d recommender. Appl. Sci. 2021, 11, 1733 7 of 17 Furthermore, we provide a pseudocode to effectively present how the different steps of our proposed approach are organized (see Table 1). Obtain Update Update Update Feedback Nodes Edges Universal domain 9 4 7 3 + + 6 8 2 1. Action 1 9 7 4 5 3 8 6 1 2. Adventure + + 3 5 4 7 9 3. Animation 8 6 1 4 7 5 3 4. Crime 2 8 6 1 5 3 4 7 9 2 8 6 5. Documentary 1 9 4 7 5 3 √ 2 𝒊𝒊 6 8 1 6. Fantasy + + + + 𝒅𝒅 → 7 4 9 5 3 7. Mystery 8 6 1 9 4 7 5 3 + + 8 6 1 8. Sci-Fi 7 4 5 3 2 8 6 9. Thriller + 1 9 7 4 5 3 + 2 8 6 10. War 1 Figure 3. The process of re-enrichment learning. Table 1. The pseudocode for re-enrichment learning. Pseudocode Input : Implicit logged feedback i , Nodes V , Edges E , Universal domain D k,t1 k,t1 t1 Output : Nodes V , Edges E , Universal domain D k,t k,t Parameter : Influence factor a / Update Nodes V ! V / k,t1 k,t for k = 1 : N do for i = 1 : M do 1 : Calculate Metadata saliency v (i ) k,t d end for : Construct Nodes V = fv (1), v (2), ..., v ( M)g k,t k,t k,t k,t end for / Update Edges E ! E / k,t1 k,t for k = 1 : N do for i = 1 : M do for j = i : M do 1 : Calculate Likelihood for influence of node s a v (i) k,t 2 : Calculate Weight for similarity between two nodes 1 v (i, j) k,t 3 : Calculate Edge attribute e (i, j) = s a v (i)  1 v (i, j) k,t k,t k,t end for : Construct Edges E = fe (1, 2), e (1, 2), ..., e ( M 1, M)g k,t k,t k,t k,t end for end for / Update Universal domain D ! D / t1 t for h = 1 : M do 1 : Calculate Domain D (h) by sorting nodes w.r.t. edge attribute end for : Construct Universal domain D = fD (1), D (2), ..., D ( M)g t t t t Appl. Sci. 2021, 11, 1733 8 of 17 4. Experiment and Discussion 4.1. Experimental Setup 4.1.1. Dataset • MovieLens dataset [43]: The MovieLens (100k) dataset was collected by the Grou- pLens Research Project at the University of Minnesota (see Figure 4). It consists of 100,000 ratings, which range from 1 to 5, obtained from 943 users on 1682 movies as items. The data were acquired through the MovieLens website during the seven- month collection period. The dataset was cleaned up by excluding users who wrote less than 20 ratings. A movie can belong to more than one genre; there are 18 genres, defined as: (i) Action, (ii) Adventure, (iii) Animation, (iv) Children’s, (v) Comedy, (vi) Crime, (vii) Documentary, (viii) Drama, (ix) Fantasy, (x) Film-Noir, (xi) Horror, (xii) Musical, (xiii) Mystery, (xiv) Romance, (xv) Sci-Fi, (xvi) Thriller, (xvii) War, and (xviii) Western. • Amazon dataset [44,45]: The Amazon review dataset contains the reviews of items and their metadata collected from the Amazon platform (see Figure 5). The dataset sampled for the experiment included around 70 million ratings of 745,018 products, ranging from 1 to 5. In the experiment, 20 categories were included: (i) Beauty, (ii) Fashion, (iii) Appliances, (iv) Arts, crafts, and sewing, (v) CDs and vinyl, (vi) Cell phones and accessories, (vii) Digital music, (viii) Gift cards, (ix) Grocery and gourmet food, (x) Industrial and scientific, (xi) Luxury beauty, (xii) Magazine subscriptions, (xiii) Movies and TV, (xiv) Musical instruments, (xv) Office products, (xvi) Patio, lawn, and garden, (xvii) Pet supplies, (xviii) Prime pantry, (xix) Software, and (xx) Sports and outdoors. Figure 4. Examples of movies from different genres (MovieLens dataset). Figure 5. Examples of goods from different categories (Amazon dataset). 4.1.2. Evaluation Learning from implicit user feedback, e.g., click and dwell time, has been an important factor in improving recommender systems [18]. The common implicit retention metric [19], the click-through rate (CTR) [6,46–49], was adopted to intuitively and directly evaluate the effectiveness of the methods for user satisfaction or retention on digital platforms. Specifically, the metric was used to measure the ratio of clicks to recommendations, as in Equation (12). Appl. Sci. 2021, 11, 1733 9 of 17 Number of click-throughs (12) CT R =  100 (%) Number of recommendations Furthermore, the metric of mean average precision [50], M AP@K, supports the consis- tency of an experiment. This metric indicates how many items a user engages with among those recommended to them, as CTR does, but also uses the sequence of feedback (see Equation (13), where S indicates the number of samples). To begin with, average precision, AP@K, is defined as the summation of the engaged precision values divided by the number of engagements, m, where P(i) is the precision at i and d(i) is a bivariate function for engagement (as in Equations (14)–(16)). M AP@K = ( AP@K) (13) n=1 AP@K = P(i) d(i) (14) i=1 Number of relevant recommendations P(i) = (15) Number of recommendations th 1 if i recommendation is engaged, d(i) = (16) 0 otherwise. A/B experiments [5,19,51] were conducted to verify the difference between applying and not applying re-enrichment learning to baseline methods. Five baseline methods are used for comparison: SVD [26], BPR [27], L-GCN [28], L-FM [35], and W&D [36]. The baseline methods are either collaborative or hybrid approaches because our proposed method treats the category preferences of users, e.g., movie genre and product category, rather than the internal content of an item, e.g., the words in a news article. We collected data from 32 participants, who clicked on all the items that seemed to be interesting. For fairness, the participants were informed of which method was used in each experiment. The time required to finish one sequence of collecting implicit feedback ranged from 10 to 30 min depending on the participant. The sequence of collecting feedback was repeated four times for every participant. A total of 4532 feedbacks were collected for the MovieLens dataset and 4830 feedbacks were collected for Amazon dataset were collected. Lastly, a non-parametric statistical hypothesis test, Wilcoxon test [52], was performed to check the significance of improvements. We adopted a significance level a of 0.05, which indicates a 5% risk of concluding that a difference exists when there is no actual difference. The interpretation of a result in the Wilcoxon test is that the null hypothesis, “The population median (h) equals the hypothesized median (h )”, is wrong when the p-value is smaller than the significance level (i.e., the difference is significant). 4.1.3. Configuration The baseline methods were applied with the settings of default parameters given in their open sources. Based on the scaling property of signal processing, the influence factor a in Equation (4), which operates as the scaling factor of the softmax function, was identically set to 4 in all cases. This parameter determines the agility of re-enrichment learning in adapting to recent short-term feedback. 4.2. Results and Discussion 4.2.1. Recommending Movies At the beginning, users were asked to respond to a survey on their preferred categories as a cold start. Otherwise, users could also skip it to simply use the given averaged preference model built upon the feedback collected from other previous users. When Appl. Sci. 2021, 11, 1733 10 of 17 responding to the survey, the selected categories directly created an initial metadata saliency. Then, a set of 10 movies, with their images and titles, was recommended to a user at a time, and the user was allowed to click on any items they were interested in. The user could also skip the entire set and go on to the next set. Table 2 shows our A/B experiments, and it compares the results of all baseline comparison methods with respect to CTR and MAP@K for the MovieLens dataset. As demonstrated in Figure 6, our proposed re-enrichment learning helps baseline methods improve the resulting retention of users by providing a feature that enriches the interest areas of the user instead of overspecializing them. Compared to the intuitive metric, CTR, MAP@K additionally reflects the order in which the user engages with items within a recommended set, so we can infer that our approach has nothing to do with recommending more attractive items in the front. Still, the results of CTR and MAP@K show a consistent tendency for improvement. In addition, Table 3 presents the results of the Wilcoxon test which essentially calculates the difference between sets of paired samples and analyzes these differences to establish if they are statistically significantly different from one another. The resulting p-values in all cases were much smaller than the significance level a, which allows the conclusion that the differences between the population distribution and the hypothesized distribution are statistically significant. A : w / o R e - e n r i c h m e n t l e a r n i n g A : w / o R e - e n r i c h m e n t l e a r n i n g B : w / R e - e n r i c h m e n t l e a r n i n g B : w / R e - e n r i c h m e n t l e a r n i n g 3 0 0 . 4 0 0 . 3 5 0 . 3 0 2 0 K 0 . 2 5 R @ 0 . 2 0 T P C A 0 . 1 5 1 0 0 . 1 0 0 . 0 5 0 0 . 0 0 A B A B A B A B A B A B A B A B A B A B N N D D D R M D R M C C F F P & P & V V G - G - - - S B S B L L W W L L (a) Average click-through rate (b) Mean average precision @ K Figure 6. Visualization of A/B experiments on the MovieLens dataset. Table 2. The results of A/B experiments on the MovieLens dataset. CTR MAP@K Method A B B A (B A)/A A B B A (B A)/A SVD 14.76% 18.52% + 3.76% " 25.47% 0.306 0.339 + 0.033 " 10.91% BPR 17.10% 20.90% + 3.80% " 22.22% 0.318 0.368 + 0.049 " 15.51% L-GCN 17.40% 20.37% + 2.97% " 17.07% 0.341 0.363 + 0.022 " 6.54% L-FM 12.89% 16.91% + 4.02% " 31.19% 0.280 0.329 + 0.049 " 17.56% W&D 14.28% 17.18% + 2.90% " 20.31% 0.290 0.324 + 0.034 " 11.77% This observation is supported by Figure 7, which shows an example comparing the initial metadata saliency of a user to the metadata saliency variations over 30 implicit feedbacks. The x-axis presents the node of a graph (i.e., movie genre), while the y-axis shows the attribute of a node (i.e., the level of metadata saliency). As can be observed in Figure 7a, the user started with two large saliencies at Action and Sci-Fi. Despite the fact that the Action and Sci-Fi genres accounted for the majority of the 30 feedbacks and the two were still ranked in the two largest saliencies, the metadata saliencies of other genres (Adventure and Thriller) prominently increased because they had relatively large Appl. Sci. 2021, 11, 1733 11 of 17 edge attributes with Action and Sci-Fi (see the thicker lines in Figure 7a). We depict all the edges connected to Action as red-colored lines and all the edges connected to Sci-Fi as blue-colored lines, which show that Adventure and Thriller are fairly interrelated with Action and Sci-Fi with respect to the thickness of the lines. It is worth observing that Adventure and Thriller were encouraged to be recommended, and were consequently selected quite a few times. Table 3. The Wilcoxon test of the A/B experiments on the MovieLens dataset. CTR MAP@K Method p-Value a p-Value a SVD 0.000001220 < 0.05 0.003038835 < 0.05 BPR 0.000001947 < 0.05 0.000007521 < 0.05 L-GCN 0.000001472 < 0.05 0.002098150 < 0.05 L-FM 0.000001110 < 0.05 0.000256538 < 0.05 W&D 0.000002135 < 0.05 0.000070817 < 0.05 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 (a) Initial metadata saliency (v ) k,0 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 (b) Metadata saliency after 30 feedbacks (v ) k,30 Figure 7. The effect of implicit feedback on metadata saliency (MovieLens dataset). Collaborative approaches find new items in which a particular user is most likely to be interested by modeling the task as a regression or classification using the rating data. By being provided with a recommendation of an item that satisfies them, the user is likely Action Adventure Animation Children's Comedy Crime Documentary Drama Fantasy Action Film-Noir Adventure Horror Animation Musical Children's Mystery Comedy Romance Crime Sci-Fi Documentary Thriller Drama War Fantasy Western Film-Noir Horror Musical Mystery Romance Sci-Fi Thriller War Western Appl. Sci. 2021, 11, 1733 12 of 17 to engage more with the platform. By replacing the rating with metadata saliency, the retention results of collaborative approaches were increased. This observation directly shows that users spent more time exploring the enriched interest areas. In addition, only small subsets of movies from the available set of movies were rated by users [16], which made it difficult to successively collect explicit cues for training. In contrast, metadata saliency is based on implicit logged feedback, which can be naturally and continuously obtained. Moreover, hybrid approaches already merge different techniques or features to avoid limitations, but they also can be further assisted by re-enrichment learning. Based on our graph-based domain transfer, the relations between the categories were effectively incorporated, and that allowed the metadata saliency to emphasize the salient interest areas of the user, as well as the interest areas implicitly linked to them. As a result, applying re-enrichment learning led to significantly improved click-through rates for the hybrid approaches. L-FM is based on the latent representation approach, which aims to learn user and item representations from the interaction data [35], while the wide linear model of W&D aims to memorize sparse feature interactions based on cross-product feature transformation [36]. These characteristics allow L-FM and W&D to be run together with graph-based domain transfer and metadata saliency. 4.2.2. Recommending Goods Likewise, the preference survey was followed by a set of 10 recommendations, and users could click on any items they wished. Experiments were carried out using the Amazon dataset [44,45], which contains Amazon products’ review data as well as the products’ metadata. Table 4 reports the results of our A/B experiments, and Figure 8 demonstrates their visual comparisons. We observed that the methods with re-enrichment learning outperformed their baselines, generally producing increased CTR and MAP@K. The results of the Wilcoxon test on the Amazon dataset show more or less the same tendency as those of the MovieLens dataset. As shown in Table 5, the p-values were fairly smaller in all cases than the significance level a, which means that the differences between A and B were significant. A : w / o R e - e n r i c h m e n t l e a r n i n g A : w / o R e - e n r i c h m e n t l e a r n i n g B : w / R e - e n r i c h m e n t l e a r n i n g B : w / R e - e n r i c h m e n t l e a r n i n g 3 0 0 . 4 0 0 . 3 5 2 5 0 . 3 0 2 0 0 . 2 5 1 5 0 . 2 0 T P C A 0 . 1 5 1 0 0 . 1 0 0 . 0 5 0 0 . 0 0 A B A B A B A B A B A B A B A B A B A B N N D D D R M D R M C C P F & P F & V V G - G - S B - S B - L W L W L L (a) Average click-through rate (b) Mean average precision @ K Figure 8. Visualization of A/B experiments on the Amazon dataset. Appl. Sci. 2021, 11, 1733 13 of 17 Table 4. The results of A/B experiments on the Amazon dataset. CTR MAP@K Method A B B A (B A)/A A B B A (B A)/A SVD 15.43% 19.21% + 3.78% " 24.50% 0.303 0.340 + 0.037 " 12.21% BPR 16.16% 19.03% + 2.87% " 17.76% 0.313 0.337 + 0.024 " 7.61% L-GCN 16.46% 19.56% + 3.10% " 18.83% 0.315 0.341 + 0.027 " 8.46% L-FM 11.79% 15.01% + 3.22% " 27.31% 0.247 0.296 + 0.049 " 19.75% W&D 11.52% 15.34% + 3.82% " 33.16% 0.242 0.285 + 0.043 " 17.71% Table 5. The Wilcoxon test of the A/B experiments on the Amazon dataset. CTR MAP@K Method p-Value a p-Value a SVD 0.000000834 < 0.05 0.000029252 < 0.05 BPR 0.000001340 < 0.05 0.000600795 < 0.05 L-GCN 0.000000833 < 0.05 0.000643566 < 0.05 L-FM 0.000000916 < 0.05 0.000043941 < 0.05 W&D 0.000001110 < 0.05 0.000104444 < 0.05 Figure 9 demonstrates an example of the variations in metadata saliency over 30 feedbacks by a user. This example is interesting because the user did not participate in the preference survey at the beginning. As such, the average metadata saliency from 70M previous feedbacks was used for the initial metadata saliency (see Figure 9a). The two categories, Digital music and Gift cards, initially obtained relatively large metadata saliencies. On the other hand, the user mostly selected Appliances, Cell phones and accessories, Industrial and scientific, and especially Sports and outdoors. Consequently, the metadata saliency levels of those categories increased during the latter feedback (see Figure 9b). The rest of the categories moved opposite to the most selected ones, yet the saliency levels, of which the edge attribute was large (see the thicker lines in Figure 9a), were rather slightly raised. Re-enrichment learning noticeably raised the click-through rates of the collaborative approaches. The advantage of collaborative approaches is that adding new data in an in- cremental manner is relatively easy, and re-enrichment learning is quite well matched with this advantage. It is easy to add a new node and its edges, starting with initial attributes set to zero. Furthermore, our proposed approach supplements the weakness of collaborative approaches when very few explicit feedbacks, i.e., ratings, are limitedly collectable from users. Furthermore, we observed that hybrid approaches with re-enrichment learning also outperformed their baselines. Re-enrichment learning effectively exploits the implicit logged feedback information and led to better click-through rate results. The observation from Figure 9 intuitively conveys that the interest areas that have close relations in the graph-based transferred domain to the areas selected by a user were nourished due to re-enrichment learning. In consequence, the recommendations based on re-enrichment learning encouraged users to consume more content. Appl. Sci. 2021, 11, 1733 14 of 17 0.08 0.06 0.04 0.02 (a) Initial metadata saliency (v ) k,0 0.08 0.06 0.04 0.02 (b) Metadata saliency after 30 feedbacks (v ) k,30 Figure 9. The effect of implicit feedback on metadata saliency (Amazon dataset). 5. Conclusions In this paper, we introduced a novel approach that effectively exploits the implicit logged feedback from users to enrich their potential interest areas and to increase user retention in recommender systems for digital platforms. Our approach includes graph- based domain transfer and metadata saliency, which are incorporated to operate as re- enrichment learning. A universal domain was built based on the graph representation of item categories and their interrelations as nodes and edges, respectively. While capturing the attentional prominence of a node, metadata saliency also confers benefits on the nearest neighbors of the node in the universal domain. Every implicit logged feedback of a user causes a change in the user ’s metadata saliency; the update in the metadata saliency is promptly reflected in the user ’s graph description, as well as in the universal domain determined by a number of users; the new environment of graph again recursively influences the next feedback of the user. The eventual goal is the improvement of user retention, rather than accurately predicting ratings, as there are many better ways to help people find interesting items than focusing only on those with high predictions of ratings. It should be pointed out that re-enrichment learning shows a constant tendency of internal and external advantages. Internally, it has a better capacity to enrich the diversity of possible interest areas of users and to help recommender systems achieve more balanced personalization. In addition, to address the issue of sparsity of explicit feedback, it extracts ALL BEAUTY AMAZON FASHION APPLIANCES ARTS CRAFTS AND SEWING CDS AND VINYL CELL PHONES AND ACCESSORIES DIGITAL MUSIC GIFT CARDS GROCERY AND GOURMET FOOD ALL BEAUTY INDUSTRIAL AND SCIENTIFIC AMAZON FASHION LUXURY BEAUTY APPLIANCES MAGAZINE SUBSCRIPTIONS ARTS CRAFTS AND SEWING MOVIES AND TV CDS AND VINYL MUSICAL INSTRUMENTS CELL PHONES AND ACCESSORIES OFFICE PRODUCTS DIGITAL MUSIC PATIO LAWN AND GARDEN GIFT CARDS PET SUPPLIES GROCERY AND GOURMET FOOD PRIME PANTRY INDUSTRIAL AND SCIENTIFIC SOFTWARE LUXURY BEAUTY SPORTS AND OUTDOORS MAGAZINE SUBSCRIPTIONS MOVIES AND TV MUSICAL INSTRUMENTS OFFICE PRODUCTS PATIO LAWN AND GARDEN PET SUPPLIES PRIME PANTRY SOFTWARE SPORTS AND OUTDOORS Appl. Sci. 2021, 11, 1733 15 of 17 the immanent features from implicit logged feedback, which is much more naturally collectable. Externally, it derives an intuitive interpretation of the relation between the recommender system and profitability. In other words, it helps find an industrial value as well as a better solution that makes users click on more items or dwell longer on platforms. In future work, we plan to extend this approach both to create other types of saliency in metadata and to fertilize the graph representation by incorporating various user informa- tion, e.g., gender, age, and occupation. We believe re-enrichment learning to be promising for industrial applications. Author Contributions: Y.B. conceived, designed, and performed the experiments; Y.B. and K.L. analyzed the data and wrote the paper. All authors have read and agreed to the published version of the manuscript. Funding: This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2020R1G1A1102041). Conflicts of Interest: The authors declare no conflict of interest. Abbreviations The following abbreviations are used in this manuscript: SVD Singular value decomposition BPR Bayesian personalized ranking GCN Graph convolution network DKN Deep knowledge-aware network NPA Neural news recommendation with personalized attention CDT Context dimension tree L-FM Light factorization machines W&D Wide and deep CTR Click-through rate References 1. Davidson, J.; Liebald, B.; Liu, J.; Nandy, P.; Van Vleet, T.; Gargi, U.; Gupta, S.; He, Y.; Lambert, M.; Livingston, B.; et al. The YouTube video recommendation system. In Proceedings of the Fourth ACM Conference on Recommender Systems, Barcelona Spain, 26–30 September 2010; pp. 293–296. 2. Amatriain, X.; Basilico, J. Netflix recommendations: Beyond the 5 stars (part 1). Netflix Tech. Blog. 2012, 6. 3. MacKenzie, I.; Meyer, C.; Noble, S. How Retailers Can Keep up with Consumers; McKinsey & Company: New York, NY, USA, 2013; Volume 18. 4. Jagerman, R.; Markov, I.; de Rijke, M. When people change their mind: Off-policy evaluation in non-stationary recommendation environments. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, Melbourne Australia, 11–15 February 2019; pp. 447–455. 5. Gomez-Uribe, C.A.; Hunt, N. The netflix recommender system: Algorithms, business value, and innovation. ACM Trans. Manag. Inf. Syst. (TMIS) 2015, 6, 1–19. [CrossRef] 6. Jannach, D.; Jugovac, M. Measuring the business value of recommender systems. ACM Trans. Manag. Inf. Syst. (TMIS) 2019, 10, 1–23. [CrossRef] 7. Rabiu, I.; Salim, N.; Da’u, A.; Osman, A. Recommender System Based on Temporal Models: A Systematic Review. Appl. Sci. 2020, 10, 2204. [CrossRef] 8. Zehlike, M.; Bonchi, F.; Castillo, C.; Hajian, S.; Megahed, M.; Baeza-Yates, R. Fa* ir: A fair top-k ranking algorithm. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore, 6–10 November 2017; pp. 1569–1578. 9. Kleinberg, J.; Raghavan, M. Selection problems in the presence of implicit bias. arXiv 2018, arXiv:1801.03533. 10. Mehrotra, R.; McInerney, J.; Bouchard, H.; Lalmas, M.; Diaz, F. Towards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fairness & satisfaction in recommendation systems. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, Torino, Italy, 22–26 October 2018; pp. 2243–2251. 11. Stoyanovich, J.; Yang, K.; Jagadish, H. Online set selection with fairness and diversity constraints. In Proceedings of the EDBT Conference, Vienna, Austria, 26–29 March 2018; pp. 241–252. 12. Zuiderveen Borgesius, F.; Trilling, D.; Möller, J.; Bodó, B.; De Vreese, C.H.; Helberger, N. Should we worry about filter bubbles? Internet Policy Rev. J. Internet Regul. 2016, 5, 1–16. [CrossRef] 13. Steck, H. Calibrated recommendations. In Proceedings of the 12th ACM Conference on Recommender Systems, Vancouver, BC, Canada, 2–7 October 2018; pp. 154–162. Appl. Sci. 2021, 11, 1733 16 of 17 14. Jannach, D.; Lerche, L.; Kamehkhosh, I.; Jugovac, M. What recommenders recommend: An analysis of recommendation biases and possible countermeasures. User Model. User Adapt. Interact. 2015, 25, 427–491. [CrossRef] 15. Lee, D.; Hosanagar, K. How do recommender systems affect sales diversity? A cross-category investigation via randomized field experiment. Inf. Syst. Res. 2019, 30, 239–259. [CrossRef] 16. Shah, K.; Salunke, A.; Dongare, S.; Antala, K. Recommender systems: An overview of different approaches to recommendations. In Proceedings of the 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), Coimbatore, India, 17–18 March 2017; pp. 1–4. 17. Patel, B.; Desai, P.; Panchal, U. Methods of recommender system: A review. In Proceedings of the 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), Coimbatore, India, 17–18 March 2017; pp. 1–4. 18. Hu, Y.; Koren, Y.; Volinsky, C. Collaborative filtering for implicit feedback datasets. In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, Pisa, Italy, 15–19 December 2008; pp. 263–272. 19. Chen, M.; Beutel, A.; Covington, P.; Jain, S.; Belletti, F.; Chi, E.H. Top-k off-policy correction for a REINFORCE recommender system. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, Melbourne, Australia, 11–15 February 2019; pp. 456–464. 20. Gu, Y.; Chang, J.; Zhang, Y.; Wang, Y. An element sensitive saliency model with position prior learning for web pages. In Proceedings of the 2019 3rd International Conference on Innovation in Artificial Intelligence, Suzhou, China, 15–18 March 2019; pp. 157–161. 21. Shen, C.; Huang, X.; Zhao, Q. Predicting eye fixations on webpage with an ensemble of early features and high-level representa- tions from deep network. IEEE Trans. Multimed. 2015, 17, 2084–2093. [CrossRef] 22. Gupta, P.; Gupta, S.; Jayagopal, A.; Pal, S.; Sinha, R. Saliency prediction for mobile user interfaces. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 1529–1538. 23. Li, J.; Su, L.; Wu, B.; Pang, J.; Wang, C.; Wu, Z.; Huang, Q. Webpage saliency prediction with multi-features fusion. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 674–678. 24. Jiang, M.; Huang, S.; Duan, J.; Zhao, Q. Salicon: Saliency in context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1072–1080. 25. Bobadilla, J.; Ortega, F.; Hernando, A.; Gutiérrez, A. Recommender systems survey. Knowl.-Based Syst. 2013, 46, 109–132. [CrossRef] 26. Koren, Y.; Bell, R.; Volinsky, C. Matrix factorization techniques for recommender systems. Computer 2009, 42, 30–37. [CrossRef] 27. Rendle, S.; Freudenthaler, C.; Gantner, Z.; Schmidt-Thieme, L. BPR: Bayesian personalized ranking from implicit feedback. arXiv 2012, arXiv:1205.2618. 28. He, X.; Deng, K.; Wang, X.; Li, Y.; Zhang, Y.; Wang, M. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. arXiv 2020, arXiv:2002.02126. 29. Kipf, T.N.; Welling, M. Semi-supervised classification with graph convolutional networks. arXiv 2016, arXiv:1609.02907. 30. Wang, H.C.; Jhou, H.T.; Tsai, Y.S. Adapting topic map and social influence to the personalized hybrid recommender system. Inf. Sci. 2018. [CrossRef] 31. Wu, C.; Wu, F.; An, M.; Huang, J.; Huang, Y.; Xie, X. NPA: Neural news recommendation with personalized attention. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA, 4–8 August 2019; pp. 2576–2584. 32. Colace, F.; Lemma, S.; Lombardi, M.; Pascale, F. A Context Aware Approach for Promoting Tourism Events: The Case of Artist’s Lights in Salerno. ICEIS 2017, 2, 752–759. 33. Casillo, M.; Colace, F.; Lemma, S.; Lombardi, M.; Pascale, F. A Tailor made System for Providing Personalized Services. In Proceedings of the Twenty-Ninth International Conference on Software Engineering and Knowledge Engineering (SEKE 2017), Pittsburgh, PA, USA, 5–7 July 2017; pp. 495–500. 34. Singh, R.; Rani, A.; Chuchra, K. A Survey on the Generation of Recommender Systems. Int. J. Inf. Eng. Electron. Bus. 2017, 9, 26–35. [CrossRef] 35. Kula, M.; Bogers, T.; Koolen, M. (Eds.) Metadata Embeddings for User and Item Cold-start Recommendations. In Proceedings of the 2nd Workshop on New Trends on Content-Based Recommender Systems co-located with 9th ACM Conference on Recommender Systems (RecSys 2015), Vienna, Austria, 16–20 September 2015; Volume 1448, pp. 14–21. 36. Cheng, H.T.; Koc, L.; Harmsen, J.; Shaked, T.; Chandra, T.; Aradhye, H.; Anderson, G.; Corrado, G.; Chai, W.; Ispir, M.; et al. Wide & deep learning for recommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, Boston, MA, USA, 15 September 2016; pp. 7–10. 37. Schnabel, T.; Swaminathan, A.; Frazier, P.I.; Joachims, T. Unbiased comparative evaluation of ranking functions. In Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval, Newark, DE, USA, 12–16 September 2016; pp. 109–118. 38. Bonner, S.; Vasile, F. Causal embeddings for recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems, Vancouver, BC, Canada, 2–7 October 2018; pp. 104–112. 39. Chaudhari, S.; Azaria, A.; Mitchell, T. An entity graph based recommender system. AI Commun. 2017, 30, 141–149. [CrossRef] 40. Chiaselotti, G.; Ciucci, D.; Gentile, T. Simple undirected graphs as formal contexts. In International Conference on Formal Concept Analysis; Springer: Nerja, Spain, 23–26 January 2015; pp. 287–302. Appl. Sci. 2021, 11, 1733 17 of 17 41. Zhao, Q.; Koch, C. Learning a saliency map using fixated locations in natural scenes. J. Vis. 2011, 11, 9. [CrossRef] 42. Godil, A.; Wagan, A.I. Salient local 3D features for 3D shape retrieval. In Three-Dimensional Imaging, Interaction, and Measurement; International Society for Optics and Photonics: San Francisco, CA, USA, 4–27 January 2011; Volume 7864, p. 78640S. 43. Harper, F.M.; Konstan, J.A. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. (Tiis) 2015, 5, 1–19. [CrossRef] 44. McAuley, J.; Targett, C.; Shi, Q.; Van Den Hengel, A. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, 9–13 August 2015; pp. 43–52. 45. He, R.; McAuley, J. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, Montréal, QC, Canada, 11–15 April 2016; pp. 507–517. 46. Haruna, K.; Akmar Ismail, M.; Suhendroyono, S.; Damiasih, D.; Pierewan, A.C.; Chiroma, H.; Herawan, T. Context-aware recommender system: A review of recent developmental process and future research direction. Appl. Sci. 2017, 7, 1211. [CrossRef] 47. Feng, Y.; Lv, F.; Hu, B.; Sun, F.; Kuang, K.; Liu, Y.; Liu, Q.; Ou, W. MTBRN: Multiplex Target-Behavior Relation Enhanced Network for Click-Through Rate Prediction. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, Online, 19–23 October 2020; pp. 2421–2428. 48. Zhang, D.; Liu, L.; Wei, Q.; Yang, Y.; Yang, P.; Liu, Q. Neighborhood Aggregation Collaborative Filtering Based on Knowledge Graph. Appl. Sci. 2020, 10, 3818. [CrossRef] 49. Huang, R.; McIntyre, S.; Song, M.; Ou, Z. An Attention-Based Latent Information Extraction Network (ALIEN) for High-Order Feature Interactions. Appl. Sci. 2020, 10, 5468. [CrossRef] 50. Schütze, H.; Manning, C.D.; Raghavan, P. Introduction to Information Retrieval; Cambridge University Press: Cambridge, UK, 2008; Volume 39. 51. Kohavi, R.; Longbotham, R. Online Controlled Experiments and A/B Testing. Encycl. Mach. Learn. Data Min. 2017, 7, 922–929. 52. Wilcoxon, F. Individual comparisons by ranking methods. In Breakthroughs in Statistics; Springer: New York, NY, USA, 1992; pp. 196–202.

Journal

Applied SciencesMultidisciplinary Digital Publishing Institute

Published: Feb 16, 2021

There are no references for this article.