BeerAdvocate.com, Leveraging User Tendencies to Improve User Experience

AnalyTeam, 2022

Introduction

In this project, we soft-cluster1 the users of BeerAdvocate within a set of predefined categories. Based on that, we analyze how attractive the website is for each category over time. Additionally, we uncover the trends of these categories and build interesting user personas for the website’s administrators. By the end, we come up with a few conclusions about what is done right in BeerAdvocate and what could be improved in order to make the website more attractive to the beer lovers community. We also provide a more general result in the form of the conclusion that natural soft-clustering of users can help uncover user groups’ behaviors which provides room for improvement in many businesses, including those that heavily rely on recommender systems and providing custom user experiences. We have available to us a dataset of BeerAdvocate data spanning between 1996 and 2017.

1natural soft-clustering: this approach is “cluster together users that satisfy a human-interpretable condition” rather than “cluster together users that are similar based on a similarity metric”. Since one user can satisfy many conditions simultaneously or none at all, any user can belong to any given number of categories including none at all. That is what we mean by natural soft-clustering of users.

Dataset

We use the dataset provided by the teaching team of “CS-401: Applied Data Analysis” at EPFL for the year 2022.

The Approach to Soft-Clustering the Users

We use a score-based approach to soft-cluster users using scores that are easily interpretable to humans. All of our scores satisfy that the larger they are the more they emphasize that the user may belong to the given category.

Conformists Category (CFM)

A user who rates beers close to their average rating, meaning that on average this user deviates little from the average opinion on the beers he/she rates.

This category may be regarded as an indicator of the herding effect where users tend, on average, to stick to the average opinion when rating beers even if the average opinion may be unfair towards a beer.

Show more A user $u$ is a conformist if he/she has a high conformism score that measures how much the user sticks with the average opinion on the beers he/she rates. The score is defined as follows: $$CFM_u = -\frac{1}{B_u} \sum_{b \in B_u} (\frac{r_{u,b}-\overline{r_b}}{\sigma_b})^2$$ $B_u$ is the set of beers the user $u$ has rated. $r_{u,b}$ is the rating given by user $u$ to beer $b$. $\sigma_b$ is the standard deviation of the ratings of the beer $b$. $\overline{r_b}$ is the average rating of beer $b$.

EXP Users Category (EXP)

A user that rates beers close to the BA score that is displayed on the website for most beers. According to BeerAdvocate administrators, the BA score is a reference score displayed on the website to give the users an idea about how a beer ranks among the beers of the same style.

Note that 94.5% of rated beers in the data available to us have a BA score.

For lack of a better name, we will reference these users as the EXP users referring to a somewhat “structured” way of rating beers that may indicate being influenced by the BA score displayed on the website.

Show more A user $u$ is an EXP user if he/she rates, on average, beers close to their BA score scaled down to the range [0, 5] using the following piecewise linear mapping based on the BA score range and meaning revealed by the website's administrators, to which we believe this mapping corresponds best: {world class : 95 - 100 mapped to ]4.75; 5], outstanding : 90 - 94 mapped to ]4.5; 4.75], very good : 85 - 89 mapped to ]4; 4.5], good : 80 - 84 mapped to ]3.5; 4], okay : 70 - 79 mapped to ]3; 3.5], poor : 60 - 69 mapped to ]2; 3], awful : <= 59 mapped to ]0; 2]}. The score is defined as follows: $$EXP_u = -\frac{1}{|B_u|} \sum_{b \in B_u} (\frac{r_{u,b}-BA_b}{\sigma_b})^2$$ $B_u$ is the set of beers rated by user $u$. $r_{u,b}$ is the rating given by user $u$ to beer $b$. $\sigma_b$ is the standard deviation of the ratings of the beer $b$. $BA_b$ is the BA score a beer $b$.

Explorers Category (XPL)

A user that rates beers that have a few ratings only if not none at all. In the latter case, we consider that user to be the one who added the beer to the website. (yes, BeerAdvocate allows any user to add new beers to its database.)

These users, therefore, shed light (positively or negatively) on unpopular beers by rating them, and some of them help populate the website with new beers which enriches the user experience on the website.

Show more This score provides us with information about which users contribute to enriching the experience on the website, either because they rate beers that do not get much attention, or because they "introduce" new beers on the website by being the first people to rate those beers. A user $u$ is an explorer if, often enough, among the beers he rated, he/she was among the first 10 users to rate that beer. The score is, therefore: $$XPL_u = \sum_{b \in B_u} \mathbb{1} [u \in U_{10}(b)]$$ $B_u$ is the set of beers rated by user $u$. $U_{10}(b)$ is the set of at most 10 users that first rated the beer $b$.

A user that rates often enough beers that are slightly okay at best.

These users are willing to risk trying out bad beers. We are interested in such users because they may contribute to the visibility of underrated beers on the website and so indirectly to the user experience.

Show more A user $u$ is an adventurer if he/she often enough risks trying out beers that have a bad rating at the time at which they rate them (less than 3.25/5, knowing that the average is at $\approx$ 3.97/5). The corresponding score is the following: $$ADV_u = \sum_{b \in B_u} \mathbb{1} [\overline{r}_{u,b}(t_{u,b}) < T]$$ $B_u$ is the set of beers rated by user $u$. Rating $\overline{r}_b(t_{u,b})$ is the average rating of beer $b$ at the time $t_{u,b}$ at which user $u$ rates beer $b$. $T=3.25$ is a cut-off determined empirically based on the data.

User Clustering

In order to choose adequate thresholds to classify the users based on the scores defined in the previous section, we look at the distributions of these scores. The figures below plot those distributions in real and logarithmic scales:

The adventurer score has a heavy-tail distribution. By the interpretability of this score, a cut-off or threshold at 0.2 classifies as adventurers the users for which at least 20% of the beers they rate have a slightly okay at best rating at the moment at which they rate them. We use this threshold that corresponds to the $90^{th}$ percentile of the ADV score distribution.

The conformist score also has a heavy-tail distribution. Here again, we choose the threshold that corresponds to the $90^{th}$ percentile that is -0.35, which by the interpretability of this score means that we classify as conformists the users who deviate on average by less than 0.35 from the average opinion.

The EXP score looks more like a skewed gaussian, but it is not since it is heavy-tailed. Again by the interpretability of this score, a threshold at -0.2 classifies users that deviate on average by less than 0.2 from the displayed BA score as EXP users.

The explorer score also follows a heavy-tailed distribution. To reiterate, by the interpretability of this score, given a threshold of approximately 0.2, we classify as explorers the users that figure in the first 10 raters of at least 20% of the beers they rate.

To What Extent Do the Categories Overlap?

Since we do soft clustering, users have a score for each category. But how users who meet the threshold score be in multiple categories? We used a simple Venn diagram to visualize the overlap of categories. The Venn diagram shows the number of users for the categories CFM, ADV, and XPL, and the number of users who are at the intersection of multiple categories.

As we can see, there are very few overlaps between explorers, conformists, and adventurers. There are some users with two categories, but almost no users with all 3 categories. This suggests that the categories that we defined target different kinds of users.

The Story Overview

In the following sections, we will study the categories defined previously on many levels: beer style preferences, locations, ratings, reviews…etc to extract as much information about these categories as possible to build relevant personas that can be leveraged by the administrators to improve the UX of their website.

How Does the Number of Ratings and Reviews Characterize the Categories?

In this section, we highlight the ratings and reviews tendencies of the four categories: XPL, ADV, CFM, and EXP.

In order to know if the number of ratings characterizes in some way each category, we plot the likelihood of a user belonging to a category as a function of the range in which his/her number of ratings falls. Each range corresponds to an inter-quantile range of around 25% of the density of the number of ratings of all the studied users. As a quick reminder, those are the users from English-speaking countries having at least 5 ratings. The figure below shows the result.

It is worth pointing out that we only consider 99% of the density as the remaining 1% contains only super users which have many folds more ratings than normal users. Since there are very few super users, their results may be non-conclusive because they may be non-representative of super users overall that is why we treat them as outliers and discard them here.