Topic: Objective Trust Rank (new feature)

Hi everyone!

Yesterday I introduced a new approach to "trust ranking" the users/ratings. The most important thing: No more manual/subjective changes. It's not working perfectly yet, but the trust rank is being assigned by the system based on a few criteria which are the same for all users.

Here's how it works:

• The trust rank of the users is now based on how frequently (and consistently) they contribute to the website. This penalizes users who only join to give their favorite (or even their own) release a perfect score and then never return, as well as users who go on a rating frenzy on one day and then stay absent for months. So in order to get a high trust rank you should simply use the website regularly. Currently this is based on ratings assigned, but it will also include other forms of contribution/usage such as assigning other tags, putting releases on playlists, submitting polls and so forth.
• Each individual rating is weighted based on your familiarity level, so a first listen rating will have a lower weight than a rating of someone very familiar with a release. Of course this can be abused, which is why ...
• ... extremely low ratings have reduced weight even with high familiarity level. Why? Simply because it is unrealistic that someone would listen to a really awful release that often.
• Also, extremely high ratings will also be de-weighted in the future, only there the criteria are less obvious. One way would be to look into the rating distribution of a user. We could define an optimum distribution and then penalize users (slightly) who deviate too much from it. Like when you only submit S-tier ratings, or only E-tier ratings, or only ratings from the extreme ends, your ratings could weigh slightly less compared to users that have a more realistic distribution.
• The same goes for familiarity - someone who always assigns the highest level could be de-weighted, meaning that we would assume that they are just trying to maximize their weight and use a mid-range familiarity level for the calculation.
• last but not least, all the bots are now at a really low weight. Previously the system would boost the weight if there were a lot of votes behind that bot's rating.

What do you think?

Mike, your commitment to the site and updating and adapting things is pretty amazing, it must be a huge amount of time, effort and money. Can only apologise for my role in giving you another job to do, and I do wish I had it in me to offer something in return, other than the feedback I definitely owe you!

All the above makes a lot of sense to me in the logic of how it works, and in taking a manual process away from your workload (once it's fully set up of course).

Many of the ratings I put in are based on a first listen, maybe a second, which definitely gives me less right to have a say than someone who has listened extensively and potentially picked up on finer detail of an album than me. So I think its good that gets a lower weighting (and slightly offsets my now concerningly high trust rank ;-) ). The way I use the site, most of the time I will forget/not bother to register generally listening to albums again, but that's 100% a me problem!

The optimum distribution idea I guess is probably a tricky one as I'm not sure it's easy to define. I take in broadly progressive music at a fairly random level and so as discussed end up listening to things a long way out of my comfort zone and so will sometimes use the lower tiers. Another person may be listening to music in a more comfortable niche for them but have a really good extensive knowledge of it, and it wouldn't be inconceivable that everything is rated at least mostly in the top half of tiers. I don't really have an answer here or a better solution, but maybe its just as you hint at above where its only the real extremes in a high volume that are de-weighted (as at low volume it's sort of already covered by the frequent/consistent part in the first bullet point)

Bots being low weight - broadly agree as at the end of the day its the opinion of the users of this site that should matter most. But there's probably a balance to be found. There are plenty of albums on here where I'm the only user that's provided a rating, along with a couple of bots. In particular with PABot, if that is based on 100 people's rating then I'm not sure my one opinion should completely negate all theirs. But equally I wouldn't want to be regularly overruled by the machines...

I really enjoy the effect of de-weighting first-listen ratings, it makes me more confident in assigning a really subjective rating on first listens. IMO users should just go with their gut feeling when they first listen to a release and then only start considering objective criteria as they get more familiar with the music. So as your rating keeps gains weight on consecutive listens, you have the opportunity to revise/refine it.

Mike, I grossly disagree with these changes. I was confused why my recent ratings didn't affect the overall aggregated rating. I don't think deweighting first listens entirely is logical at all, and I find it unfair. I also don't always update my play counts if I agree with the rating I already have, but I don't also don't frequently re-listen to an album more than once, and it feels completely pointless to rate on a first listen then. Not only do I not have the patience to wait for a second listen to have the rating count, but it's also forcing me to micro-manage my already existing ratings, which is the opposite of accessible and easy to use. A first listen has no business being disregarded.

We're not professional reviewers, we're just ordinary people who make no money from this, and any time on the site is essentially wasted if not spent well, and in my case, I'm using this site to publicly document my listen log and collection, which I have recorded on an Excel spreadsheet, anyway (thanks to the jerk admins who made RYM an unreliable medium to do so). High and low ratings exist for a reason. Deweighting those is also just... baffling, and I don't agree with your logic on doing this either. Rating aggregates are going to be skewed towards a very center-oriented bell curve, with very little standard deviation, eg rating diversity between albums will be greatly diminished. I understand what you're trying to do, but the damaging affects here are much worse than your goal of trying to prevent abusers, which don't really exist on this site currently, because there's only a handful of users here and we're pretty much all rating in good faith. The fact that it's so small is even more the reason NOT to limit the efficacy of our expression. Any deweighted rating in my opinion is wasted time I could spend on something else that is more accomplishing... This isn't a professional review site. Like RYM and Metacritic (with big emphasis on the "Y" and "meta," respectively), this is meant to aggregate the opinions of an ordinary listener, or at least those with prog tendencies, without the pomposity of a a well-known reviewer.

There is no such thing as an "objective rating," as you're putting it. Even amongst professional reviews. Every opinion is purely subjective, between listening tastes, production preferences, musical techniques, and anything else that goes into a review. For instance, I hate diminished dynamic range and I will always deduct points for a louder master, but evidentially you seem to like the louder engineering sound, per your complaint on my use of the term "brickwall." In neither preference is there an "objective" mindset. This applies to everything that can be evaluated... whether you like chugging riffs or djent-oriented syncopation as opposed to more technical and complex melodic structures, it all comes down to preference. The masses can disagree with a "professional" reviewer, and vice versa, and sometimes I will even take a contrarian stance if I find an album overrated or underrated. Nobody in any scenario is "objectively" correct, and claiming otherwise is a foolish trap.

Now there are multiple reasons why I would listen to an "extremely" low rated album multiple times. There could be the case that you second guess your rating, or you need to listen to it again to be sure it really deserves a low rating, even if you don't like it, such is the case with Opeth's Blackwater Park for me. Each time I listened to it, I dropped the rating further, though I was confused why I disagree so much with the masses on this one, hence my four or five listens on it, despite a one star rating; I was hoping it would eventually click, but it never did. There's also the case of evolved music taste, such that a listener may have previously liked an album and listened to it frequently, but over the years they started to dislike that style because their preference changed. This is my case for Stone Sour's Come What(ever) May - I loved it for some time in my early years of high school and listened to it at least fifteen times, but by the time I was in my early 20s, I had developed a dislike to that style of edgy, condescending, and cynical heavy modern alt rock that defines the band, and nowadays I don't particularly find Corely Taylor to be a good vocalist. Hence that's how I ended up listening to that one star album more than ten times; I used to like it but from a retrospective viewpoint, I think it's a lousy album. Maybe I'm a little harsh on either album, but that doesn't make me objectively wrong nor correct, nor should my opinion be disregarded.

I welcome the further deweighting of the bots, but honestly, I don't like trust system otherwise, and I explained why in the past.

Frankly, I may just stop using the site if one listen ratings are ignored. May continue for documenting my collection and polls, but I would find the rating aspect to not be a wise use of time tbh... If you insist on giving an album with more listens higher weight, that can be achieved by increasing the weight of ratings with more listens rather than deweighing a lower count. But my opposition to that lies in precisely your logic for deweighing lower ratings and that is: "why would someone listen to album they don't like multiple times?" - or in this scenario, a user who likes an album will likely listen to it more than someone who doesn't like it, meaning that ratings with more listens, and thus likely higher ratings, will almost always have more weight than those with fewer listens, and therefore lower ratings. It worsens the "lower standard deviation, more central bell curve" effect I discussed above, or in cases of universally disliked albums, like Metallica's St. Anger, it actually theoretically increases the mean score to be skewed to the right with a longer leftward trail in which the deweighted mean and median would be located. In other words, the rating wouldn't feel "organic," and feels like it's artificially aggregated, with preference to positivity.

First listen ratings are not ”ignored“. I do think that they should weigh less than ratings on repeated listens. You are of course free to disagree.

I am puzzled that you find these changes ”unfair”. They are applied to all users in the same way.

For what its worth, at least one of my first listen ratings has had a clear impact on the overall rating of an album since the changes so can vouch for them definitely not being ignored.

Know it might not be as simple as this but Mike, what's the % de-weight for a first listen compared to, say 2-3 listens?

^ I've just adjusted the code and the scores are being recalculated. Previously (when you wrote your post) the trust rank for low first listen ratings went from 8 to 1 (I never reduce it to zero). Now I've increased it considerably so that it should only go down to 2-3 from 8.

I’m curious as to how many regular raters contribute as opposed to “once in a while” raters such as myself. I notice often it’s the same 5 or 6 raters barreling through ratings daily. I think maybe the top 5 raters should have a vote on changes, being that they are so active and keep the site moving. I say this in good faith. My only criticism (being a not often contributor) is that the site is perhaps a bit cumbersome and heavy, like RYM. (Although probably necessary given the huge amount of releases). A pairing down of details could be helpful

IMO First listens are often the only listens, so they should fully count until someone decides to listen again and perhaps change the rating if so desired.

I also don’t think album covers should weigh in on album ratings. And I question “composition” ratings as well. Individual Track ratings seem to add a lot of unnecessary weight also. And having 100 numbers to choose from when giving a A for instance (I-100) feels like minutiae. Just some random thoughts to consider. It’s your guys website. It’s obviously your baby to do as you wish. All the best!

^ Thanks for the feedback!

A pairing down of details could be helpful

It's not an easy task - I guess the key to improvements here would be to find ways to "hide" much of the information so that everything looks simpler, but the info can still be accessed for those looking for them. Which bits of information would you see as secondary, e.g. something to take out of the general views?

I think maybe the top 5 raters should have a vote on changes

I'm not prepared to make this a fully democratic process, but I do try to listen to feedback from all of you who use the website the most.

IMO First listens are often the only listens, so they should fully count until someone decides to listen again and perhaps change the rating if so desired.

If you only listen to a release once, submit a first listen rating and then never return to it, I still think that should weigh less than a rating you submit on more than one listen.

I also don’t think album covers should weigh in on album ratings.

They do not! None of the individual rating categories (e.g. album cover, composition, musicianship and so on) have any impact on the overall rating you submit.

Individual Track ratings seem to add a lot of unnecessary weight also

No, once you submit a rating for the release the individual track ratings have no impact on it. It's only when you haven't submitted a release rating yet that the system computes a release rating from the track ratings. And that calculated release rating has a lower weight.

And having 100 numbers to choose from when giving a A for instance (I-100) feels like minutiae

Well, the A tier has only 10 numbers to choose from (80-89), but yes, many users would probably opt for a simpler scale. Would you prefer just choosing a tier (F-S)? I'm thinking about implementing that as a starting point for new users. I can think of three "levels" a user could choose from:

1. Just select a tier (F-S)
2. Select a graded tier (e.g. A-, A or A+)
3. Select a numeric value (how it currently works)

We already have the plain tier selection for the sub rating categories (e.g. cover art).

Thanks for the explanation about what affects the overall ratings. I misunderstood how things like cover art and track ratings affect overall ratings. The website is comprehensive and very well done, and the huge amount of releases and ratings are likely what makes it ‘feel’ heavy. (A lot of ground to cover month by month.) The tier option you suggest sounds good.

Personally, I probably listen to only 1-3 new releases (mostly on Bandcamp) a week, and will be sure to contribute more when I can.

I try to steer clear of rating in genres I don’t like. I think it would be unfair for me to rate an album in the death metal genre, knowing how much I dislike the entire genre. I think if one dislikes a genre, ( RIO ) for example, it’s probably better to not rate at all rather than give it a stinker. I know that’s a controversial statement. Just thinking out loud.

I try to steer clear of rating in genres I don’t like

My approach as well, for the most part. I still think it's important to allow these "stinkers" since identifying them as such would require a subjective decision :-)

the huge amount of releases and ratings are likely what makes it ‘feel’ heavy

Yes, there's a lot of music being releases every month. I am trying to reduce the amount, I overdid it last year when I made sure to add trending releases on RYM every week. I'm more "chilled" these days :-)