Talk:AdamH Card Ratings

From DominionStrategy Wiki
(Difference between revisions)
Jump to: navigation, search
Line 68: Line 68:
  
 
Take it down then.
 
Take it down then.
 +
 +
-Adam
 +
 +
---
 +
 +
Wero, I asked you to delete the page instead of continuing to revert it back to a sortable table. Just delete the stupid thing already.
  
 
-Adam
 
-Adam

Revision as of 13:56, 6 February 2019

What is the purpose for making the list not sortable? It's a feature that was requested by multiple others immediately upon the creation of this article, it's not a performance issue, and the default sort is identical to the current arrangement. As far as I can see it's a strict improvement. -panama

---

AdamH: I'll try to explain why I don't think the data should be sortable on this page: let's say there was a button that allowed the user to replace each rating with a random number, that would misrepresent the data -- I feel that sorting cards by rating misrepresents the data in a similar way (though not quite as egregiously).

The data gathered doesn't give a ranking of cards, it gives a rating for each individual card. Comparing cards directly is not something that the data shows, so statements like "King's Court is better than Donate" don't follow from this data even though that's the first thing you see if you sort by mean. The way I gathered data was crafted for a purpose, knowing the limitations of the data, and presenting the data in a way consistent with what the data actually shows is important for its integrity -- putting a sort button in only decreases the integrity of the data.

If people are saying they want a sort button, well, frankly I think they don't understand how to read the data. As someone who is presenting the data, I feel it's my obligation to take whatever measures I can to prevent people from misreading it. Of course people can take the data and sort it using their own tools (and people have done this) and I can't stop them, but I don't want to encourage that. I'd rather make it clear what the data is for and instill an understanding in them why the data doesn't show that.

If "the community" decides that they absolutely want a sort button, of course I'm going to lose an edit war. In that case, I would like for my name to be completely removed from this page because I don't want it shown that I endorse presenting the data in this way.

Hopefully this helps explain my position.

---

Thanks for your response, AdamH. Could you elaborate on which aspects of your data gathering process make sorting inappropriate? It appears to me that since the same things were asked about each card, it naturally invites comparison. The problems with the KC/Donate example are not that it is a comparison fundamentally, but rather that their scores are so close (likely well within the margin of error) and that it uncritically assumes that higher rated <--> "better". Furthermore, comparison is extremely helpful to contextualize these numbers (what does a rating of 4.53 mean in a vacuum?). Also, there are reasons beyond comparison to want to sort the data--after editing the table to sortable, I immediately used it to check if the data had been rescaled to have the means span from 0-10 or not.

Perhaps a good solution could be to leave the table sortable, but with a disclaimer where you explain the pitfalls of sorting for comparison? Since (by my current understanding) it has valid uses that can improve the utility of the data to users, I am strongly inclined to leave sorting available.

On removal and edit wars, I'm not highly familiar with the norms of this wiki, but I think that entire thing is a more confrontational take than I'd prefer. I believe standard wiki dispute resolution protocol is something like [thoughtful discussion and attempting to reach a compromise] -> [getting outside opinions from other editors] -> [moderated dispute resolution], moving up the chain only if a satisfying solution isn't reached at each prior step, so hopefully no edit war is necessary, haha.

-panama

---

AdamH: Rating cards has a lot of fundamental issues, meaning that there are big problems with trying to make any kind of conclusions from the data. These issues include the fact that there is no commonly accepted (or even internally consistent) way to evaluate the power level of cards, much less condense that to a scale of 0-10. When people input their numbers, they do so with whatever ideas they have in their mind for what makes a good card, which can vary wildly between people, which results in a ton of noise in the data (as you noticed). It's not uncommon for two people to rate a card up to 4 points apart but not have any real disagreement about how "good" the card is, just how to apply it to a number.

Given these limitations, the data can really only tell you a few things, and those things are very vague. I don't see any conclusion you can draw from the data that sorting it helps you make.

> Could you elaborate on which aspects of your data gathering process make sorting inappropriate? It appears to me that since the same things were asked about each card, it naturally invites comparison. The problems with the KC/Donate example are not that it is a comparison fundamentally, but rather that their scores are so close (likely well within the margin of error) and that it uncritically assumes that higher rated <--> "better". Furthermore, comparison is extremely helpful to contextualize these numbers (what does a rating of 4.53 mean in a vacuum?).

The relevant comparison(s) to make here are between a card and the 0-10 scale. When cards get compared to each other is when it starts to deviate from what the data is trying to show. Sorting the data invites the reader to make only the "smaller" comparisons (comparisons between cards of similar rating) which are the least valid ones to make.

I see the only real use case for this data as being something like this: a person has a decent idea of all the cards and how powerful they think those cards are. They can go through this list and spot any scores that are large disagreements from their own held opinions. From there, they can discuss with other people who can have a conversation about what the card is good for, etc, in the context of games of Dominion (which is probably the more valuable thing).

Sorting the data leads to conversations about which is better between Donate and King's Court, which IMO are completely pointless because it has no bearing on how to play real games of Dominion better -- they're both very good and that's all there is to say about comparing the two cards.


> Also, there are reasons beyond comparison to want to sort the data--after editing the table to sortable, I immediately used it to check if the data had been rescaled to have the means span from 0-10 or not.

This is a good reason for someone to sort the data on their own, but not a reason for the data to be presented in a way that encourages readers to sort it.

Anyone who wants to have these other discussions or compute these stats is free to do so (I'll even link a Google Sheet below which gives access to some more power tools and data categories, etc.) but the main point of all these ratings is to give a general idea of card power level from 0-10 and I don't think other things which can misrepresent what the data is trying to show belong on the page that is just presenting the data.

https://docs.google.com/spreadsheets/d/1R9eApE1O1977H6h8RvvoVl_PIMltWxlQDzdYPF3PQDE


---

markus: If you don't want people to compare KC and Donate, you should just cut the mean column. Median is also more robust to some outliers in your sample. If I want to know which cards are more relevant, I want to see which cards have a higher score. Sorting them alphabetically is not useful. Sort by median and you get exactly that. In addition, add some columns on whether a card is a village etc., and one can sort to see which are the more relevant villages.

---

AdamH: I can see the argument for removing the mean column -- it helps with a lot of the issues with the data. On the other hand, it doesn't eliminate all of the problems with sorting the data.

As for the rest of your suggestions, these would be very useful if there was a filter wiki tables -- without that I don't think it's useful to include here (especially if the table isn't going to be sortable).

If I edit this table again I'm going to modify the script I used to make the wiki table to at least include the expansion as a column. I'm toying with the idea of just putting the card name and the rating(s) in the same box, making it impossible to sort by card rating but maybe using the sortable table to function as a less-good way of filtering by the other fields that are available. This might be an acceptable compromise.


  • Adam, multiple people have told you that they want this table to be sortable by ranking. If that means you want your name taken off of it, then it's just some random ratings, and that's kind of pointless. This table is either going to have your name on it and be sortable by ranking, or it's not going to be on the wiki at all.

P.S. you can sign your post using four tildes in a row, like so: Werothegreat (talk) 11:48, 1 February 2019 (EST)

---

Take it down then.

-Adam

---

Wero, I asked you to delete the page instead of continuing to revert it back to a sortable table. Just delete the stupid thing already.

-Adam

Personal tools
Namespaces

Variants
Views
Actions
Navigation
Toolbox