Talk:AdamH Card Ratings

From DominionStrategy Wiki
(Difference between revisions)
Jump to: navigation, search
 
(9 intermediate revisions by 4 users not shown)
Line 12: Line 12:
  
 
Hopefully this helps explain my position.
 
Hopefully this helps explain my position.
 +
 +
---
 +
 +
Thanks for your response, AdamH. Could you elaborate on which aspects of your data gathering process make sorting inappropriate? It appears to me that since the same things were asked about each card, it naturally invites comparison. The problems with the KC/Donate example are not that it is a comparison fundamentally, but rather that their scores are so close (likely well within the margin of error) and that it uncritically assumes that higher rated <--> "better". Furthermore, comparison is extremely helpful to contextualize these numbers (what does a rating of 4.53 mean in a vacuum?). Also, there are reasons beyond comparison to want to sort the data--after editing the table to sortable, I immediately used it to check if the data had been rescaled to have the means span from 0-10 or not.
 +
 +
Perhaps a good solution could be to leave the table sortable, but with a disclaimer where you explain the pitfalls of sorting for comparison? Since (by my current understanding) it has valid uses that can improve the utility of the data to users, I am strongly inclined to leave sorting available.
 +
 +
On removal and edit wars, I'm not highly familiar with the norms of this wiki, but I think that entire thing is a more confrontational take than I'd prefer. I believe standard wiki dispute resolution protocol is something like [thoughtful discussion and attempting to reach a compromise] -> [getting outside opinions from other editors] -> [moderated dispute resolution], moving up the chain only if a satisfying solution isn't reached at each prior step, so hopefully no edit war is necessary, haha.
 +
 +
-panama
 +
 +
---
 +
 +
AdamH: Rating cards has a lot of fundamental issues, meaning that there are big problems with trying to make any kind of conclusions from the data. These issues include the fact that there is no commonly accepted (or even internally consistent) way to evaluate the power level of cards, much less condense that to a scale of 0-10. When people input their numbers, they do so with whatever ideas they have in their mind for what makes a good card, which can vary wildly between people, which results in a ton of noise in the data (as you noticed). It's not uncommon for two people to rate a card up to 4 points apart but not have any real disagreement about how "good" the card is, just how to apply it to a number.
 +
 +
Given these limitations, the data can really only tell you a few things, and those things are very vague. I don't see any conclusion you can draw from the data that sorting it helps you make.
 +
 +
> Could you elaborate on which aspects of your data gathering process make sorting inappropriate? It appears to me that since the same things were asked about each card, it naturally invites comparison. The problems with the KC/Donate example are not that it is a comparison fundamentally, but rather that their scores are so close (likely well within the margin of error) and that it uncritically assumes that higher rated <--> "better". Furthermore, comparison is extremely helpful to contextualize these numbers (what does a rating of 4.53 mean in a vacuum?).
 +
 +
The relevant comparison(s) to make here are between a card and the 0-10 scale. When cards get compared to each other is when it starts to deviate from what the data is trying to show. Sorting the data invites the reader to make only the "smaller" comparisons (comparisons between cards of similar rating) which are the least valid ones to make.
 +
 +
I see the only real use case for this data as being something like this: a person has a decent idea of all the cards and how powerful they think those cards are. They can go through this list and spot any scores that are large disagreements from their own held opinions. From there, they can discuss with other people who can have a conversation about what the card is good for, etc, in the context of games of Dominion (which is probably the more valuable thing).
 +
 +
Sorting the data leads to conversations about which is better between Donate and King's Court, which IMO are completely pointless because it has no bearing on how to play real games of Dominion better -- they're both very good and that's all there is to say about comparing the two cards.
 +
 +
 +
> Also, there are reasons beyond comparison to want to sort the data--after editing the table to sortable, I immediately used it to check if the data had been rescaled to have the means span from 0-10 or not.
 +
 +
This is a good reason for someone to sort the data on their own, but not a reason for the data to be presented in a way that encourages readers to sort it.
 +
 +
Anyone who wants to have these other discussions or compute these stats is free to do so (I'll even link a Google Sheet below which gives access to some more power tools and data categories, etc.) but the main point of all these ratings is to give a general idea of card power level from 0-10 and I don't think other things which can misrepresent what the data is trying to show belong on the page that is just presenting the data.
 +
 +
https://docs.google.com/spreadsheets/d/1R9eApE1O1977H6h8RvvoVl_PIMltWxlQDzdYPF3PQDE
 +
 +
 +
---
 +
 +
markus: If you don't want people to compare KC and Donate, you should just cut the mean column. Median is also more robust to some outliers in your sample. If I want to know which cards are more relevant, I want to see which cards have a higher score. Sorting them alphabetically is not useful. Sort by median and you get exactly that.
 +
In addition, add some columns on whether a card is a village etc., and one can sort to see which are the more relevant villages.
 +
 +
---
 +
 +
AdamH: I can see the argument for removing the mean column -- it helps with a lot of the issues with the data. On the other hand, it doesn't eliminate all of the problems with sorting the data.
 +
 +
As for the rest of your suggestions, these would be very useful if there was a filter wiki tables -- without that I don't think it's useful to include here (especially if the table isn't going to be sortable).
 +
 +
If I edit this table again I'm going to modify the script I used to make the wiki table to at least include the expansion as a column. I'm toying with the idea of just putting the card name and the rating(s) in the same box, making it impossible to sort by card rating but maybe using the sortable table to function as a less-good way of filtering by the other fields that are available. This might be an acceptable compromise.
 +
 +
 +
* Adam, multiple people have told you that they want this table to be sortable by ranking.  If that means you want your name taken off of it, then it's just some random ratings, and that's kind of pointless.  This table is either going to have your name on it and be sortable by ranking, or it's not going to be on the wiki at all.
 +
 +
P.S. you can sign your post using four tildes in a row, like so: [[User:Werothegreat|Werothegreat]] ([[User talk:Werothegreat|talk]]) 11:48, 1 February 2019 (EST)
 +
 +
---
 +
 +
Take it down then.
 +
 +
-Adam
 +
 +
---
 +
 +
Wero, I asked you to delete the page instead of continuing to revert it back to a sortable table. Just delete the stupid thing already.
 +
 +
-Adam
 +
 +
---
 +
 +
Adam, I find your take on the sorting unpersuasive. The problems you describe like high variance are problems inherent to rating itself, not to comparison, and saying you don't see the uses for sorting given them doesn't mean those uses don't exist.
 +
 +
> The relevant comparison(s) to make here are between a card and the 0-10 scale. When cards get compared to each other is when it starts to deviate from what the data is trying to show. Sorting the data invites the reader to make only the "smaller" comparisons (comparisons between cards of similar rating) which are the least valid ones to make.
 +
 +
This is not the only way to use a sort to make comparisons. For instance, a very common practice in games is to use a sorted list as the basis for a tier list, in which you look for natural divisions in the data to carve out tiers, inviting comparison between cards that are far away on the scale (you will clearly see that KC is "better" than Transmute) while discouraging comparisons between nearby ones.
 +
 +
> This is a good reason for someone to sort the data on their own, but not a reason for the data to be presented in a way that encourages readers to sort it.
 +
 +
This is an unjustified assertion and completely backwards. This takes about a second to perform if the list is sortable on the wiki while it may take minutes to perform if someone has to download and sort it themselves. It's exactly the kind of task that a sortable table on the wiki makes most worthwhile.
 +
 +
On a broader note, I think I disagree philosophically with the mindset you're approaching this from. Wikis are public resources, and I think trying to gatekeep what people can and can't do with the info that you put on them is presumptuous and condescending, especially since it's clear you've been overlooking common use cases that involve sorting, such as checking for data rescaling and forming a tier list. Imo we should assume that people are in general better judges of what best serves their own purposes than we are, and so we should try to accommodate the broadest range of common use cases that we can.
 +
 +
My hope would be that this data can remain on the website for common use, in a sortable form. Taking it down over a relatively minor disagreement about its presentation seems extreme to me. I also think this discussion of removing your name from the page kind of misses the point of contributing to a wiki, where you're releasing the data to the community of people who will also contribute, and further changes don't have to represent an endorsement by you of those changes. After all, it does say right below this very edit window "If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here."
 +
 +
[[User:Panama|Panama]] ([[User talk:Panama|talk]]) 14:42, 7 February 2019 (EST)Panama
 +
 +
* With Panama's response, I have reconsidered.  I'm going to revert the table to being sortable, and keep the page on here, as clearly there is some desire to have this information available.  What I will do, however, is amend my ultimatum from deleting the page if AdamH reverts it: I will instead block AdamH from editing for 1 month if he reverts it again, because hey, I can do that, and technically he is removing information from the page, which is one of the drop-down reasons I can use for blocking a user.
 +
 +
You're not going to win this, AdamH.  Please don't make me take action that will make you hate me even more. [[User:Werothegreat|Werothegreat]] ([[User talk:Werothegreat|talk]]) 15:53, 7 February 2019 (EST)
 +
 +
---
 +
 +
I've talked to several other people about this, including Qvist. These people were actually willing to have conversations with me about what's going on here and what features they actually want from these ratings, as opposed to making threats and turning this into a competition.
 +
 +
Based on those conversations, people were asking for many features related to this data that a wiki table won't support. Maintaining the page is also an issue, since the poll is constantly open and is updated several times a year, I don't want to be responsible for keeping this wiki table up-to-date, especially given how hostile you've been, Wero.
 +
 +
So what I've done is this: I made a Google Doc that's suitable for sharing with anyone interested in the data. It contains the raw data as well as statistics on that data, which satisfy all of the features that people who were willing to have a conversation with me have actually asked for. And yes, the data is sortable by all fields, including rating.
 +
 +
The most important thing here is that people aren't going to sort the data by rating when what they're really trying to accomplish is some kind of filtering or something else (which the Google Doc supports). The next most important thing is that nobody has to maintain this wiki table whenever the data gets updated (which it already has).
 +
 +
So the table that was on the page before my edit is now obsolete and I'm no longer willing to maintain it. I'll continue to maintain the Google Doc, so I'm going to delete the table and replace it with a link to the Google Doc. If you want to block me from editing wiki pages based off that, go right ahead man. I'm ready to wash my hands of this situation
 +
 +
-AdamH

Latest revision as of 14:58, 20 February 2019

What is the purpose for making the list not sortable? It's a feature that was requested by multiple others immediately upon the creation of this article, it's not a performance issue, and the default sort is identical to the current arrangement. As far as I can see it's a strict improvement. -panama

---

AdamH: I'll try to explain why I don't think the data should be sortable on this page: let's say there was a button that allowed the user to replace each rating with a random number, that would misrepresent the data -- I feel that sorting cards by rating misrepresents the data in a similar way (though not quite as egregiously).

The data gathered doesn't give a ranking of cards, it gives a rating for each individual card. Comparing cards directly is not something that the data shows, so statements like "King's Court is better than Donate" don't follow from this data even though that's the first thing you see if you sort by mean. The way I gathered data was crafted for a purpose, knowing the limitations of the data, and presenting the data in a way consistent with what the data actually shows is important for its integrity -- putting a sort button in only decreases the integrity of the data.

If people are saying they want a sort button, well, frankly I think they don't understand how to read the data. As someone who is presenting the data, I feel it's my obligation to take whatever measures I can to prevent people from misreading it. Of course people can take the data and sort it using their own tools (and people have done this) and I can't stop them, but I don't want to encourage that. I'd rather make it clear what the data is for and instill an understanding in them why the data doesn't show that.

If "the community" decides that they absolutely want a sort button, of course I'm going to lose an edit war. In that case, I would like for my name to be completely removed from this page because I don't want it shown that I endorse presenting the data in this way.

Hopefully this helps explain my position.

---

Thanks for your response, AdamH. Could you elaborate on which aspects of your data gathering process make sorting inappropriate? It appears to me that since the same things were asked about each card, it naturally invites comparison. The problems with the KC/Donate example are not that it is a comparison fundamentally, but rather that their scores are so close (likely well within the margin of error) and that it uncritically assumes that higher rated <--> "better". Furthermore, comparison is extremely helpful to contextualize these numbers (what does a rating of 4.53 mean in a vacuum?). Also, there are reasons beyond comparison to want to sort the data--after editing the table to sortable, I immediately used it to check if the data had been rescaled to have the means span from 0-10 or not.

Perhaps a good solution could be to leave the table sortable, but with a disclaimer where you explain the pitfalls of sorting for comparison? Since (by my current understanding) it has valid uses that can improve the utility of the data to users, I am strongly inclined to leave sorting available.

On removal and edit wars, I'm not highly familiar with the norms of this wiki, but I think that entire thing is a more confrontational take than I'd prefer. I believe standard wiki dispute resolution protocol is something like [thoughtful discussion and attempting to reach a compromise] -> [getting outside opinions from other editors] -> [moderated dispute resolution], moving up the chain only if a satisfying solution isn't reached at each prior step, so hopefully no edit war is necessary, haha.

-panama

---

AdamH: Rating cards has a lot of fundamental issues, meaning that there are big problems with trying to make any kind of conclusions from the data. These issues include the fact that there is no commonly accepted (or even internally consistent) way to evaluate the power level of cards, much less condense that to a scale of 0-10. When people input their numbers, they do so with whatever ideas they have in their mind for what makes a good card, which can vary wildly between people, which results in a ton of noise in the data (as you noticed). It's not uncommon for two people to rate a card up to 4 points apart but not have any real disagreement about how "good" the card is, just how to apply it to a number.

Given these limitations, the data can really only tell you a few things, and those things are very vague. I don't see any conclusion you can draw from the data that sorting it helps you make.

> Could you elaborate on which aspects of your data gathering process make sorting inappropriate? It appears to me that since the same things were asked about each card, it naturally invites comparison. The problems with the KC/Donate example are not that it is a comparison fundamentally, but rather that their scores are so close (likely well within the margin of error) and that it uncritically assumes that higher rated <--> "better". Furthermore, comparison is extremely helpful to contextualize these numbers (what does a rating of 4.53 mean in a vacuum?).

The relevant comparison(s) to make here are between a card and the 0-10 scale. When cards get compared to each other is when it starts to deviate from what the data is trying to show. Sorting the data invites the reader to make only the "smaller" comparisons (comparisons between cards of similar rating) which are the least valid ones to make.

I see the only real use case for this data as being something like this: a person has a decent idea of all the cards and how powerful they think those cards are. They can go through this list and spot any scores that are large disagreements from their own held opinions. From there, they can discuss with other people who can have a conversation about what the card is good for, etc, in the context of games of Dominion (which is probably the more valuable thing).

Sorting the data leads to conversations about which is better between Donate and King's Court, which IMO are completely pointless because it has no bearing on how to play real games of Dominion better -- they're both very good and that's all there is to say about comparing the two cards.


> Also, there are reasons beyond comparison to want to sort the data--after editing the table to sortable, I immediately used it to check if the data had been rescaled to have the means span from 0-10 or not.

This is a good reason for someone to sort the data on their own, but not a reason for the data to be presented in a way that encourages readers to sort it.

Anyone who wants to have these other discussions or compute these stats is free to do so (I'll even link a Google Sheet below which gives access to some more power tools and data categories, etc.) but the main point of all these ratings is to give a general idea of card power level from 0-10 and I don't think other things which can misrepresent what the data is trying to show belong on the page that is just presenting the data.

https://docs.google.com/spreadsheets/d/1R9eApE1O1977H6h8RvvoVl_PIMltWxlQDzdYPF3PQDE


---

markus: If you don't want people to compare KC and Donate, you should just cut the mean column. Median is also more robust to some outliers in your sample. If I want to know which cards are more relevant, I want to see which cards have a higher score. Sorting them alphabetically is not useful. Sort by median and you get exactly that. In addition, add some columns on whether a card is a village etc., and one can sort to see which are the more relevant villages.

---

AdamH: I can see the argument for removing the mean column -- it helps with a lot of the issues with the data. On the other hand, it doesn't eliminate all of the problems with sorting the data.

As for the rest of your suggestions, these would be very useful if there was a filter wiki tables -- without that I don't think it's useful to include here (especially if the table isn't going to be sortable).

If I edit this table again I'm going to modify the script I used to make the wiki table to at least include the expansion as a column. I'm toying with the idea of just putting the card name and the rating(s) in the same box, making it impossible to sort by card rating but maybe using the sortable table to function as a less-good way of filtering by the other fields that are available. This might be an acceptable compromise.


  • Adam, multiple people have told you that they want this table to be sortable by ranking. If that means you want your name taken off of it, then it's just some random ratings, and that's kind of pointless. This table is either going to have your name on it and be sortable by ranking, or it's not going to be on the wiki at all.

P.S. you can sign your post using four tildes in a row, like so: Werothegreat (talk) 11:48, 1 February 2019 (EST)

---

Take it down then.

-Adam

---

Wero, I asked you to delete the page instead of continuing to revert it back to a sortable table. Just delete the stupid thing already.

-Adam

---

Adam, I find your take on the sorting unpersuasive. The problems you describe like high variance are problems inherent to rating itself, not to comparison, and saying you don't see the uses for sorting given them doesn't mean those uses don't exist.

> The relevant comparison(s) to make here are between a card and the 0-10 scale. When cards get compared to each other is when it starts to deviate from what the data is trying to show. Sorting the data invites the reader to make only the "smaller" comparisons (comparisons between cards of similar rating) which are the least valid ones to make.

This is not the only way to use a sort to make comparisons. For instance, a very common practice in games is to use a sorted list as the basis for a tier list, in which you look for natural divisions in the data to carve out tiers, inviting comparison between cards that are far away on the scale (you will clearly see that KC is "better" than Transmute) while discouraging comparisons between nearby ones.

> This is a good reason for someone to sort the data on their own, but not a reason for the data to be presented in a way that encourages readers to sort it.

This is an unjustified assertion and completely backwards. This takes about a second to perform if the list is sortable on the wiki while it may take minutes to perform if someone has to download and sort it themselves. It's exactly the kind of task that a sortable table on the wiki makes most worthwhile.

On a broader note, I think I disagree philosophically with the mindset you're approaching this from. Wikis are public resources, and I think trying to gatekeep what people can and can't do with the info that you put on them is presumptuous and condescending, especially since it's clear you've been overlooking common use cases that involve sorting, such as checking for data rescaling and forming a tier list. Imo we should assume that people are in general better judges of what best serves their own purposes than we are, and so we should try to accommodate the broadest range of common use cases that we can.

My hope would be that this data can remain on the website for common use, in a sortable form. Taking it down over a relatively minor disagreement about its presentation seems extreme to me. I also think this discussion of removing your name from the page kind of misses the point of contributing to a wiki, where you're releasing the data to the community of people who will also contribute, and further changes don't have to represent an endorsement by you of those changes. After all, it does say right below this very edit window "If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here."

Panama (talk) 14:42, 7 February 2019 (EST)Panama

  • With Panama's response, I have reconsidered. I'm going to revert the table to being sortable, and keep the page on here, as clearly there is some desire to have this information available. What I will do, however, is amend my ultimatum from deleting the page if AdamH reverts it: I will instead block AdamH from editing for 1 month if he reverts it again, because hey, I can do that, and technically he is removing information from the page, which is one of the drop-down reasons I can use for blocking a user.

You're not going to win this, AdamH. Please don't make me take action that will make you hate me even more. Werothegreat (talk) 15:53, 7 February 2019 (EST)

---

I've talked to several other people about this, including Qvist. These people were actually willing to have conversations with me about what's going on here and what features they actually want from these ratings, as opposed to making threats and turning this into a competition.

Based on those conversations, people were asking for many features related to this data that a wiki table won't support. Maintaining the page is also an issue, since the poll is constantly open and is updated several times a year, I don't want to be responsible for keeping this wiki table up-to-date, especially given how hostile you've been, Wero.

So what I've done is this: I made a Google Doc that's suitable for sharing with anyone interested in the data. It contains the raw data as well as statistics on that data, which satisfy all of the features that people who were willing to have a conversation with me have actually asked for. And yes, the data is sortable by all fields, including rating.

The most important thing here is that people aren't going to sort the data by rating when what they're really trying to accomplish is some kind of filtering or something else (which the Google Doc supports). The next most important thing is that nobody has to maintain this wiki table whenever the data gets updated (which it already has).

So the table that was on the page before my edit is now obsolete and I'm no longer willing to maintain it. I'll continue to maintain the Google Doc, so I'm going to delete the table and replace it with a link to the Google Doc. If you want to block me from editing wiki pages based off that, go right ahead man. I'm ready to wash my hands of this situation

-AdamH

Personal tools
Namespaces

Variants
Views
Actions
Navigation
Toolbox