How to measure controversy in an on-line discussion?

This is the first of a series of blog entries on controversy modeling from a computer science perspective.

We will first take a look into measures which intent to quantify the degree of controversy in online discussions. For this task one can take advantage of the fact that many website which allow users to discuss about specific topics show the structure of the discussion with indentation. The following figure gives an example of this structure in Slashdot, a popular website for people interested in reading and discussing about technology and its ramifications.

The nested structure of comments on Slashdot

If a comment B is indented below another comment A, B is a reply to the content of comment A. The resulting structured discussions can become very complicated as the next figure shows.

A typical discussion on Slashdot
Growth of a nested discussion on Slashdot. The center node corresponds to a short news story. Nodes connecting to this central node represent comments to this news story, while the other nodes represent replies connected to their parent comment. 

The question is now how to use this structural information to quantify the amount of controversy in the discussion.  We assume that the deeper a discussion becomes the more controversial it should be. Deeper means here longer branches in the radial tree representation of the discussion or in simpler words longer sequences of replies, replies to replies, etc.

But using just the maximal depth of a discussion is too sensible to outliers, as for example two users who become entangled in an otherwise insignificant and unrepresentative discussion can easily reach a considerable depth through repeated replies to each others comments.

To avoid this Gómez et al. (2008) introduced a generalization of the h-index (usually used to asses the performance of researchers) as a balanced depth measure of the discussion. It is defined as follows

The h-index of a discussion is the maximum number h which fulfills that there are at least h comments at depth h but there are not h+1 or more comments at depth h+1.

The following figure illustrates how this measure is calculated for a small example thread.

Calculation of the h-index of an example discussion on Slashdot. The thread on the left is represented as hierarchical tree on the right. The red line indicates the h-index of the discussion. The discussion has 6 comments at a depth of at 3, but only 3 at a depth of 4, which implies that its h-index is 3.

The discussion has 9 comments in both first and second levels, 6 comments in the third level and 3 comments in the fourth level, which gives an h-index of 3. This discussion has a maximum depth of 17, but the balanced depth measure reduces this value significantly.
We finish this entry with a list of the most controversial topics on Slashdot from August 2005 to August 2006 according to this measure.

Top-15 controversial discussions according to their h-index and their corresponding ranks (within parenthesis) according to the number of comments or their maximum depth.

Comparing the rankings of h-index, max. depth and number of comments, it becomes evident that that neither the depth nor the most commented discussions are those with the highest controversy according to the h-index measure.

Reference

Gómez V., Kaltenbrunner A., López V. (2008)
Statistical analysis of the social network and discussion threads in Slashdot,
WWW2008 17th International World Wide Web Conference, Social Networks and Web 2.0 track, Beijing, China.

3 Responses to “How to measure controversy in an on-line discussion?”

  1. Great post!

    I like the approach of identifying controversial topics through their higher rate of ‘nested replies’ and I love the idea of using a generalized h-index to rule out outliers.
    This is particularly interesting because h-index is a much discussed (and in fact highly controversial) indicator. This means that we can draw on h-index literature to find even better indicators. For example, see http://www.harzing.com/pop_hindex.htm.

    One alternative indicators that seems particularly interesting for early detection of controversies if the so-called m-quotient. The m-quotient is just the h-index divided by the number of years an academic has been active (in our case, that would be the duration of the discussion). This should help identify topics that are generating a lot of discussion in a little time.

    It would be very interesting to test this methods on the slashdot post concerning climate (http://slashdot.org/index2.pl?fhfilter=climate). Is that feasible?

    Also, I know that Eric, Esther, Bernard and probably someone else in Amsterdam is working along the same line but on Wikipedia. Can you share your ideas? would it be possible to apply the same type of detection to Wikipedia (a much larger and representative community)?

  2. Barcelona Media says:

    Thank you for your comment, Tommaso.

    The next entry will be precisely about applying this (and other) metrics on Wikipedia, but we’d be very interested in the work of Amsterdam on this subject.
    We are currently working precisely on a very similar idea to the m-index you mention. I expect some first results around end of January.

    Regarding applying it to Slashdot:
    The discussion in Slashdot evolve very fast, ~50% of all comments appear within the first two hours (~80% with 10 hours) after a news story is published (see Figures 3 and 4 in http://arxiv.org/pdf/0708.1579 ). Under this small timescales the human circadian rhythm prevails over any other temporal dynamics. Basically the speed of the discussion depends on the time of the day a story is published.

  3. Barcelona Media says:

    BTW: Can we add the possibility to use nested comments in this blog?
    Something like that maybe?
    http://bavotasan.com/2009/how-to-add-nested-comments-to-your-wordpress-theme/

Leave a Reply

Spam protection by WP Captcha-Free