Diffusion quality diffuses in the digital public square

Introduction

Studies of online social influence have demonstrated that friends have important effects on many types of behavior in a wide variety of settings. However, we know much less about how influence works among relative strangers in digital public squares, despite important conversations happening in such spaces. This paper presents the results of a study on large public Facebook Pages where the authors randomly used two different methods - most frequent and social feedback - to order comments on posts. The authors found that the social feedback condition results in higher quality viewed comments and response comments.

After measuring the average quality of comments written by users before the study, the authors find that social feedback has a positive effect on response quality for both low and high quality commenters. The authors draw on a theoretical framework of social norms to explain the empirical result. It was also found that the social feedback condition led people to stay on topic.

Studies of social influence primarily focus on socially connected individuals, yet discussions on important issues often occur between relative strangers. In large public online discussions, which the authors term digital public squares, understanding social influence process is crucial for designing systems that encourage meaningful discussions.

This paper presents the results from a large-scale study on comments sections from public Facebook Pages. The experimental change is whether to display comments on posts ranked by social feedback (treatment) or most recent (control). The study makes the following contributions to understanding online discussions:

• Provides a framework for studying the quality of comments and demonstrate that text-only models can be used to predict or measure quality.
• Evaluate ranking methods on the dimension of quality shown to users and characterize how showing higher quality can improve user experience.
• Use pre-treatment information to distinguish between competing hypotheses of selective turnout versus within viewer quality change (i.e. social feedback causes changes in how they participate)
• Provides evidence that ranking affects the social norms operating in an online discussion environment, leading to increased quality and feedback, as well as improved quality of response comments. We additionally find that social feedback condition encourages increased relevance of response comments in certain cases
• Methodologically, the paper demonstrates that within-subjects designs can be used for studying discussions, with the benefit that we do not change ranking for all posts a user sees, or for all users that see a post. This allows ranking methods to be evaluated with minimal user experience change.

Previous research on social influence has established that product adoption, sentiment, political participation, and cultural taste are transmitted through online social networks. Studies have established that social influence can operate even in anonymous-interaction cases, both offline and online. Researchers have investigated methods for representing discussions in latent spaces, ranking comments by inferred community preferences, and modeling the lifecycle of discussion threads. From a content perspective, researchers have developed notions of quality in online discussions and identified patterns of trolling behavior in online forums.

Works in social science seek to understand how social norms affect social behavior, particularly in public spaces. When different norms are activated in otherwise identical places, different behavior outcomes are observed. It has been observed that the violation against one norm can encourage the violation of a separate norm. The research identifies two important classes of norms: descriptive and injunctive norms. In a social situation, the former indicates which behavior is most common whereas the latter provides information about which behavior is appropriate. A public space is defined as a space where citizens can come together as equals to exchange issues relating to the common good. Scholars have argued that the Internet has pluralized the public sphere into ”networked publics” organized around foci, which have properties (e.g. searchability) not found in offline discussions.

Authors are the people who wrote the comment that the viewer is currently seeing. The viewers are the subjects of the experiment whose behavior we are studying. The authors are producing the content that we are deciding how to order, providing different stimuli for the viewers.

Methods

The authors randomized the method for ranking discussions on these Page posts for viewer-post pairs. If we denote viewers $v$ and posts $p$, then each $(v,p)$ pair has a small chance of being in the test, and if it was included in the test, had an equal chance of being assigned to social feedback ranking or most recent ranking.

Both social feedback and most recent are available for viewers to choose on posts made by large Page via a drop down element at the top right of the comments section. The authors changed the default method for presenting post discussions while still allowing viewers the option to choose alternative ranking methods. $(v,p)$ pairs were placed in a single condition for the duration of the test, meaning that viewers of the same post’s comments resulted in the same default ranking. In conducting analyses, the authors analyzed all data in aggregate and present only aggregate measures here.

Measuring Discussion Quality

The method of discussion quality is focused on whether a comment adds to a conversation. The authors operationalize this in the context of a post, according to guidelines seen in the table below. The authors chose a ”know it when you see it” description of quality because they wanted the measure to reflect the raters’ intuitive judgements.

Latent quality is often the objective of recommender methods. In essence, the authors ask the raters how specific comments align with the latent notion of quality.

Results

The following variables are measured as outcomes of the test:

• Response comments: number of response comments written by viewers of the post
• Response likes: The number of likes by the viewer on any comment they see on a post in the test
• Viewed quality: The average model-predicted quality of the most highly ranked two or more comments viewed
• Response quality the quality average model-predicted quality of the response comments written by the viewer
• Response similarity: The cosine similarity between the test of the most highly ranked comments the viewer saw and response comments they wrote in response.

Interestingly, up until discussions reach 1000 comments, likes and shown quality increase in the control case, and social feedback widens the gap between the treatment and control cases. This suggests that there are basic dynamics of conversations present under most recent: as discussions grow larger, they become higher quality. Social feedback ranking accelerates this process, quickly leading to higher quality, more liked discussions.

Compared to discussions with 301 to 1000 comments, likes and shown quality decrease in the most recent condition, while the number of response comments grows very large, there is a pile-on effect of many individuals writing low-quality comments - perhaps tagging friends to alert them to a post of particularly broad appeal. For very large discussions displayed by social feedback, the increase in quality levels off, relative to the 301-1000 group. This could indicate that rich-get-richer effects have resulted in certain comments getting a huge number of likes, preventing new, high-quality comments from rising to the top.

The treatment either encourages high quality commenters to turn out a higher rate selective turnout or it encourages a fixed group of commenters to write higher quality comments.

The social feedback condition makes salient a descriptive social norm for higher-quality commenting.

Conclusion

The quality of dicussions diffuses in digital public squares. The result found is surprising because individuals generally do not know one another. This implies that improbing ranking methods has both first-order and second-order effects. Ranking improvesw the reading experience by displaying higher quality content for readers. Then, for the small fraction of individuals that choose to participate in the discussion, displaying better comments encourages higher-quality participation.