Social listening is changing the way economists talk about sexism and gender bias in the profession, illustrating how insights from online conversations can have massive offline impacts. Similarly, understanding conversations from audiences that your institution serves (e.g., professionals working in the field you’re preparing students for, alumni, or prospective students) can transform your marketing, program development, recruitment, or fundraising strategy.
I’m using a ground-breaking economics study of online conversation to help you better understand how conversation analysis works in the context of a real-world example. The wrap-up explores how the same methodology might apply to revenue-generating opportunities on a college campus.
Before diving super deep into economics social listening data, let's set the stage by talking about forums. Economists, particularly graduate students, regularly discuss the state of the profession, their job search, papers, and conferences on the Economic Job Market Rumors (EJMR) forum. There are currently more than 30,000 discussion threads from the last eight years. It’s a treasure trove of conversation data.
Online discussion forums are not just an economics phenomenon—niche forums exist for many topics and industries. Here are just a few examples.
I’m an avid user of forums. True story—I met my husband over ten years ago on BrewerFan.net, a forum dedicated to fans of the Milwaukee Brewers baseball team and their minor league affiliates. Forums of varying sizes exist for nearly every topic. BuiltWith estimates that almost 80,000 sites run forum software, with about half located in the United States. While some of you may think that forums have been replaced by Facebook groups, that’s not the case for all populations. Twenty-two percent of 18–29 year-olds use Reddit, the largest discussion forum on the internet. With over 200 million unique visitors per month to almost 150,000 active subreddits, Reddit ranks among the top 25 sites in the world.
In addition to topics you and your institution are interested in, people also talk about your campus on forums. Our 2019 Online Conversation Benchmarks found that after removing Twitter and Instagram, on average 14 percent of the remaining conversation about college campuses appeared on forums, and for some campuses it was much higher.
The conversations on these forums varies in depth, complexity, and general appropriateness. Among economists, there’s been a general feeling that the comments on EJMR were sexist and derogatory towards women. But economists don’t operate on feelings; they need data.
In 2017, Alice Wu, then a UC Berkeley economics student, decided to test her hypothesis that there was gender stereotyping on the EJMR forum. While she followed methodology that in the academic research realm is classified as text mining, natural language processing, machine learning, and econometrics, when I heard about her research I immediately thought, “That is social listening!” I think you’ll agree with me.
Social listening methodology made sense for this research because, as Alice wrote, “even in lab settings gender biases may not manifest in easily observable behavior … if researchers were able to observe interpersonal interactions in a controlled setting, subjects who are concerned about political and social correctness would not necessarily reveal their true stereotype beliefs.” Enter an online forum, in this case EMJR, where “anonymity presumably eliminates any social pressure participants may feel to edit their speech, and thus creates a natural setting to capture what people believe but would not openly say.”
Alice had a glut of data at her fingertips, and she went to work collecting it—harvesting the anonymous opinions of her fellow economists. She collected 1,143,416 posts from 131,913 threads that were created or updated from January 2014 to July 2016. If you’re not familiar with forums, threads are a group of posts on a given subject. To compare them to a long email chain, a thread is the subject line and a post is each individual email within that chain.
Next, Alice determined which posts were related to gender. To do that she identified gender classifiers that appeared within the top 10,000 most-used words in the data set. She found 44 words associated with females and 134 words associated with males. These words could be pronouns, names of prominent economists with a known gender, gender identifiers like men or women, or relational words like sister or bro. These identifiers (along with more sophisticated modeling to appropriately classify posts with both female and male words) allowed her to code posts as female or male for further analysis. She found that more than half the threads included gender related posts, and about 20 percent of all posts in her dataset related to gender.
Then, she conducted further analysis to identify what topics were covered in the posts relating to gender, as well as the words that were used most often in posts about men and women. She classified the top 10,000 words into 15 categories. For example, words like beautiful, handsome, attractive, body, and fat went in the “physical attributes” category. The 15 categories were further grouped into six topics of interest. The two topics of interest her study focused on are academic/professional and personal/physical. The academic/professional topic included three categories: economics, professional, and general discussion about academics, while the personal/physical topic had three different categories: physical attributes, personal information, or gender-related.
Alice explains that innovative research methods combined with rich online conversation data sets can “get at a phenomenon that is otherwise difficult to quantify.” She wanted to investigate and quantify the “feeling” that women were discussed differently than men on the EJMR forum. She quantified it, alright.
Her research found that posts discussing women contain 43 percent less academic or professional terms, and 192 percent more words related to personal information or physical attributes. She also found that the words most strongly associated with threads about women are mostly inappropriate—words she wouldn’t want to say out loud in her advisor’s office. I can publish some of them here to give you an idea: attractive, pregnant, bang, feminazi, raped, ugly. You get the idea. Additionally, she found that when high-profile economists are discussed on the forum, women received more attention than their male counterparts.
The full text of Alice’s research can be found in her honors thesis, which was covered by The New York Times—twice!—and discussed by prominent economists, including Janet Yellen (former Chair of the Federal Reserve), at the annual meeting of the American Economics Association. Her paper concludes that there is a “need for changes to maintain an inclusive online environment for everyone in the academic community. The casual setting of this online forum cannot be an excuse for gender stereotyping conversations, and the freedom to express one’s opinions anonymously should not be abused to create a sense of isolation, which can be discouraging and harmful to the academic and professional development of all genders.”
This research has already led to real-world change, including the creation of the American Economic Association’s professional code of conduct and sponsored programs to get more women into economics. If you want to learn more about the research and hear from Alice, listen to Episode 910 of the Planet Money podcast, Economics, Sexism, Data. Alice is now a PhD candidate at Harvard, where she appears to be continuing her research in this area. The Sonarians will definitely be following it.
From a research perspective, niche forums are an online focus group of passionate stakeholders freely expressing their opinions. When considering a dedicated subject matter forum as a data source, opportunities for campuses could include:
These are just a few ideas we’ve had while speaking with campus professionals. The truth of the matter is, the opportunities for research in this area are potentially limitless—Reddit can even inform your content marketing strategy. If you’re interested in pursuing social listening research that your institution can use to impact offline outcomes and generate revenue, get in touch. When you work with Campus Sonar, we provide actionable insights that support your strategic priorities. Don’t worry about planning the research, buying fancy software to collect the data or training analysts to make sense of it—we’ve got that taken care of so you can spend your time crafting and implementing strategies that work.