Social listening is changing the way economists talk about sexism and gender bias in the profession, illustrating how insights from online conversations can have massive offline impacts. Similarly, understanding conversations from audiences that your institution serves (e.g., professionals working in the field you’re preparing students for, alumni, or prospective students) can transform your marketing, program development, recruitment, or fundraising strategy.
I’m using a ground-breaking economics study of online conversation to help you better understand how conversation analysis works in the context of a real-world example. The wrap-up explores how the same methodology might apply to revenue-generating opportunities on a college campus.
Forums Hold Deep Conversation Insights
Before diving super deep into economics social listening data, let's set the stage by talking about forums. Economists, particularly graduate students, regularly discuss the state of the profession, their job search, papers, and conferences on the Economic Job Market Rumors (EJMR) forum. There are currently more than 30,000 discussion threads from the last eight years. It’s a treasure trove of conversation data.
Online discussion forums are not just an economics phenomenon—niche forums exist for many topics and industries. Here are just a few examples.
- Student Doctor for medical students
- r/ApplyingToCollege for prospective students
- Minecraft Forum for avid Minecrafters
- D3Boards for fans of NCAA Division III athletics
- AVS Forum for home theater enthusiasts
I’m an avid user of forums. True story—I met my husband over ten years ago on BrewerFan.net, a forum dedicated to fans of the Milwaukee Brewers baseball team and their minor league affiliates. Forums of varying sizes exist for nearly every topic. BuiltWith estimates that almost 80,000 sites run forum software, with about half located in the United States. While some of you may think that forums have been replaced by Facebook groups, that’s not the case for all populations. Twenty-two percent of 18–29 year-olds use Reddit, the largest discussion forum on the internet. With over 200 million unique visitors per month to almost 150,000 active subreddits, Reddit ranks among the top 25 sites in the world.
In addition to topics you and your institution are interested in, people also talk about your campus on forums. Our 2019 Online Conversation Benchmarks found that after removing Twitter and Instagram, on average 14 percent of the remaining conversation about college campuses appeared on forums, and for some campuses it was much higher.
Forum Conversations Are Often Raw and Unfiltered
The conversations on these forums varies in depth, complexity, and general appropriateness. Among economists, there’s been a general feeling that the comments on EJMR were sexist and derogatory towards women. But economists don’t operate on feelings; they need data.
In 2017, Alice Wu, then a UC Berkeley economics student, decided to test her hypothesis that there was gender stereotyping on the EJMR forum. While she followed methodology that in the academic research realm is classified as text mining, natural language processing, machine learning, and econometrics, when I heard about her research I immediately thought, “That is social listening!” I think you’ll agree with me.
Methodology to Collect and Analyze Forum Conversations
Social listening methodology made sense for this research because, as Alice wrote, “even in lab settings gender biases may not manifest in easily observable behavior … if researchers were able to observe interpersonal interactions in a controlled setting, subjects who are concerned about political and social correctness would not necessarily reveal their true stereotype beliefs.” Enter an online forum, in this case EMJR, where “anonymity presumably eliminates any social pressure participants may feel to edit their speech, and thus creates a natural setting to capture what people believe but would not openly say.”
Alice had a glut of data at her fingertips, and she went to work collecting it—harvesting the anonymous opinions of her fellow economists. She collected 1,143,416 posts from 131,913 threads that were created or updated from January 2014 to July 2016. If you’re not familiar with forums, threads are a group of posts on a given subject. To compare them to a long email chain, a thread is the subject line and a post is each individual email within that chain.
Next, Alice determined which posts were related to gender. To do that she identified gender classifiers that appeared within the top 10,000 most-used words in the data set. She found 44 words associated with females and 134 words associated with males. These words could be pronouns, names of prominent economists with a known gender, gender identifiers like men or women, or relational words like sister or bro. These identifiers (along with more sophisticated modeling to appropriately classify posts with both female and male words) allowed her to code posts as female or male for further analysis. She found that more than half the threads included gender related posts, and about 20 percent of all posts in her dataset related to gender.
Then, she conducted further analysis to identify what topics were covered in the posts relating to gender, as well as the words that were used most often in posts about men and women. She classified the top 10,000 words into 15 categories. For example, words like beautiful, handsome, attractive, body, and fat went in the “physical attributes” category. The 15 categories were further grouped into six topics of interest. The two topics of interest her study focused on are academic/professional and personal/physical. The academic/professional topic included three categories: economics, professional, and general discussion about academics, while the personal/physical topic had three different categories: physical attributes, personal information, or gender-related.
Online Forum Conversation Confirms Gender Bias in the Economics Profession
Alice explains that innovative research methods combined with rich online conversation data sets can “get at a phenomenon that is otherwise difficult to quantify.” She wanted to investigate and quantify the “feeling” that women were discussed differently than men on the EJMR forum. She quantified it, alright.
Her research found that posts discussing women contain 43 percent less academic or professional terms, and 192 percent more words related to personal information or physical attributes. She also found that the words most strongly associated with threads about women are mostly inappropriate—words she wouldn’t want to say out loud in her advisor’s office. I can publish some of them here to give you an idea: attractive, pregnant, bang, feminazi, raped, ugly. You get the idea. Additionally, she found that when high-profile economists are discussed on the forum, women received more attention than their male counterparts.
The full text of Alice’s research can be found in her honors thesis, which was covered by The New York Times—twice!—and discussed by prominent economists, including Janet Yellen (former Chair of the Federal Reserve), at the annual meeting of the American Economics Association. Her paper concludes that there is a “need for changes to maintain an inclusive online environment for everyone in the academic community. The casual setting of this online forum cannot be an excuse for gender stereotyping conversations, and the freedom to express one’s opinions anonymously should not be abused to create a sense of isolation, which can be discouraging and harmful to the academic and professional development of all genders.”
This research has already led to real-world change, including the creation of the American Economic Association’s professional code of conduct and sponsored programs to get more women into economics. If you want to learn more about the research and hear from Alice, listen to Episode 910 of the Planet Money podcast, Economics, Sexism, Data. Alice is now a PhD candidate at Harvard, where she appears to be continuing her research in this area. The Sonarians will definitely be following it.
Uses for Forum Conversation Insights in Higher Education
From a research perspective, niche forums are an online focus group of passionate stakeholders freely expressing their opinions. When considering a dedicated subject matter forum as a data source, opportunities for campuses could include:
- Gather intelligence about the desires, regrets, and upcoming trends in specific professional areas to inform program design. What are entry-level engineers at tech start-ups saying about their skillsets and career prospects? What do public health professionals feel they need (education, experience, certifications) to move into leadership positions? What are stay-at-home parents looking to re-enter the workforce looking for in distance education programs? What do aspiring esports athletes have to say about collegiate offerings?
- Understand the historical and real-time conversation about programs that compete with yours for rankings. What individuals and publications seem to influence the conversation? Is your program mentioned more or less often than your peers this year? What faculty members or graduates are getting the most attention? Does volume and sentiment of program conversation amongst academics correlate with changes in rankings?
- Identify opportunities for continuing education and certificate programs. What do hobbyists want to learn about their area of interest? What experiences are attracting a lot of attention, and how can your institution recreate those as a course offering? Who is seen as trustworthy and credible within the special interest community, and would they make a suitable instructor?
These are just a few ideas we’ve had while speaking with campus professionals. The truth of the matter is, the opportunities for research in this area are potentially limitless—Reddit can even inform your content marketing strategy. If you’re interested in pursuing social listening research that your institution can use to impact offline outcomes and generate revenue, get in touch. When you work with Campus Sonar, we provide actionable insights that support your strategic priorities. Don’t worry about planning the research, buying fancy software to collect the data or training analysts to make sense of it—we’ve got that taken care of so you can spend your time crafting and implementing strategies that work.
