July 21, 2013
In a new paper, recently published in the open access journal PLOSONE, Benjamin Mako Hill and I build on new research in survey methodology to describe a method for estimating bias in opt-in surveys of contributors to online communities. We use the technique to re-evaluate the most widely cited estimate of the gender gap in Wikipedia.
A series of studies have shown that Wikipedia’s editor-base is overwhelmingly male. This extreme gender imbalance threatens to undermine Wikipedia’s capacity to produce high quality information from a full range of perspectives. For example, many articles on topics of particular interest to women tend to be under-produced or of poor quality.
Given the open and often anonymous nature of online communities, measuring contributor demographics is a challenge. Most demographic data on Wikipedia editors come from “opt-in” surveys where people respond to open, public invitations. Unfortunately, very few people answer these invitations. Results from opt-in surveys are unreliable because respondents are rarely representative of the community as a whole. The most widely-cited estimate from a large 2008 survey by the Wikimedia Foundation (WMF) and UN University in Maastrict (UNU-MERIT) suggested that only 13% of contributors were female. However, the very same survey suggested that less than 40% of Wikipedia’s readers were female. We know, from several reliable sources, that Wikipedia’s readership is evenly split by gender — a sign of bias in the WMF/UNU-MERIT survey.
In our paper, we combine data from a nationally representative survey of the US by the Pew Internet and American Life Project with the opt-in data from the 2008 WMF/UNU-MERIT survey to come up with revised estimates of the Wikipedia gender gap. The details of the estimation technique are in the paper, but the core steps are:
- We use the Pew dataset to provide baseline information about Wikipedia readers.
- We apply a statistical technique called “propensity scoring” to estimate the likelihood that a US adult Wikipedia reader would have volunteered to participate in the WMF/UNU-MERIT survey.
- We follow a process originally developed by Valliant and Dever to weight the WMF/UNU-MERIT survey to “correct” for estimated bias.
- We extend this weighting technique to Wikipedia editors in the WMF/UNU data to produce adjusted estimates of the demographics of their sample.
Using this method, we estimate that the proportion of female US adult editors was 27.5% higher than the original study reported (22.7%, versus 17.8%), and that the total proportion of female editors was 26.8% higher (16.1%, versus 12.7%). These findings are consistent with other work showing that opt-in surveys tend to undercount women.
Overall, these results reinforce the basic substantive finding that women are vastly under-represented among Wikipedia editors.
Beyond Wikipedia, our paper describes a method online communities can adopt to estimate contributor demographics using opt-in surveys, but that is more credible than relying entirely on opt-in data. Advertising-intelligence firms like ComScore and Quantcast provide demographic data on the readership of an enormous proportion of websites. With these sources, almost any community can use our method (and source code) to replicate a similar analysis by: (1) surveying a community’s readers (or a random subset) with the same instrument used to survey contributors; (2) combining results for readers with reliable demographic data about the readership population from a credible source; (3) reweighting survey results using the method we describe.
Although our new estimates will not help us us close the gender gap in Wikipedia or address its troubling implications, they give us a better picture of the problem. Additionally, our method offers an improved tool to build a clearer demographic picture of other online communities in general.
This weekend, Andrés and I attended the CrowdCamp Workshop at CHI in Austin, Texas. The workshop was structured a lot like a hackathon, with the objective being to work in teams to produce projects, papers, or research.
The group I worked with coalesced around a proposal made by Niki Kittur, who suggested that we envision how crowdsourcing and distributed work contribute to solving grand challenges, such as economic inequality and the ongoing impact of the 2008 financial crisis.
We then spent the better part of the weekend outlining an ambitious set of scenarios and goals for the future of crowdwork.
While many moments of our conversation were energizing, the most compelling aspects derived from the group’s shared desire to imagine crowdwork and distributed online collaboration as potentially something more than the specter of alienated, de-humanized piece-work that it is frequently depicted to be.
To spur our efforts, we used a provocative thought experiment: what it would take for crowdwork to facilitate fulfilling, creative, and sustainable livelihoods for us or our (hypothetical or real) children?
Despite the limits of this framing, I think it opened up a discussion that goes beyond the established positions in debates about the ethics and efficiencies of paid crowdsourcing, distributed work, and voluntary labor online (all of which are, to some extent, encompassed under the concept of crowdwork in this case). It also hellped us start imagining howwe, as designers and researchers of crowdwork platforms and experiences, would go about constructing an ambitious research agenda on the scale of a massive project like the Hadron Collider.
If everything goes according to plan, this effort will result in at least a paper within the coming few weeks. Assuming that’s the case, our group will be sharing more details about the workshop and our vision of the future of crowdwork soon.
The paper builds on Ned’s work at Mathworks developing collaborative programming competitions for the MATLAB community. Adopting “the perspective of the code” it analyzes what happens when you set a horde of geeks loose on a fun, challenging programming problem in a networked collaborative environment.
To sum up my reactions really briefly, I thought the paper was an exciting step in the process of looking under the hood of collaborative knowledge production. Gulley and Lakhani argue that as programmers improved the performance of code relative to a discreet problem, they did so through “tweaks” and “leaps.”
“Tweaks” represent small refinements that improve the performance of existing code; “Leaps” represent more sudden and large-scale advances in performance (usually driven by introducing a more substantive or extensive change in the code).
Tweakers and Leapers benefit from each other’s work, but the biggest beneficiary of their combined interactions was the code itself. Within one week of the competitions, thousands of eyeballs had produced startling solutions to complex algorithmic problems.
There’s a lot more to be learned from this kind of work – especially from the sort of experimental data created in the setting of these sort of large-scale collaborative games. In particular, I’m interested in thinking about how programmers (whether as individuals or communities) adapted to the challenges over time. It seems like it might be possible to design a game that could test whether efficient collaborative problem solving techniques “evolved” over the course of the game(s). In addition, it would be fascinating to test the results of this kind of collaboration against those produced by more hierarchical or individuated models of innovative work.
Look for links to the soon-to-be-published version of the paper on the “publications” section of Karim’s HBS faculty page.
In the meantime, I’m told that video and audio of today’s presentation should be available on the Berkman Center’s “interactive section” by tomorrow afternoon at the latest.
May 15, 2008
Back at the Berkman@10 Conference after lunch and watching the Q&A following a discussion of networked cooperation between Yochai Benkler and Jimmy Wales.
Meanwhile, over on twitter & the conference backchannel, people seem to be getting bent out of shape about the fact that Wales made a comment implying that he thought “crowdsourcing” was a bad way to think about networked collaboration. It sounded to me that what Wales meant was that the way some folks in the private sector talk about crowdsourcing (hypothetical sample quote: it’s like outsourcing to India, only cheaper…) misses the point of cooperative social action entirely.
I’m sympathetic to that idea (for many reasons). First and foremost because the possibility of distributed peer-production of knowledge (like Wikipedia) raises a number of fundamental questions about the nature of business and markets. In theory, we’ve wound up developing markets to resolve a large scale collective action problem of goods distribution.
As Esther Dyson is currently clarifying in her comment, the big problem is that many folks in the private sector just don’t understand that networked collaboration offers an alternative to how we build social institutions of exchange. Instead, some folks think its just another tool for profit-making and (in the worst instances) rent-extraction.
What if networked cooperation made it so that scarcity were not a precondition of successful (i.e. profitable) exchange? What if we could do better than markets (as we currently know them)?
That’s the promise of the commons.
Yesterday, Kos responded to the pro-Clinton strikers. His post argues the following:
- The site has never claimed to include everyone on the left or all Democrats.
- The site has embraced a particular vision of political mobilization and the transformation of the Democratic Party (50 state strategy; grassroots oriented; change the DLC; reject consultants; no PAC money, etc.). Hillary Clinton actively opposes that vision.
- Clinton is not winning the primaries and cannot win without dividing the party and staging a coup against the popular vote via superdelegates.
- Clinton’s response has been to foment “civil war” within the party and for that she deserves whatever the blogosphere, Keith Olbermann, and others can throw at her.
The long quote below sums it all up nicely. The bold text was in the original:
To reiterate, [Clinton] cannot win without overturning the will of the national Democratic electorate and fomenting civil war, and she doesn’t care.
That’s why she has earned my enmity and that of so many others. That’s why she is bleeding super delegates. That’s why she’s even bleeding her own caucus delegates (remember, she lost a delegate in Iowa on Saturday). That’s why Keith Olbermann finally broke his neutrality. That’s why Nancy Pelosi essentially cast her lot with Obama. That’s why Democrats outside of the Beltway are hoping for the unifying Obama at the top of the ticket, and not a Clinton so divisive, she is actually working to split her own party.
Meanwhile, Clinton and her shrinking band of paranoid holdouts wail and scream about all those evil people who have “turned” on Clinton and are no longer “honest power brokers” or “respectable voices” or whatnot, wearing blinders to reality, talking about silly little “strikes” when in reality, Clinton is planning a far more drastic, destructive and dehabilitating civil war.
People like me have two choices — look the other way while Clinton attempts to ignite her civil war, or fight back now, before we cross that dangerous line. Honestly, it wasn’t a difficult choice. And it’s clear, looking at where the super delegates, most bloggers, and people like Olbermann are lining up, that the mainstream of the progressive movement is making the same choice.
And the more super delegates see what is happening, and what Clinton has in store, the more imperative it is that they line up behind Obama and put an end to it before it’s too late.
I agree with Kos’ assessment of the primary situation and the problems with the Clinton campaign’s reprehensible actions. I also agree that the pro-Clinton “strike” on the site is a violation of the norms established many years ago. This was clear from Allegre’s diary entry announcing the strike, in which s/he argues for a strange vision of Democratic unity in which party members don’t criticize each other (in a really bizarre twist, Allegre then mis-attributes that idea of unity to Barack Obama…this is polemical bunk). The Daily Kos leadership and community have never embraced that kind of vision. From a strategic perspective, I agree that they never should.
Kos’ post interests me for other reasons then. In it, he re-iterates the norms governing the community through a reference to the founding ideals of the site and an extension of those ideals to the current primary election situation. The preservation of the site’s original ideals depends on such occasional interventions from the community leader. In turn, the ideals and norms maintain the basis for large-scale collaboration and conversation.
But if that’s the case, does it negate what I wrote earlier about the significance of defection from large-scale collaborative communities? I don’t think so. Highly symbolic defections like this one still matter even if they are not grounded in an accurate interpretation of community norms. This skirmish, no matter how mundane or over-blown it has been, is part of the ongoing process of managing discursive production on the site.