Some initial thoughts on the Otey vs. CrowdFlower case

The question of whether paid crowd work violates U.S. employment and minimum wage laws may finally make it into court thanks to Christopher Otey, an Oregon resident who is suing CrowdFlower Inc. for wages he claims the company owes him as an “employee.”

You can (and should) read the full text of Otey’s complaint or coverage of the story on Crowdsourcing.org or MissionLocal.

I have a few preliminary, and mostly mixed, feelings about this. However, I should preface everything by saying that (1) I have known one of the defendants named in the suit, CrowdFlower CEO Lukas Biewald, for many years through mutual acquaintances at Stanford, where we were both enrolled at the same time; and (2) I worked as a paid, independent consultant with CrowdFlower on several projects between 2008-2011. That said, I have never held, nor hold at this time, any material interest, financial or otherwise, in the company.

My initial reaction is that I can’t believe it’s taken this long for someone, somewhere in the United States to sue one of the companies engaged in distributing paid crowdsourcing work for violation of the Fair Labor Standards Act (FLSA). Smart lawyers like Alek Felstiner and Jonathan Zittrain have been making some form of the argument that this is a major issue for Crowdsourcing for at least three years now. Felstiner even made the case in a series of posts on CrowdFlower’s blog here, here, and here in 2010. I am hardly the only person to regard as remarkable the fact that a whole venture-funded industry has sprung up around a set of activities that, on the surface, seem to resemble a massive minimum wage violation scheme.

At the same time, there are a lot of reasons to believe that crowdsourcing represents a fundamentally different sort of phenomenon than the varieties of “work” and workplace abuses the US congress sought to regulate with the FLSA back in 1938. For starters, crowd work is radically flexible – in terms of time and location – as well as minimal in terms of the commitment, skill, and obligations required of workers. As a result, it’s not clear that the relationships established between requesters and providers of work in this context are really anything like relational contracts that exist between traditional employers and employees. Crowd workers do what they for a variety of reasons, in a variety of ways, and under a variety of conditions, making it pretty hard to determine whether they ought to be considered employees of the organizations that may play a role in compensating them for their efforts (and this is potentially an important point since CrowdFlower plays something of a middle-man role between the individuals and companies that post tasks to its site and those who complete the tasks and receive compensation in exchange for their labor).

One particular challenge posed by the suit and the fact that Otey and his attorneys have chosen to seek compensation under US minimum wage laws ($7.50 per hour). Depending on the outcome, the impact of a ruling against CrowdFlower could therefore make paid crowd work as it exists today financially impractical within the United States. While such a ruling might represent a crucial step in enforcing legal, ethical, and financial standards of fairness in online environments, it might also undermine the growth of a valuable source of future innovation, employment, research, and creativity. Crowd-based systems (whether paid or unpaid) of distributed information creation, processing, and distribution have accounted for some of the most incredible accomplishments in the short history of the Internet, including Wikipedia, ReCaptcha, Flickr, Threadless, Innocentive, Kiva, Kickstarter, YouTube, Twitter, and the Google search engine.

As some colleagues and I have argued in a forthcoming paper, The Future of Crowd Work, there are many ways in which paid crowd work as it exists today does not look like the kind of job you would necessarily want your child to take on as a career.  And yet, while crowd work is very, very far from ideal by almost any standard, I would be disappointed if the impact of this case somehow resulted in the destruction of the industry and the stifling of the innovative research and applications that have developed around it. The outcome will boil down to the ways in which paid labor – even flexible, remote, and relatively straight-forward tasks that are paid only $0.01 – is regulated as compared with volunteer labor.

Journatic and the future of crowdsourced journalism

newsroom panorama by David Sim (http://www.flickr.com/photos/victoriapeckham/)

Crowdsourcing, outsourcing, and other sorts of distributed work have long since made inroads into professional journalism, but a recent scandal involving a few major metropolitan newspapers outsourcing their local reporting to a company named Journatic reveals the scope and extent of those inroads.

Since This American Life first broke the story a couple of weeks ago, the details of the Journatic story have made their way all over the Internet (See, e.g., coverage from Poynter, Romenesko, and Gigaom for some of the more thoughtful examples).

The basics are straightforward: Journatic is a company that specializes in generating content for a variety of purposes, among them local news stories (they also have a sister company called Blockshopper that provides a similar service for real estate listings). It seems that typically a client – say, a major U.S. newspaper like the Chicago Tribune, for example – contracts with Journatic, which then hires dozens of independent subcontractors (mainly in the Philippines and the U.S.) who construct and edit hyperlocal news items in a distributed, piecemeal fashion before passing the finished product back to the client for publication.

You can get a much better feel for the process by listening to the TAL interviews with Journatic editor Ryan Smith, or by reading Smith’s tendentious editorial about his experience (has has subsequently quit working for Journatic, although – interestingly – he was not fired or even reprimanded for his efforts to publicly criticize the company’s practices and products).

The stickiest part of the scandal seems to be that the Trib, along with several other major metropolitan dailies (the San Francisco and Houston Chronicles as well as the Chicago Sun Times) , had been printing these stories under false by-lines (such as Jake Barnes – the name of a famous Hemingway character), which violates the paper’s own ethical standards.

I find the story pretty engaging for several reasons:

The fact that Journatic figured out how to crowdsource journalism is actually pretty impressive. Some friends at CMU have been trying for a while now to generate magazine-style writing using workers on Amazon’s Mechanical Turk. Likewise, I’d like to develop and test methods for crowdsourcing peer review of academic papers. Apparently, the folks at Journatic have already solved many of the practical problems involved in performing a complex knowledge-based task like reporting using a globally distributed workforce of highly variable skill.

Second, despite the rhetoric surrounding the story, Journatic is neither the end of journalism as we know it nor its salvation. While I share the concerns voiced by Smith, TAL reporter Sarah Koenig, and others over the wages paid to Journatic’s Philippino contractors as well as the confusion about the Tribune’s apparent willingness to buck its own editorial policies about attribution in this case, these issues need to be distinguished from questions about whether crowdsourcing is “bad” or “good” for the future of media. I believe the emergence of companies specializing in crowdsourced journalism is merely another wrinkle in a complex organizational ecosystem where incumbent firms are struggling to retain some sort of comparative, competitive advantage in the face of declining revenues. When you consider Journatic in the context of other experiments in crowdsourced journalism, such as some of ProPublica’s distributed reporting project, CNN’s iReports, or even the political blogosphere, paying workers around the world to assemble stories sounds less like a violation of basic journalistic principles and more like the latest in a long line of process innovations that might or might not help to reinvent the field.

Last, but not least, many of us (myself included) may not like the fact that the cost of local news coverage has exceeded the demand in many places, but I think there’s got to be a more effective response than petitioning Sam Zell to stop outsourcing. Instead, I’d like to see a combined effort to improve Journatic’s models of content production in order to (1) address the ethical concerns raised in the Tribune scandal; (2) improve the quality of coverage in order to correct some of the terrible reporting practices documented by Smith in his op-ed; and (3) more effectively integrate teams of remote and on-site local reporters.

Ultimately, you can’t ignore the fact that Journatic smells bad. They paid off contractors not to talk to the media a few months ago, provide SEO and content farm services on the backs of cheap overseas labor, and when faced with complaints about the fact that their real estate listing service, BlockShopper, violated people’s privacy, they responded by issuing a Zuckerbergian declaration against expectations of privacy online and hiding the identities of their writers. Oh, and they also hide their company’s website from Google’s robots (go to http://journatic.com and use the “view source” option in your browser to see their robots.txt policy).

That said, the whole situation offers a chance to think about what a more responsible, ethical, and constructive version of crowdsourced journalism could look like. For that reason alone, I think Journatic deserves even more attention than it has already received.

Five things I learned this week (back in the East Bay edition)

Bike Lane, 41st Street, Oakland. By Eric Fischer

This past week, I returned to the East Bay, where I had the honor of participating in the 149th commencement exercises of the University of California, Berkeley. After a week on the road at CHI, a high speed tour of Chicago, and the chaos of graduation festivities, I’m happy to finally have had the chance to catch up on a little email and bike riding this weekend (not in that order). Here are five things I learned about along the way:

My cousin, Megan Cohen, is not only a badass playwright (the most produced female playwright in the Bay Area!) , but also happens to be a badass interview subject. She had me at: “I write, basically, like I’m screaming one last message out before being hit by a truck.”

Carrotmob seems like an intriguing activist application of crowdsourcing aimed at facilitating citizen-driven enhancements of communities and businesses. The basic idea is that if you can find enough people (a mob) to spend their money (the carrot) at a given place and time and for a given purpose, you can encourage a particular business or government organization to change in some prosocial way.

Some friendly editor sent me a pointer to the third issue of Limn a few days ago and it looks interesting. The theme is apparently “crowds and clouds” (with a heaping spoonful of the occupy movement stirred in for good measure) and it features the work of some wonderful and brilliant people. Also, who doesn’t want to read a well-designed, cc-licensed publication that indexes its issues at zero?

While on a brief excursion to the peninsula this past week, I learned about The San Francisco Public Press, an independent, non-commercial web and print publication featuring in-depth reporting about the Bay Area. The website not only features original reporting, but also a feed of curated local news stories.

The photo above was taken in my Oakland neighborhood by Eric Fischer (link to his Flickr photostream). Browsing Eric’s sets of photos, I found several fantastic sets of images related to maps, public transit, and bicycles in the Bay Area. My personal favorite was this gigantic colorful map of places to go on public transit in the East Bay printed in 1965 by the Alameda County Transit authority.

The Future of Crowdwork: CrowdCamp Workshop at CHI 2012

Think big! What would it take to make crowdsourcing and crowdwork a more sustainable, fulfilling, and efficient sector of economic and social production? (photo by John McNabb, cc-by-nc-nd)

This weekend, Andrés and I attended the CrowdCamp Workshop at CHI in Austin, Texas. The workshop was structured a lot like a hackathon, with the objective being to work in teams to produce projects, papers, or research.

The group I worked with coalesced around a proposal made by Niki Kittur, who suggested that we envision how crowdsourcing and distributed work contribute to solving grand challenges, such as economic inequality and the ongoing impact of the 2008 financial crisis.

We then spent the better part of the weekend outlining an ambitious set of scenarios and goals for the future of crowdwork.

While many moments of our conversation were energizing, the most compelling aspects derived from the group’s shared desire to imagine crowdwork and distributed online collaboration as potentially something more than the specter of alienated, de-humanized piece-work that it is frequently depicted to be.

To spur our efforts, we used a provocative thought experiment: what it would take for crowdwork to facilitate fulfilling, creative, and sustainable livelihoods for us or our (hypothetical or real) children?

Despite the limits of this framing, I think it opened up a discussion that goes beyond the established positions in debates about the ethics and efficiencies of paid crowdsourcing, distributed work, and voluntary labor online (all of which are, to some extent, encompassed under the concept of crowdwork in this case). It also hellped us start imagining howwe, as designers and researchers of crowdwork platforms and experiences, would go about constructing an ambitious research agenda on the scale of a massive project like the Hadron Collider.

If everything goes according to plan, this effort will result in at least a paper within the coming few weeks. Assuming that’s the case, our group will be sharing more details about the workshop and our vision of the future of crowdwork soon.

Five things I learned this week (future tense edition)

Kolob Reservoir, Utah (August, 2012).

For this edition of my occasional “five things” series, I’m trying out a twist on the usual theme (ideas, places, people, or things that I’ve run across in the preceding week) by discussing five things I’ll learn about next week.  So, without further ado, here are five things I am excited to encounter in the coming days…

1. CHI and CrowdCamp – I’m headed to Austin, Texas at the end of the week to present at CHI and participate in the CrowdCamp workshop. The lineup and agenda for CrowdCamp look incredibly exciting – the plan is to rapidly brainstorm, design, and (if possible) implement crowdsourcing projects. Given the past accomplishments of many of the other people who will be in the room, I’m excited!
2. New Zion Missionary Church (no website) – As part of my Austin trip, I hope to make a pilgrimage or two to as many of the regional holy sites of barbecue as I possibly can. In the case of New Zion Missionary Baptist Church (link points to a 2010 review on the Full Custom Gospel BBQ blog), I have heard that the slow smoked brisket can sometimes resemble a religious experience.
3. May Day Occupy actions in New York – Tuesday marks the first of May and, so it seems, a day of rebirth for the Occupy Movement. A few friends will be attending the New York actions and I’ll try to remember to link to anything they write or photograph.
4. The Onyx Boox M92 – Perhaps as a result of the extra attention that went to Mako’s setup a couple of weeks ago, I’ve succumbed and ordered my own e-book reader. I chose the Onyx Boox M92 because it checked all the boxes that mattered to me (linux based, large E-ink screen,  file format agnostic, vendor agnostic, and not reinforcing the Amazon empire) and because it seems to compare well against similar devices.
5. Calibre – Mako and Alan Toner kindly introduced me to Calibre – a very widely adopted and popular piece of free software to manage e-reader libraries -  this afternoon, but I won’t really start playing with it until my reader arrives next week.

Crowd labor relations revisited

I recently had a pilot version of a crowdsourcing task fail pretty spectactularly, but after discussing the failure with Mako I’ve concluded that my experience helps illustrate some interesting comparisons between labor relations in a distributed online market and more traditional sorts of employment and jobs.

The failure in this case started early: I did a mediocre job designing the task. It’s not really worth going into any details except to say that (out of laziness) I made it really easy for workers to either (a) purposefully respond with spammy results; (b) slack off and not provide responses; (c) try to complete the task but unintentionally do a bad job and therefore provide poor quality results; or (d) try to complete the task and do so successfully. I also did not do a good job incorporating any effective means of differentiating between whether the workers who did not provide accurate results were spamming, shirking, or simply failing

So why does this experience have anything to do with the nature of employment relations?

First, think about it from the employer’s (or the work “requester’s”) point of view. A major part of creating an effective crowdsourcing job consists in minimizing the likelihood or impact of (a)-(c) either by means of algorithmic estimation and/or clever task design. It’s not necessary that every worker provide you with perfect results or even perfect effort, but ideally you find some way to identify and/or remove work and workers that introduce unpredictable sources of bias into your results. Once you know what kind of results you’ve got, it’s possible to make appropriate corrections in the event that some worker has been feeding you terrible data or maybe just unintentionally sabotaging your task by doing a bad job.

In other words, low quality results can provide employer-requesters with useful information if (and only if) the employer-requester finds a way to identify it and use it to their advantage. This means that a poorly designed task is not just one that doesn’t elicit optimal performance from workers, but also one that doesn’t help an employer-requester differentiate between spammers, slackers, passive saboteurs, and those workers who really are trying and (at least most of the time) completing a given task successfully.

When I design a job I always assume that a relatively high proportion of the workers are trying to complete the task in good faith (sure, there are some spammers and slackers out there, but somehow they don’t seem to make up the majority of the labor pool when there’s a clear, well-designed, reasonably compensated task to be done). As a result, if I get predominantly crap responses back from the workers, I assume that they are (maybe somewhat less directly than I might like) providing me with negative feedback on my task design.

Now from the workers’ point of view, I suspect the situation looks a bit different. They have fewer options for dealing with employer-requesters who are trying to scam them. Most distributed labor markets lack features that would support anything resembling collective bargaining or collective action on the part of workers. Communications by workers to employer-requesters are limited and, consequently, there usually aren’t robust mechanisms for offering or coordinating feedback or complaints.

As a result, the most effective communications tool the workers possess is their work itself. Not surprisingly, some of them seem to use their work to engage acts of casual slacking and sabotage that resemble online versions of the “weapons of the weak” described by James C. Scott in his book on everyday resistance tactics among rural peasants.

The ease with which crowdsourcing workers can pursue these relatively passive forms of resistance and tacit feedback relates to a broader, more theoretically important point: in most situations, a member of an online crowd should have a much easier time quitting or resisting than workers in (for example) a factory when they decide they’re unhappy with an employment relationship for any reason. Why?  First off, crowdsourcing workers usually don’t have personal ties to a company, brand, co-workers, managers, etc. Second of all, the structure of online labor markets makes the cost of leaving any one job extraordinarily low. An office worker who (upon being confronted by, e.g., an unpleasant or unethical task) leaves her position risks giving up not only valuable resources like future wages or benefits, but also loses physical stability in her life, contact with friends and colleagues, and the respect or professional support of her superiors. In contrast, a worker in an online crowd who decides to leave her job loses almost nothing. While there is some risk associated with actively spamming or slacking (in some crowdsourcing markets, workers with low quality ratings can be banned or prevented from working on certain jobs), it’s still substantially easier to just walk away and find another task to do.

These are just some of the reasons why theoretical predictions from classical wage and employment economics – for example, that a $0.01 decrease in wages will result in some proportion of employees leaving their jobs – don’t hold up in traditional or crowdsourcing labor markets. The interesting point is that the reasons why these classical theories don’t hold up in crowdsourcing systems don’t have much to do with the complications introduced by social relations since social relations (between workers and employers as well as between workers and workers) are severely constrained in most online labor markets.

 

(Note: The first version of this post was written pretty late at night, so I didn’t include many links to sources. I’ll be trying to add them over the next few days.)

Peer review without the peers?

Academic peer review tends to be slow, imprecise, labor-intensive, and opaque.

A number of intriguing reform proposals and alternative models exist and hopefully, some of these ideas will lead to improvements. However, whether they do or not, I suspect that some form of peer review will continue to exist (at least for the duration of my career) and that many reviewers (myself included) will continue to find the process of doing the reviews to be time-consuming and something of a hassle.

The most radical solution is to shred the whole process T-Rex style.

Gideon Burton, 2009, cc-by-sa

This is sort of what has already happened in disciplines that use arXiv or similar open repositories where working papers can be posted and made available for immediate critique and citation. Such systems have their pros and cons too, but if nothing else they decrease the amount of time, money and labor that go into reviewing for journals and conferences, while increasing the transparency. As a result, they provide at least a useful complement to existing systems.

Over a conversation at CrowdConf in November, some colleagues and I came up with a related, but slightly less radical proposal: maybe you could keep some form of academic peer review, but do it without the academic peers?

Such a proposition calls into question one of the core assumptions underlying the whole process – that reviewers’ years of training and experience (and credentials!) have endowed them with special powers to distinguish intellectual wheat from chaff.

Presumably, nobody would claim that the experts make the right judgment 100% of the time, but everybody who believes in peer review agrees (at least implicitly) that they probably do better than non-experts would (at least most of the time).

And yet, I can’t think of anybody who’s ever tested this assumption in a direct way. Indeed, in all the proposals for reform I’ve ever heard, “the peers” have remained the one untouchable, un-removable piece of the equation.

That’s what got us thinking: what if you could reproduce academic peer review without any expertise, experience or credentials? What if all it took were a reasonably well-designed system for aggregating and parsing evaluations from non-experts?

The way to test the idea would be to try to replicate the outcomes of some existing review process using non-expert reviewers. In an ideal world, you would take a set of papers that had been submitted for review and had received a range of scores along some continuous scale (say, 1 to a protocol to distribute the review process across a pool of 5 – like papers reviewed for ACM Conferences). Then you would develop non-expert reviewers (say, using CrowdFlower or some similar crowdsourcing platform). Once you had review scores from the non-experts, you could aggregate them in some way and/or compare them directly against the ratings from the experts.

2007 diylibrarian cc-by-nc-sa

Would it work? That depends on what you would consider success. I’m not totally confident that distributed peer review would improve existing systems in terms of precision (selecting better papers), but it might not make the precision of existing peer review systems any worse and could potentially increase the speed. If it worked at all along any of these dimensions, implementing it would definitely reduce the burden on reviewers. In my mind, that possibility – together with the fact that it would be interesting to compare the judgments of us professional experts against a bunch of amateurs – more than justifies the experiment.