Paper Reading #6: TurKit, human computation algorithms on mechanical turk

TurKit, human computation algorithms on mechanical turk


Greg Little, Lydia B. Chilton, Max Goldman, Robert C. Miller

Greg Little is a student and researcher at the CSAIL lab at MIT.
Lydia B. Chilton is a graduate student at the University of Washington who also interned at Microsoft Research.
Max Goldman is a graduate student and researcher at the CSAIL lab at MIT.
Robert C. Miller is an associate professor in the Computer Science department at MIT.

Summary


In this paper, the researchers presented a new interface called "TurKit" that interacted with an existing tool known as MTurk. These tools allow for the consideration of human computation in systems. Human computation is a great way to allow computations that a computer might now be able to do. For example it might be difficult for a programmed system to recognize what's in a photo. However, using TurKit, you can design systems that allow the humans to essentially fill in the blanks.

The TurKit researchers specifically added a new API extension to MTurk, the idea of "crash-and-rerun" programming and an online interface for TurKIt.

TurKit allows for instructions computed in a human computation to be saved for later. This also allows for synchronization across various human computation inputs. In the API extension they added the ability for various to be used across human computation tasks (also know as human intelligence tasks or HITs). The extension also adds some "fake" multithreading abilities.

They researchers seemed to have multiple problems they hoped to address with TurKit. First, they wanted to give developers a better interface to use with MTurk. Second, by cutting down on costly operations and storing data, it's easier to save data for later or to save on computation costs.

A few ways they used to test TurKit were through iterative writing and blurry text recognition. Several outside groups also tested the use of TurKit in various ways such as psychophysics anaylsis.

Discussion


The usage of TurKit could be extremely beneficial in very many areas. For example, data labeling is a huge problem in many different environments. For example, searching for images can be difficult be the user might know what's in the picture but they can't search by tags normally. Adding in a labeling system would be beneficial to organization without a doubt.

Also mentioned in the article is the benefits of iterative writing. This is a system that would make Wikipedia much more accessible. Currently, editing Wikipedia is an extremely daunting task but if TurKit was used, articles could write on articles that need more information or fill in needed information for another article.

I think the researchers definitely fixed the problems they discussed and they presented a viable product that definitely has potential for many future products.