Initial Impressions

I joined Amazon's Mechanical Turk program on November 5, 2005 after reading about it on Slashdot. As could be expected, the Turk site was inundated at the time from being slashdotted, but I was able to get signed up and complete a single Image Adjustment Hit. I liked what I saw, so I gave it a few days and tried again on the night of the 8th.

So while I layed on the couch with my laptop and watched old episodes of Law and Order off the Tivo, I proceeded to submit 287 hits over the course of about three hours. I chose to work exclusively on Image Adjustment Hits. This type of Hit is the lowest paid, but can be completed the quickest. I could have completed more, but Amazon still seems to be having some performance issues which made it necesssary to go through a lot of extra steps in the process. Once this gets cleared up I think it might be possible to increase the number of Image Adjustment Hits returned to 500 or more in a three hour span of time.

There seemed to be a delay of approximately 10 minutes last night from the time I submitted a hit to when it was approved or rejected. As of this morning, 45 hits are still pending so I'm not sure what the delay is there. I suspect that a certain image is shown a predetermined number of times to different users and once that number is reached, the results are tallied and if the choice you made matches up to the majority it is accepted and paid. This is just a guess of course based on my observations, but it reminds me of some of the techniques used in standardized testing to correct for errors.

Of the remaining hits I returned last night, 34 were rejected and 208 were approved. This is a rejection rate of 14% of the hits that were processed. I don't know how good that rate compares to others, but at the speed I was doing them I suspect it is not too bad. I accepted any hit that showed up without looking at it first. I think it would be possible to get a better acceptance rate by only accepting the Image Adjustment hits that are obviously "None of the Above."

I was working on images mostly from Phoenix, Arizona last night and often there would be a series of images that pictured an empty lot or just the side of a warehouse. I could scan these in about 5 seconds and choose "None of the Above" and click submit. I found that if the series of images actually contained the business in question, the process slowed down and could take 30 seconds or more. I suspect these cases are also likely where most of the rejected hits originated from. If you have two images in the series that are very similar, there's a good chance that my choice might not agree with the majority of those submitting the hit.

I think another possible source of rejection on a hit could come from business names not matching what is on the front of the store. So if I'm looking for "Frank's Taco Stand" and the pictures contain a business with a big sign that says "Hot Tacos" I'd be inclined to say that was a match, but I'm not sure if MTurk would accept that.

At 3 cents a hit, I'm not going to make big amounts of money doing this, but since I spend an hour or two just vegging out on the couch most nights anyway, I might as well make a little extra cash while I'm doing it. So far I've earned about 7 bucks which is way less than minimum wage, but how many jobs let you make money by laying on the couch? My goal is to make enough to order a new caving helmet from Amazon so I need about 10 more nights.