tag:blogger.com,1999:blog-188032182024-03-07T12:21:36.491-08:00Turk LurkerWelcome to Turk Lurker. Here you will find my thoughts and observations on the recently unveiled Mechanical Turk project from Amazon. I've long been a fan of distributed work projects and this blog will detail my experiences on this exciting new opportunity brought to us from Amazon. Come join the Turker Nation. You too will be assimilated.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.comBlogger49125tag:blogger.com,1999:blog-18803218.post-75363115227188682252013-07-11T13:09:00.001-07:002013-07-11T13:09:21.165-07:00Mechanical Turk a Sweatshop?Xeni Jardin at Boing Boing has dredged up this tired old headline once again, asking "<a href="http://boingboing.net/2013/07/05/is-amazons-mechanical-turk-a.html">Is Mechanical Turk a Sweatshop</a>?".<br />
<br />
The <a href="http://priceonomics.com/who-makes-below-minimum-wage-in-the-mechanical/">article</a> referenced adds nothing new either. Yes, you may only get 3 cents for doing a "job", but it's not like Amazon is holding anything over your head and making you work in poor environment. Just the fact that you even have the ability to spend 5 seconds on something and actually make any money at all is pretty much amazing. Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com1tag:blogger.com,1999:blog-18803218.post-36001094635563770262011-03-16T13:01:00.000-07:002011-03-16T13:04:16.705-07:00My New ProjectI've pretty much given up on the Mechanical Turk these days. I am working on a 4x Browser Space Game called <a href="http://projectlibertine.blogspot.com">Project Libertine</a>. It's design is based on some of the old Play by Mail games I used to play back in the 1980's and 1990's.<br /><br />It's a long way from being ready, but I'm updating the development blog fairly regularly. I've actually got some plans about using Mechanical Turk to automate some of the game administration once it goes live. Like approving character and ship names and such.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com4tag:blogger.com,1999:blog-18803218.post-62650838661031964552007-07-02T20:57:00.002-07:002007-07-02T20:58:00.403-07:00July 3rd System UpgradeA new message was posted on the Mechanical Turk today from Amazon:<br /><br />We are upgrading our system on Tuesday, July 3rd, between the hours of 12:00am and 3:00am PDT. During these times, users will not be able to register for new Mechanical Turk accounts. Some users will be not be able to transfer funds or view account balances. Existing users can still submit HITs, but some payments will be delayed until the upgrade has finished. Statistics may also be inconsistent during this maintenance period, but will be correct after the system upgrade.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com1tag:blogger.com,1999:blog-18803218.post-15525368138767487812007-07-02T20:57:00.001-07:002007-07-02T20:57:19.281-07:00Back to TurkingThere's a book I want to order from Amazon, so I've been doing some Turking again the last few days. It's still not as fun as the good old days right after it launched, but I've managed to make a few buck while laying on the couch late at night. I did about 50 short transcription hits for Videojug.com over the weekend, but so far they haven't been approved. The "Are these watches different?" hits from Amazon remind me of the good old days somewhat, since they're quick to do, but they seem to trickle them out and they're quickly gone. I probably won't have the patience to get ripped off for my work long enough to get the entire amount I need for the book, but it was fun to spend a few hours back in the slave yard.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com3tag:blogger.com,1999:blog-18803218.post-15179217047394962222007-01-18T11:09:00.000-08:002007-01-18T11:16:45.859-08:00eMail from AmazonI haven't gotten any eMail from Amazon about The Mechanical Turk for months, but last week I got one telling me about new hits available. The Geospatial Vision hits look very interesting, but also somewhat time consuming, especially for $0.08 a hit. I'm sure my little buddy Eric will be all over them, but he's desperately trying to avoid getting a real job anyway. I checked too late to see what the Enrovia Research hits were like. They must have been pretty good since there don't appear to be any available today.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1168290028665489672007-01-08T12:29:00.000-08:002007-01-08T13:00:28.700-08:00MTurk Mentioned on MSN.comThe Mechanical Turk was mentioned in <a href="http://articles.moneycentral.msn.com/CollegeAndFamily/RaiseKids/4realJobsYouCanDoFromHome.aspx">an article</a> on MSN.com by Liz Pulliam Weston that talked about jobs you can do from home. As usual, they made MTurk sound like a great idea, but then pointed out the truth that most jobs there just don't pay very well. <br /><br />Personally, I've been scanning the available hits every few weeks myself and there's just doesn't seem like there's much worth doing out there right now. Still no "killer apps," which has plagued the project since the beginning.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com2tag:blogger.com,1999:blog-18803218.post-1163527314452853072006-11-14T09:57:00.000-08:002006-11-14T10:01:54.476-08:00A Great Charity For TurkersOver at the <a href="http://turkers.proboards80.com">Turker Nation</a> messageboards, Turker Iteki mentioned a great charity that is set up in a way that makes it extremely easy for Turkers to contribute to.<br /><br /><a href="http://www.childsplaycharity.org/">Child's Play</a> is a charity set up by Internet gamers that allows you to browse a wish list on Amazon, then select an item from the list and purchase it through Amazon, who will send it on to the hospital that requested the item. So Turkers can spend some of their hard earned credit to help out sick kids all across the country.<br /><br />Even if you're not currently turking, please consider donating to this very worthy cause.<br /><br /><br /><div align="center"><a href="http://www.childsplaycharity.org/"><span style="font-size:130%;">http://www.childsplaycharity.org/</span></a></div>Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1150128245722996722006-06-12T08:59:00.000-07:002006-06-12T09:04:05.753-07:00QuotedI was interviewed recently by <a href="http://www.wired.com">Wired Magazine</a> about my previous experiences with the Mechanical Turk. A 30 minute phone interview turned into a two-sentence quote in the June 2006 issue. If you'd like to read the article, follow <a href="http://www.wired.com/wired/archive/14.06/crowds.html">this link</a>. My quote is on page four. I don't remember using the word "gimpy", but maybe I did. What I was trying to say was that MTurk seems to be in a pause right now, with little in the way of interesting work, and it's just waiting for someone to come up with a good way to make use of it's full potential.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com2tag:blogger.com,1999:blog-18803218.post-1146173033878692402006-04-27T14:16:00.000-07:002006-04-27T14:23:53.896-07:00Apparel Color NormalizationI haven't been very active with the Mechanical Turk recently, but today I saw a type of hit I haven't seen before. They're called Apparel Color Normalization and it involves the turker looking at a very small swatch of color on the screen and choosing the closest matching color from a list with adjacent radio buttons.<br /><br />The hits are from Amazon Requestor and are paying two cents each. I did five just to check them out and they can be completed fairly quickly. I did notice that several didn't actually have images to judge, so I skipped those. I suspect these are just to provide more traffic on the MTurk website since they have claimed that the Mechanical Turk will come out of Beta soon.<br /><br />One other minor curiousity I noticed is that they used the British spelling for the color "grey" instead of the normal American "gray." I have no idea why I'm mentioning this, but it did catch my eye while I was working on the hits.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com1tag:blogger.com,1999:blog-18803218.post-1145908099571446952006-04-24T12:38:00.000-07:002006-04-24T12:48:19.596-07:00MTurk Market ForcesI realize blogs are not expected to be fact checked. I've made some assumptions that weren't true on this and my other blogs I'm sure. I just wanted to point out a nitpick in an otherwise excellent <a href="http://www.oreillynet.com/mac/blog/2006/04/earning_the_big_bucks_not_with.html">blog entry</a> by <a href="http://www.oreillynet.com/pub/au/1796">Erica Sadun</a> about the Mechanical Turk.<br /><br />The blog entry states "Early tasks that started at 75 cents, soon dropped to 65, and from 65 to 60, and from 60 to 40, all the way down to the current rate of about 1-3 cents per task." This statement is somewhat misleading I feel, since the earliest hits did pay 75 cents, but required a good deal of work on the part of the Turker doing the work. These hits involved doing product write-ups, usually on auto parts. The price paid on these dropped gradually, but not all the way down to 3 cents. The hits that paid 3 cents or less were the Image Adjustment hits or the Artist Title hits, which took considerably less time.<br /><br />I do agree that market forces are in play on the Mechanical Turk. The Casting Words hits are a perfect example of this. It's my opinion though that market forces were not having much effect on MTurk in the heyday soon after it's SlashDotting. I think Amazon was simply setting prices on a whim to see what they could get away with.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com1tag:blogger.com,1999:blog-18803218.post-1144781599174445852006-04-11T11:52:00.000-07:002006-04-11T11:53:19.203-07:00Modular SoftwareA <a href="http://www.iht.com/articles/2006/04/05/business/software.php">recent article</a> mentions MTurk and Casting Words, one of the more successful uses of MTurk seen so far.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1137002139969689992006-01-11T09:47:00.000-08:002006-01-11T09:55:39.980-08:00Slow TurkingOur new baby boy was born two weeks ago today, and along with my father passing away two weeks before that I haven't been involved in the Mechanical Turk scene much lately. There doesn't appear to be too much of interest going on right now, although there have been a few new types of hits to show up.<br /><br />I read recently that the scientists at the University of California, Berkley who are in charge of the Stardust space probe are going to <a href="http://www.cnn.com/2006/TECH/space/01/10/stardust.search.ap/">enlist the help of the public</a> in a distributed project to scan photographs to identify which sections of their capture area contain dust particles. This would have been a great use of the Mturk system, but they have no need to pay since I'm sure they'll get enough volunteers to easily finish off the 30,000 man hours required to go through the dataset four times.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1135358834048736702005-12-23T09:25:00.000-08:002005-12-23T09:27:14.056-08:00Casting Words Style GuideRachel at Castingwords.com has started a <a href="http://castingwords.jot.com/Turkers+Transcription+Style+Guide">Style Guide</a> to help people better understand how to complete these types of hits on <a href="http://mturk.com">Mturk</a>.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1135184048182519962005-12-21T08:49:00.000-08:002005-12-21T08:54:08.183-08:00Casting Words ForumRachel at <a href="http://castingwords.com">Casting Words</a> has agreed to moderate a forum at <a href="http://turkers.proboards80.com">Turker Nation</a> where Turkers can easily communicate with with her and other Turkers interested in pod casting transcription work. Casting Words was one of the early requestors outside of Amazon to post hits on MTurk and it's been interesting to follow their progress so far.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1135183749406777862005-12-21T08:37:00.000-08:002005-12-21T08:49:09.416-08:00Low User ActivityI was unable to attend last Thursday night's training session hosted by Jeff Barr due to my father passing away that night. Hopefully there will be a transcript available.<br /><br />From what I can tell, things are slow currently on <a href="http://mturk.com">Mturk</a>. A few new types of hits have shown up, but honestly I find the Top Three hits to be completely uninteresting. If MTurk moves towards becoming just a new way of doing survey work I'd be greatly disappointed.<br /><br />The new types of hits also lead me to make a suggestion to the Mturk team. It would be a great thing to be able to sort the available hits just by clicking on column headers instead of having to choose from a dropdown list and clicking a button. I realize this would involve changing the layout of how hits are presented to users, but now is a great time to do so while MTurk is still in Beta status. It would also be very desirable to have an easy way to exclude hits from the list, so I wouldn't have to scroll through a lot of Top Three hits to see if there's anything else more interesting to work on.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com1tag:blogger.com,1999:blog-18803218.post-1134528839303736082005-12-13T18:07:00.000-08:002005-12-13T18:53:59.316-08:00Amazon Training SessionI participated in the online presentation by Jeff Barr this evening. I found it interesting although the initial presentation was geared more towards management and only offered a high level overview of the Mechanical Turk service.<br /><br />Of much more interest was the question and answer session at the end. I'm not sure who besides Jeff Barr and John Hsia were answering questions since they weren't introduced, but it was said they were development team members. I had a few questions prepared before I signed on, and I asked a few more that I thought of later. <br /><br />1. Alan asks: Approximately how many individual turkers are regularly submitting hits on a daily basis?<br /><br />Answer: That is currently confidential information.<br /><br />2. Alan asks: Has the MTurk team considered any methods of verifying worker identity and vetting worker backgrounds in order to allow them to view more sensitive data?<br /><br />Answer: Yes we have, nothing in that regard has been implemented yet. You can add your own system using qualifications by only granting people quals who have passed a particular test.<br /><br />--What I was really getting at here was whether Amazon had any interest in providing a pool of workers who could be trusted to view private or confidential information. From what I gathered tonight I think they're more interested in providing the interface and letting Requestors work out this sort of thing for themselves. There is apparently nothing preventing you from only allowing exactly who you want to have access to the Hits that you provide to the MTurk system.<br /><br />3. Alan: I can imagine ways to use MTurk where you might only pay for a hit if a worker happens to find something. This would lead to a high rejection rate that might affect their ability to do other hits. Any thoughts on how to handle that?<br /><br />Answer: We are considering exposing info about the requesters such as their approval and rejection rates for different types of hits. This would give workers insight into if they want to work on those particular types of HITS or not.<br /><br />--Again I didn't really explain what I was getting at very clearly. I might want to have workers to look through a set of scanned receipts and flag any that have problems. If I only wanted to pay when they found an erroneous receipt, I'd have to reject those that don't. It occured to me I could account for this by adjusting the rate I paid for each submitted hit, but it might be useful to have a way to only selectively pay for a hit without just rejecting them.<br /><br />4. Alan: How long do you anticipate MTurk staying in Beta?<br /><br />Answer: Until we are ready to launch!<br /><br />-- A bit of a flippant answer, but it was an off-the-cuff question. What I should have asked is if anything will change once MTurk leaves Beta or if there's any reason to wait until it leaves Beta status to submit requests.<br /><br />5. Alan: Will work always have to be done through MTurks website?<br /><br />Answer: Today we don't have a set of APIs, but we are actively looking at ways to incorporate MT into the worker side of your applications. We would love to hear of your requirements in that space.<br /><br />-- Someone else asked basically the same question, but it didn't make it to the chat window. The answer was to their question. It sounds like they're open to other interfaces being developed, but don't have the tools available yet.<br /><br />6. Rob: I've noticed you've added CAPTCHA support to the site, is this feature going to be automatically applied to HITs requestors submit?<br /><br />Answer: Yes, it is applied for all new users after the first 5 HITS and then progressively thereafter, for all hits in the system, not just Amazon ones.<br /><br />-- The image captcha's are here to stay, which is a great thing and a good first step in causing the script kiddies as much grief as possible.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com2tag:blogger.com,1999:blog-18803218.post-1134146983821789452005-12-09T08:43:00.000-08:002005-12-09T08:49:43.833-08:00Still Hittin' ItI've still been doing about 200 or so hits a day, just to keep up with things and add a little extra money to my Amazon account. At that rate I'd make about $100 a month, which is enough to at least make me smile.<br /><br />One thing that doesn't make me smile is still having to fight for hits. Clicking accept 20 times in a row and not getting a hit to work on is the only thing that still just makes me go red when I'm on the MTurk site. The Monolith is still silent about it of course. Meh.<br /><br />I'm signed up for an online training session/overview this Tuesday evening with host Jeff Barr. It should be interesting and I hope to get a few questions in and I promise not to heckle Jeff too much. I'm pretty sure I'm bigger than him, but he may work out more. Heh, heh, heh.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com1tag:blogger.com,1999:blog-18803218.post-1133812310097430082005-12-05T11:19:00.000-08:002005-12-05T11:51:50.116-08:005k HitsI recently submitted my 5000th hit since I started participating in the MTurk project about a month ago. In that month I made a little over $131.00. Not too bad for piddling around in my spare time I suppose.<br /><br />I've also recently broken down and done a few more Artist Confirmation hits. I still don't really like them, but I have a stubborn, pitiful technique of doing them so if I make three bucks a day doing them I'm happy. I basically skip the hits until I see something like "Pat Boone" or "The Temptations" or something that is obviously "Various Artists." Combining that with have to hit Accept about 10 times before I actually get a hit makes for very slow progress.<br /><br />I think Amazon's stubborn refusal to do anything about the hits so easily being given to someone else before you have a chance to accept them is soon going to turn me off of the entire project. It's too frustrating. Apparently hits are just shown to as many people as possible until someone clicks accept. It's like they've put you in one of those Vegas "money chambers" where the dollar bills are blowing all around you and you've got so much time to grab as many as you can.<br /><br />In the past few days people have also been posting on the message boards that they always choose "None of the Others" when doing IA hits. I personally find this distasteful, but it's hard to not expect it. It's the best way to make the highest amount of money in the shortest amount of time so until they implement some method to discourage it, IA hits are probably dead.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1133398345615353112005-11-30T16:42:00.000-08:002005-11-30T16:52:25.636-08:00I've Been Hit HikingSo I guess I spoke too soon about the Podcast Transcription hits. I saw a few today and they were back down to four cents apiece. I broke down and did about twenty artist confirmation hits, but none have been processed as of yet. I don't think I'll do many more since I just honestly don't like doing them.<br /><br />There's been rumors of a new Product Description hit that pays upwards of seventy-five cents, but I haven't seen one on the hit list to confirm it yet. As soon as I see one listed I'll try it out.<br /><br />I managed to snag about 40 IA hits earlier this evening, but it was a frustrating process. I guess I'm going to have to admit I'm a stubborn old goat and try out some of the auto-accept scripts. I really hope the MTurk guys change the process soon so that you get at least a few seconds to accept a hit before someone else can. It would also be great to have a Submit and Assign button that just accepts the next hit in the group, but there seems to be some indication from the Monolith that they want us to be more selective in what hits we accept. Check out <a href="http://turkers.proboards80.com/index.cgi?board=general&action=display&thread=1133393446">this thread </a>on Turker Nation for more information on that.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1133373765803913682005-11-30T09:56:00.000-08:002005-11-30T10:02:45.803-08:00More Turk LinksI've been slow to update the links to other MTurk web sites, and for that I apoligize. Life is getting more hectic and is only going to get worse I suspect when the baby gets here.<br /><br /><a href="http://turking.com">Turking.com</a> is a great place for more information and they have extensive message boards, although you do have to register to view them.<br /><br />A <a href="http://hkturker.wordpress.com/">Hong Kong Mechanical Turk Addict</a> is another personal blog that you can check out as well.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1133301183953753862005-11-29T12:57:00.000-08:002005-11-29T14:42:53.053-08:00A Matter of TrustThere's an old saying that "Trust must be earned." As it stands right now on Amazon's Mechanical Turk, everyone starts out being trusted. That's very noble, but it causes problems when the script kiddie barbarians show up at the gate and they get waved in with the rest of us.<br /><br />My experience as a parent and what I learned when studying for my education degree tells me that trust must be given to a child as they show more responsibility. If they stay within the box that defines the rules of the adult/child relationship, the box expands and they are given more freedom. If they step outside that box, it shrinks down to a more restrictive level until the child earns the trust to expand it again.<br /><br />My computer science degree and my experience in the IT field allows me to define a way to reflect this in a system to govern hit acceptance in the MTurk environment. As I understand it, the method of accepting or rejecting hits is left to the client, so this is my thoughts on how to do that in a way that rewards success and limits the damage that auto-accept and submit scripts can do. I'm going to use the A9 Image Adjustment (IA) hits for this, but the method should work with any of the hit types seen so far on MTurk.<br /><br />If we extrapolate my example of the adult/child relationship, then we have Amazon as the adult and the workers as the child. I'm not implying of course that the Turk Wurkers are children, but the relationship is similar because the adult/A9 has no reason to trust the child/worker when the relationship begins. If a method is put into place that allows trust by A9 to be measured and modified for each worker, I think many of the current problems with IA hits could be eliminated.<br /><br />To begin, a variable has to be assigned to each Turker that measures Trust. This would be a simple integer value and would begin at a low number of say five (5) for new workers. A value of zero would imply that there is a neutral level of Trust between A9 and the worker. The value would increase as trust is gained and decrease as trust is removed. Any negative value would imply lack of trust and will be discussed later.<br /><br />In order to determine whether you trust a new worker, you have to have a basis to judge them against. This requires a seed of absolute trust. For each group of images that you plan to test in the MTurk system, you have internal workers or admins select a small percentage of them and choose the definite correct answer to establish them as Trust Markers (TM). These sets of images can be placed semi-randomly in the work flow. Each correct answer will raise the Trust level of a worker by one point. The higher the Trust value is for a given worker, the less often these Trust Markers need to show up for that worker. In this way A9 would have to spend less on these types of images as more trust is gained. This is analogous to a company spending more on a worker when they are first hired in order to train them.<br /><br />An incorrect answer to a TM would result in one point being subtracted from that workers Trust value. Once Trust goes negative, the worker would no longer be allowed to accept any hits until Trust reaches zero again. Negative points could decay at a rate set by A9, so it could take an hour or a day before the new user could try again. The value could also be allowed to increment back to the starting point of five after a certain amount of time. This would allow for a bit more leniency.<br /><br />Just the addition of this functionality would severely hamper how much damage a scripter could do to the results, but one more feature is needed to eliminate the need to pay them for the random hits they did get correct before being locked out. I call this feature a Trust Lock.<br /><br />A Trust Lock is created by taking a standard set of A9 images, including the "None of the Others" image and changing one of the images so that it reads "Submit This Image" in the same style as the NotO image. The worker would obviously be required to submit that particular image to answer the hit correctly. The Trust Lock would be dropped in much less frequently than the TMs, but answering the Trust Lock image set incorrectly would lead to an immediate Trust value of negative one (-1). This sounds harsh, but only a script or someone not paying attention would miss one of these.<br /><br />In addition, all hits submitted since the last time you correctly answered a Trust Lock hit would be automatically rejected, whether they were answered correctly or not. Again, this sounds harsh but there's no reason to ever answer one incorrectly unless you're running an auto-accept script or working at a pace that is too high.<br /><br />So let's use a few real world examples to walk through the methodology I just covered. First, let's say Johnny Turker heard about MTurk from his roommate and logs in and creates an account. He's assigned an initial Trust value of five (Trust = 5). Johnny then goes off and selects a group of IA hits to work on and starts turking.<br /><br />At some point within the first 20 hits, Johnny is unknowingly presented a Trust Marker hit, and being inexperienced, gets it wrong. This drops his Trust value to four. At this point the MTurk system may decide to assign the next TM within 10 to 15 hits, since the trust level is less. If he had answered the first TM correctly, the MTurk system may wait until 20 to 25 more hits, depending on the algorithm used to determine how often these hits are presented to the worker. The system could also decide to immediately drop in a Trust Lock hit since missing the first TM that was presented could raise suspicions of him using a script.<br /><br />Regardless, within the first 50 hits Johnny is presented with his first Trust Lock hit and he answers it correctly. At that point all the hits he submitted before the Trust Lock are eligible to be processed, while all the hits after this point will be processed when the next Trust Lock is answered correctly. <br /><br />After doing about 100 hits, Johnny calls it a night and logs out with a Trust value of say six (6) since he improved how often he answered the TMs correctly. Later that night, Johnny falls under the influence of an evil script kiddy buddy down the hall in his dorm, who tells him he has a script that will randomly answer IA hits and make him lots of money while he sleeps.<br /><br />Johnny installs the script and it starts running. Within 20 hits or so it encounters it first TM hit. It has a 1 in 7 chance of answering this correctly, which is quite possible but somewhat unlikely. Since Johnny's Trust value is still relatively low, he will also be presented a Trust Lock hit soon as well. Between the two types of Trust hits, it is unlikely the script will run for very many hits before Johnny is locked out of accepting any more hits. This also makes it unlikely that A9 will have to pay him for the submitted hits since they can know with some confidence that they're likely junk submittals.<br /><br />The Trust value also allows Amazon and A9 to remove more of these restrictions after Trust reaches a certain level. The restriction on hits being processed until a Trust Lock is passed could be removed after Trust reaches a value of 25 or whatever level is determined to be appropriate. The Trust level could also be used to allow access to other new types of hits that pay better or that are more sensitive to script manipulation.<br /><br />A possible algorithm for how often a turker is presented with TMs could be TL(20) - random(1...TL(10)). So at a Trust Level of 10 a turker would see a TM within the next 100 to 199 hits, which is 200 minus a random number between 1 and 100. Trust Locks could occur more randomly, but should one probably be presented to the turker within a few hits of a TM being answered incorrectly.<br /><br />The existing qualifications can be combined with the Trust level to create new seeds of TM hits. If enough people with a certain level of Trust and a high level of accuracy agree on a certain image, it could be turned into a new TM hit. The Trust value could also be used to reduce the number of workers a hit has to be presented to to verify it. The higher the Trust value of a turker, the more weight is given to their response, so one Turker with a Trust value of 100 and an accuracy of 90% could replace multiple submitals by turkers with lower values. <br /><br />This all leads me to a method where A9 could get more value out of their IA hits, but I'll leave that for the next gigantipost. Please feel free to comment on anything I missed or ways this method could be abused. I'll edit this post with any changes we come up with.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com3tag:blogger.com,1999:blog-18803218.post-1133297847394548222005-11-29T12:55:00.000-08:002005-11-29T12:57:27.406-08:00Transcription Hits Up to a NickelI saw a few of the Transcription Hits today at lunch. They quickly disappeared, but I did notice they were paying five cents each instead of two cents. That's a great improvement, but IA hits at three cents each are a much better pay out. Of course, I haven't been able to actually receive an IA hit for the last two days, so the difference is a moot point.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1133218622266110672005-11-28T14:52:00.000-08:002005-11-28T14:57:02.283-08:00Turk PropsThere was a <a href="http://aws.typepad.com/aws/2005/11/amazons_mechani.html">recent post</a> on the Amazon Web Services Blog recently that I missed during my holiday down time. Thanks to them for mentioning this blog and the <a href="http://turkers.proboards80.com/index.cgi">Turker Nation</a> message board. There wasn't a lot of information revealed in the blog post, but they seem to indicate that they're actively seeking new types of hits to enter into the system from outside customers. Personally, I think they're going to have to do a lot more testing on ideas to discourage script kiddies and other dillholes before they start trying to get paying customers.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0tag:blogger.com,1999:blog-18803218.post-1133125919004240552005-11-27T13:01:00.000-08:002005-11-27T13:11:59.033-08:00BreaksI've taken a break since Tuesday from work and MTurk for the most part. I wanted to do a lot of hits last night but was only able to do about 200 since the Image Adjustment hits available were running low and I don't like doing the Confirm Artist Name hits.<br /><br />There are rumors on the Turker Nation boards that people are running scripts just to accept hits and randomly click an image and submit them as rapidly as possible. There has been no word from the Monolith at Amazon of course, so there's no way to confirm or deny it. Greedy people running scripts with 25 tabs open at once on their browser could also account for some of it.<br /><br />Hopefully Amazon will seriously consider putting a hard limit of less than 10 hits that can be checked out at once. I'm sure this will slow some people down, but doing 1000 hits an hour is not really providing valuable work anyway. It's just taking advantage of the fact that the A9 images are skewed so heavily towards the "None of the Above." If "None of the Above" were only the correct option about 20% of the time, you'd have to spend more than 4 seconds on a hit or your accuracy would suffer greatly.<br /><br />I suspect that Amazon might not care about getting "real" work done with IA hits so much right now. I see the same IA hits over and over again so I think they're recycling a lot in order to maintain load on the system and get more data they can use to optimize things.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com2tag:blogger.com,1999:blog-18803218.post-1132813477321835522005-11-23T22:22:00.000-08:002005-11-23T22:24:37.346-08:00Transcription HitsA new type of MTurk Hit appeared recently. For these you listen to a small audio podcast and transcribe what you hear. While generating some interest because they are new, these hits will not be very popular if they continue to pay only two cents per hit. In the time it takes you to do one of these, you could finish multiple Image Adjustment hits.Wayward Hatchhttp://www.blogger.com/profile/03570663283489154363noreply@blogger.com0