Friday, December 23, 2005
Casting Words Style Guide
Rachel at Castingwords.com has started a Style Guide to help people better understand how to complete these types of hits on Mturk.
Wednesday, December 21, 2005
Casting Words Forum
Rachel at Casting Words has agreed to moderate a forum at Turker Nation where Turkers can easily communicate with with her and other Turkers interested in pod casting transcription work. Casting Words was one of the early requestors outside of Amazon to post hits on MTurk and it's been interesting to follow their progress so far.
Low User Activity
I was unable to attend last Thursday night's training session hosted by Jeff Barr due to my father passing away that night. Hopefully there will be a transcript available.
From what I can tell, things are slow currently on Mturk. A few new types of hits have shown up, but honestly I find the Top Three hits to be completely uninteresting. If MTurk moves towards becoming just a new way of doing survey work I'd be greatly disappointed.
The new types of hits also lead me to make a suggestion to the Mturk team. It would be a great thing to be able to sort the available hits just by clicking on column headers instead of having to choose from a dropdown list and clicking a button. I realize this would involve changing the layout of how hits are presented to users, but now is a great time to do so while MTurk is still in Beta status. It would also be very desirable to have an easy way to exclude hits from the list, so I wouldn't have to scroll through a lot of Top Three hits to see if there's anything else more interesting to work on.
From what I can tell, things are slow currently on Mturk. A few new types of hits have shown up, but honestly I find the Top Three hits to be completely uninteresting. If MTurk moves towards becoming just a new way of doing survey work I'd be greatly disappointed.
The new types of hits also lead me to make a suggestion to the Mturk team. It would be a great thing to be able to sort the available hits just by clicking on column headers instead of having to choose from a dropdown list and clicking a button. I realize this would involve changing the layout of how hits are presented to users, but now is a great time to do so while MTurk is still in Beta status. It would also be very desirable to have an easy way to exclude hits from the list, so I wouldn't have to scroll through a lot of Top Three hits to see if there's anything else more interesting to work on.
Tuesday, December 13, 2005
Amazon Training Session
I participated in the online presentation by Jeff Barr this evening. I found it interesting although the initial presentation was geared more towards management and only offered a high level overview of the Mechanical Turk service.
Of much more interest was the question and answer session at the end. I'm not sure who besides Jeff Barr and John Hsia were answering questions since they weren't introduced, but it was said they were development team members. I had a few questions prepared before I signed on, and I asked a few more that I thought of later.
1. Alan asks: Approximately how many individual turkers are regularly submitting hits on a daily basis?
Answer: That is currently confidential information.
2. Alan asks: Has the MTurk team considered any methods of verifying worker identity and vetting worker backgrounds in order to allow them to view more sensitive data?
Answer: Yes we have, nothing in that regard has been implemented yet. You can add your own system using qualifications by only granting people quals who have passed a particular test.
--What I was really getting at here was whether Amazon had any interest in providing a pool of workers who could be trusted to view private or confidential information. From what I gathered tonight I think they're more interested in providing the interface and letting Requestors work out this sort of thing for themselves. There is apparently nothing preventing you from only allowing exactly who you want to have access to the Hits that you provide to the MTurk system.
3. Alan: I can imagine ways to use MTurk where you might only pay for a hit if a worker happens to find something. This would lead to a high rejection rate that might affect their ability to do other hits. Any thoughts on how to handle that?
Answer: We are considering exposing info about the requesters such as their approval and rejection rates for different types of hits. This would give workers insight into if they want to work on those particular types of HITS or not.
--Again I didn't really explain what I was getting at very clearly. I might want to have workers to look through a set of scanned receipts and flag any that have problems. If I only wanted to pay when they found an erroneous receipt, I'd have to reject those that don't. It occured to me I could account for this by adjusting the rate I paid for each submitted hit, but it might be useful to have a way to only selectively pay for a hit without just rejecting them.
4. Alan: How long do you anticipate MTurk staying in Beta?
Answer: Until we are ready to launch!
-- A bit of a flippant answer, but it was an off-the-cuff question. What I should have asked is if anything will change once MTurk leaves Beta or if there's any reason to wait until it leaves Beta status to submit requests.
5. Alan: Will work always have to be done through MTurks website?
Answer: Today we don't have a set of APIs, but we are actively looking at ways to incorporate MT into the worker side of your applications. We would love to hear of your requirements in that space.
-- Someone else asked basically the same question, but it didn't make it to the chat window. The answer was to their question. It sounds like they're open to other interfaces being developed, but don't have the tools available yet.
6. Rob: I've noticed you've added CAPTCHA support to the site, is this feature going to be automatically applied to HITs requestors submit?
Answer: Yes, it is applied for all new users after the first 5 HITS and then progressively thereafter, for all hits in the system, not just Amazon ones.
-- The image captcha's are here to stay, which is a great thing and a good first step in causing the script kiddies as much grief as possible.
Of much more interest was the question and answer session at the end. I'm not sure who besides Jeff Barr and John Hsia were answering questions since they weren't introduced, but it was said they were development team members. I had a few questions prepared before I signed on, and I asked a few more that I thought of later.
1. Alan asks: Approximately how many individual turkers are regularly submitting hits on a daily basis?
Answer: That is currently confidential information.
2. Alan asks: Has the MTurk team considered any methods of verifying worker identity and vetting worker backgrounds in order to allow them to view more sensitive data?
Answer: Yes we have, nothing in that regard has been implemented yet. You can add your own system using qualifications by only granting people quals who have passed a particular test.
--What I was really getting at here was whether Amazon had any interest in providing a pool of workers who could be trusted to view private or confidential information. From what I gathered tonight I think they're more interested in providing the interface and letting Requestors work out this sort of thing for themselves. There is apparently nothing preventing you from only allowing exactly who you want to have access to the Hits that you provide to the MTurk system.
3. Alan: I can imagine ways to use MTurk where you might only pay for a hit if a worker happens to find something. This would lead to a high rejection rate that might affect their ability to do other hits. Any thoughts on how to handle that?
Answer: We are considering exposing info about the requesters such as their approval and rejection rates for different types of hits. This would give workers insight into if they want to work on those particular types of HITS or not.
--Again I didn't really explain what I was getting at very clearly. I might want to have workers to look through a set of scanned receipts and flag any that have problems. If I only wanted to pay when they found an erroneous receipt, I'd have to reject those that don't. It occured to me I could account for this by adjusting the rate I paid for each submitted hit, but it might be useful to have a way to only selectively pay for a hit without just rejecting them.
4. Alan: How long do you anticipate MTurk staying in Beta?
Answer: Until we are ready to launch!
-- A bit of a flippant answer, but it was an off-the-cuff question. What I should have asked is if anything will change once MTurk leaves Beta or if there's any reason to wait until it leaves Beta status to submit requests.
5. Alan: Will work always have to be done through MTurks website?
Answer: Today we don't have a set of APIs, but we are actively looking at ways to incorporate MT into the worker side of your applications. We would love to hear of your requirements in that space.
-- Someone else asked basically the same question, but it didn't make it to the chat window. The answer was to their question. It sounds like they're open to other interfaces being developed, but don't have the tools available yet.
6. Rob: I've noticed you've added CAPTCHA support to the site, is this feature going to be automatically applied to HITs requestors submit?
Answer: Yes, it is applied for all new users after the first 5 HITS and then progressively thereafter, for all hits in the system, not just Amazon ones.
-- The image captcha's are here to stay, which is a great thing and a good first step in causing the script kiddies as much grief as possible.
Friday, December 09, 2005
Still Hittin' It
I've still been doing about 200 or so hits a day, just to keep up with things and add a little extra money to my Amazon account. At that rate I'd make about $100 a month, which is enough to at least make me smile.
One thing that doesn't make me smile is still having to fight for hits. Clicking accept 20 times in a row and not getting a hit to work on is the only thing that still just makes me go red when I'm on the MTurk site. The Monolith is still silent about it of course. Meh.
I'm signed up for an online training session/overview this Tuesday evening with host Jeff Barr. It should be interesting and I hope to get a few questions in and I promise not to heckle Jeff too much. I'm pretty sure I'm bigger than him, but he may work out more. Heh, heh, heh.
One thing that doesn't make me smile is still having to fight for hits. Clicking accept 20 times in a row and not getting a hit to work on is the only thing that still just makes me go red when I'm on the MTurk site. The Monolith is still silent about it of course. Meh.
I'm signed up for an online training session/overview this Tuesday evening with host Jeff Barr. It should be interesting and I hope to get a few questions in and I promise not to heckle Jeff too much. I'm pretty sure I'm bigger than him, but he may work out more. Heh, heh, heh.
Monday, December 05, 2005
5k Hits
I recently submitted my 5000th hit since I started participating in the MTurk project about a month ago. In that month I made a little over $131.00. Not too bad for piddling around in my spare time I suppose.
I've also recently broken down and done a few more Artist Confirmation hits. I still don't really like them, but I have a stubborn, pitiful technique of doing them so if I make three bucks a day doing them I'm happy. I basically skip the hits until I see something like "Pat Boone" or "The Temptations" or something that is obviously "Various Artists." Combining that with have to hit Accept about 10 times before I actually get a hit makes for very slow progress.
I think Amazon's stubborn refusal to do anything about the hits so easily being given to someone else before you have a chance to accept them is soon going to turn me off of the entire project. It's too frustrating. Apparently hits are just shown to as many people as possible until someone clicks accept. It's like they've put you in one of those Vegas "money chambers" where the dollar bills are blowing all around you and you've got so much time to grab as many as you can.
In the past few days people have also been posting on the message boards that they always choose "None of the Others" when doing IA hits. I personally find this distasteful, but it's hard to not expect it. It's the best way to make the highest amount of money in the shortest amount of time so until they implement some method to discourage it, IA hits are probably dead.
I've also recently broken down and done a few more Artist Confirmation hits. I still don't really like them, but I have a stubborn, pitiful technique of doing them so if I make three bucks a day doing them I'm happy. I basically skip the hits until I see something like "Pat Boone" or "The Temptations" or something that is obviously "Various Artists." Combining that with have to hit Accept about 10 times before I actually get a hit makes for very slow progress.
I think Amazon's stubborn refusal to do anything about the hits so easily being given to someone else before you have a chance to accept them is soon going to turn me off of the entire project. It's too frustrating. Apparently hits are just shown to as many people as possible until someone clicks accept. It's like they've put you in one of those Vegas "money chambers" where the dollar bills are blowing all around you and you've got so much time to grab as many as you can.
In the past few days people have also been posting on the message boards that they always choose "None of the Others" when doing IA hits. I personally find this distasteful, but it's hard to not expect it. It's the best way to make the highest amount of money in the shortest amount of time so until they implement some method to discourage it, IA hits are probably dead.
Subscribe to:
Posts (Atom)