Here is an update to the Fossett story. 50,000 searchers. Lots of false leads given to the Civil Air Patrol.
1. "Leads" Should have needed multiple blind confirmations. (Labor is free make redundancy your test on eliminating false leads.)
2. The user interface didn't work.
3. Poor images made the task very difficult.
4. Didn't leverage the network to sort and prioritize the volunteers.
5. Didn't realize the complexity of working with volunteers vs. Turks getting paid.
Link: Online Fossett Searchers Ask, Was It Worth It?.
Looking back, Diana Francis says she should have known it would be a big waste of time. She sat for hours each day in her husband's home office in Houston scouring little digital snapshots of the Nevada desert on Amazon.com, in hopes that she'd help locate vanished millionaire aviator Steve Fossett.
Finally, though, she decided the exercise was tedious and unproductive.
"It was so exciting and new when we started it and it seemed like it could really help them, but eventually it was disheartening, and I realized I had no idea what I was actually looking for," says Francis, who participated for a couple of weeks while her kids were at school. "You know the saying, 'a needle in a haystack'? Well, this literally was like looking for a needle in a haystack the size of a small European country."
She's not the only one now expressing doubts about Amazon's Mechanical Turk, a high-tech aspect of the Fossett search that received such vast media hype that Mechanical Turk's director, Peter Cohen, won't do interviews about it any more. The online retail giant took the most up-to-date satellite images of the 17,000-square-mile search area, broke it into smaller chunks, and had more than 50,000 volunteers look at randomly distributed segments. In Mechanical Turk parlance, each segment was a small job, known as a Human Intelligence Task or HIT, which required the assigned volunteer to flag anything thought to be out of the ordinary.
Fosset disappeared Sept. 3 during what was planned as a brief jaunt from a ranch 90 miles southeast of Reno, Nevada. The massive online effort didn't lead to the discovery of Fossett or the single-engine Citabria Super Decathalon he was flying. But neither did the dozens of planes and hundreds of ground searchers who made up the biggest search for a missing aircraft in U.S. history. To date, it remains a mystery what happened to Fossett.
Amazon closed the search last week, almost a month after the official on-site search ceased. Now that it's over, Amazon spokeswoman Kay Kinton says the company has learned much, and she gives the system high marks for its ability to update and adapt as the situation changed.
Still, many of those who participated have mixed feelings about their experiences. Francis, who says she's "not that much of a geek," regrets taking part, but many who are more knowledgeable about the technology say it was a worthwhile exercise that should help Amazon refine its methods in the future.
"There was always the hope that people with good eyes would hit the right image, but it's also a learning experience," says Ken Barbalace of Portland, Maine, who runs the website EnvironmentalChemistry.com and who looked at 25,000 HITs. "We can't figure out how to make it a valuable tool until you work on it and change things."
The most important change Amazon needs to make for the future, Barbalace says, is that the interface ought to offer a way for searchers to toggle between the image they're given and an image of the same section prior to the date of the search target's disappearance. That would have helped volunteers know whether the things they were spotting were new.
Instead, some volunteers took the GPS coordinates from the squares they were issued and fed them into Google Earth for older images, slowing down their progress. And in the last couple of weeks when Mechanical Turk started using higher-resolution images, the GPS coordinates were no longer listed with the images, which made matching the photos even more of a challenge.
Some volunteers believed that information was withheld because Amazon began to worry that helpers would try to actually go to the sites themselves to search. But Kinton says it's because the source at that point changed from satellite imagery to images taken from aircraft, which didn't have GPS coordinates attached.
Another intense Turker, Andy Chantrill of Castle Donington, England, says he wishes Amazon had provided the searchers with more information about the overall effort. The 25-year-old software designer says he put in 85 hours poring over 20,000 HITs. Since each square was reviewed by up to 10 people, he says he'd like to know how many others had flagged ones he looked at.
"The value of the contribution is hard to quantify because ultimately we failed to find Steve, but it seems reasonable to imagine that this could work," Chantrill says. "I don't see any downsides to it, so long as people don't pester the professional search-and-rescue teams with poor leads."
Yet that is exactly what happened, much to the exasperation of Civil Air Patrol Maj. Cynthia Ryan, who says her e-mail and voicemail boxes were flooded with leads from folks working on the Mechanical Turk. Many times, they mistook search aircraft in the air for Fossett's plane -- even though it's unlikely Fossett's plane would have appeared intact.
"The crowdsourcing thing added a level of complexity that we didn't need, because 99.9999 percent of the people who were doing it didn't have the faintest idea what they're looking for," Ryan says.
"In the early days, it sounded like a good idea," Ryan continues. "In hindsight, I wish it hadn't been there, because it didn't produce a darn thing that was productive except for being a giant black hole for energy, time and resources. There may come a day when this technology is capable of doing what it says it can deliver, but boy, that's not now."
