The Macrosite for News, Analysis and Opinion about the Future of the Internet
Steven Krein

Why Human Power Is the Future of Web Search

Written by Steven Krein
12/5/2007 6 comments
DISCUSS   Digg   Del.icio.us   Reddit   Email This   TWEET THIS

The race for the ultimate online search engine continues to unfold as engineers, math geniuses, and computer scientists work to create more efficient and accurate search tools. But a new crop of search services are taking a much different approach: combining great technology with the power of human intelligence.

In recent years, the hottest Web companies have put endless efforts into creating innovative search technologies and complex algorithms -- the computer programs that index the Web and seek to match your search terms to the most relevant sites. Google is perhaps the greatest example, and its algorithm is helping the company try to achieve an awe-inspiring mission: to organize the world's Web information.

Yet, in the past year, it has become clear that even these complex algorithms have their limits. Faced with a tsunami of new Web pages, user-generated content, and black-hat search engine optimizers constantly trying to "game" the algorithm, it has become increasingly difficult for algorithm-based searches to effectively sort through the clutter on the Web. As a result, users waste time searching, and often become frustrated because they never find what they are looking for.

A powerful trend is re-emerging to answer the frustration of online search engine users. We are getting back to the basics of "people-powered search." Now, a growing number of human search guides are becoming available to assist with queries and help determine the most relevant search results. Acting as intelligent filters, human search guides are armed with even better search tools and new collaboration technologies that make it easy for lots of people to contribute at the same time.

The human-powered search engine concept has evolved from the time Yahoo was using human surfers to index the Web back in 1994. Like most companies competing with Google, Yahoo had to give up the human touch because of the explosive growth of content available on the Web. Now, the goal isn't to keep up with the Web, but to use human intelligence to select the best content, filter the junk, and deliver only the most relevant content to other searchers.

Algorithms get us part of the way there. They instantly scour vast amounts of information, tell us if a page contains our search term, how popular it is, and how many other sites link to it. What they can't do -- what only a human guide can do -- is use critical thinking and experience to better assess which content is trustworthy, relevant, and useful.

New services are emerging that combine these powerful technologies with the wisdom, expertise, and skills of people. They are called "human-powered" sites -- they start with algorithm-based search but add a layer of human intelligence. Companies like OrganizedWisdom Health (where I serve as CEO), ChaCha, and Mahalo are human-powered search engines that have taken on this model, hiring freelance and full-time guides to deliver their own expertise and help users make sense of their search results.

These guides come from all over and often include bloggers, librarians, researchers, stay-at-home parents, and students. They're usually paid based on their productivity and the quality of their work. Fore example, at OrganizedWisdom, we pay guides $10 to $15 for every WisdomCard, or search results page, they create on a particular health topic.

Consider the value of human-guided search for the health sector. Good information can have a huge impact on people's lives, but conversely, confusing or misleading information can be detrimental. Spam sites lead users to unproven, sometimes dangerous treatments, and clutter search results pages, causing many users to give up before locating the information they need. In today's managed-care industry, doctors can't afford to take the time to answer all our questions. Good information is available on the Internet -- we just need to cut through the astounding amount of spam and clutter in the healthcare space to get it. Human-powered search is helping do this.

The algorithms aren't going away, of course, nor should they. But in the coming years we will see Web search and other services increasingly run by this combination of human judgment and technology-driven search. It's not humans versus machines, but humans and machines, driving innovation and solving today's greatest online challenges.

— Steven H. Krein, CEO, OrganizedWisdom Health

DISCUSS   Digg   Del.icio.us   Reddit   Email This
Current display:       newest comments first       display in chronological order
Paul Whyte
Researcher
Saturday December 8, 2007 4:16:44 PM
no ratings
 I think you are perfectly right. I was reading an on-line article a few days ago about the Semantic search engine and was really impressed. Rather than using human intelligence, equipping search engines with semantic intelligence would by far improved the efficiency and capabilities of today's search engines.Though the ideal search engine may be beyond our reach,at least semantic search technology will bring us closer.Of course it has it own drawbacks in the fact that a searcher has to to search for information in a semantic way instead of using single or double key words but at least we can expect to get better search results than what we are getting now.
Phavanhna
Researcher
Saturday December 8, 2007 2:46:39 PM
no ratings
1 saves

I agree that human knowledge and experience are two of the main contributors in a good database search.  Since there is immense amount of information available on the Internet which some of those are not reliable?

To judge if a web content is reliable able or not, it depends on many factors such as author's qualification, data currency, and the purpose of the content. A search engine is programmed to search for the key words that had been put in the page title and the content of the page. Hence, any good web developer is able to make a web site result in a top hit of the search engine despite its real content. On the other hand, the internet search guides that are manual are far more convenience to users who use a very human search content such as: “Where should I go to eat for the coming Christmas?” and it will help users to get to their information quicker since the information is already been categorized and reviewed by someone else. Having said that, human power is one of the most expensive resources; therefore, if everyone is moving to the subject directories search. We might see more advertisements on our search page or more and more database will be no longer free.

Generally speaking, the subject directories and the search indexes should be combined and balanced out in a way that less human will be required to intervene but certain pattern of good content can be learned or recognized by machines.   

Raza
Rank: Cave Painter
Saturday December 8, 2007 6:25:46 AM
no ratings

the concept behind hakia.com is very interesting. Of-course the ability to find specific information depends on ability to ask specific questions

When I tried hakia, some results were very good but some were not that good

e.g

http://www.hakia.com/search.aspx?q=cheap+stores+in+Atlanta

 

But the company surely has some future. In my opinion, it is going to be acquired by a big gun in future ..............

-Raza 

James Johnson
Staff
Thursday December 6, 2007 10:58:49 PM
no ratings

Hey Steve,

OK, when it comes to scalability, Google is king! But there is way too much information on the Web for human-powered search engines to thrive for long. I believe semantic search technology is a more viable process that will have a greater influence on the next-generation Web.

Tomio Geron does an excellent job of describing how semantic search is already making an impact on the Web in his article, Taking on Google: Is Semantic Technology the Answer?

Geron describes it this way:

"The so-called semantic web will be based on search technology that will enable people to type in their questions using everyday language. Developers of next generation search engines say their technology will “understand” the language within those queries—much more precisely than Internet giant Google’s technology.

The new search engines will then scour databases that have already “read” documents from across the web to find the best matches. Specific answers can be found for queries such as “Which is the best oil company in the world?” or differentiate between “movies by women,” and “movies about women.”

If you want to experience a very good semantic search engine, take a look at Hakia.com. According to the article, it received positive reviews for its ability to quickly find very specific information.

james

Elisa Lucia Cundiff
IQ Crew
Thursday December 6, 2007 1:18:21 PM
no ratings

Okay, for those of you who don't know what Cha-Cha is; it's a site designed for your grandmother. Because Grandma doesn’t get Google.

Cha-cha uses real people, who they pay minimum wage, to answer your queries over an AIM-style box. And guess what your guides usually use to search? Google.

The outcome? Your guide will find the same information that you could in a longer period of time and with fewer results.

So, yeah, this site is useful for the internet illiterate but that is a quickly shrinking group of people. Plus, the global trend is to create technology that allows customers to do it themselves; grocery store checkout, airline check-in etc. But we already have that access with Google. Why would we reverse the process?

You seem to suggest that we need this because content quality is degenerating on the net and additionally that this increased clutter causes “users to give up before locating the information they need” because much of the information, even when found, is misleading or just plain wrong. You’re right. The internet is full of lots of different types of crap .

This became pretty clear my freshmen year in college when I entered my symptoms into a WebMD-like site and was informed that I had Multiple Sclerosis. I didn’t. So, the truth is - I actually like your idea (organizedwisdom.com) as it seeks to use professionals to validate specific medical sites. That’s smart. And that could be useful in other areas too; online encyclopedias (as used by Wikipedia), How-To sites (like howthingswork.com), recipe sites, amongst many others. But it’s different than providing a guide – it’s providing a site that posts good sites.

Though, there may still be work to do. Because even with a network of human filters, there is no way to guarantee that this info is of quality. (As evidenced by some of the Wiki debate ).

Mashka
Researcher
Wednesday December 5, 2007 6:50:26 PM
no ratings

Hello,Steven! Thank you for the interesting article.

Well, first  of all, speaking about search in general, people often accuse "bad searching engines", at that time, people  just don't formulate the key words properly. there's a  good web-site about searching engins that explains a lot

And of course, the competence of the information on the web-sites leaves much to be desired. But I am afraid that not everybody understands that.People get used to trust pseudoscintific information-there as a great discussion about it in public and academic discurses-so I think, people need to be  enlightened to use only reliable sources of information,from this it follows the next question: what are reliable sources:))?

The ThinkerNet does not reflect the views of TechWeb. The ThinkerNet is an informal means of communication to members and visitors of the Internet Evolution site. Individual authors are chosen by Internet Evolution to blog. Neither Internet Evolution nor TechWeb assume responsibility for comments, claims, or opinions made by authors and ThinkerNet bloggers. They are no substitute for your own research and should not be relied upon for trading or any other purpose.
previous posts from Steven Krein
Steven Krein
Steven Krein   4/3/2008   6 comments
The options for finding great health information on the Internet are endless. But as with any good situation, there is a downside. The Web also harbors sinkholes of pseudoscience, shady activities, and, in the worst cases, outright fraud, all published in the name of “health.”  Nowadays, you can buy powerful drugs online without a prescription -- some that could kill you if you use them incorrectly.
5
of
IETV: the thinkerNet on film
5
of
2pm EST
Tue
Dec 1st
an IBM information resource
sponsored content
big blue blog
Todd Watson
Todd Watson   11/20/2009   Post a comment
While Google introduces its new Chrome OS (which I'm hearing will be widely available in one year?  Did I mishear that?), IBM announced 10 new products today to help companies using IBM System z mainframe technology.
white papers & case studies
an IBM information resource
sponsored content
Smarter Collaboration: How to Thrive in a Challenging Business Environment
Market conditions are changing faster than ever, and organizations need to improve their agility and adaptability in order to provide better service and improve processes. The ability to work with customers, business partners, and employees as effectively as possible - while at the same time holding down costs - is a key to success.

READ THIS eBOOK
your weekly update of news, analysis, and
opinion from Internet Evolution - FREE!

REGISTER HERE
Wanted! Site Moderators
Internet Evolution is looking for a handful of readers to help moderate the message boards on our site – as well as engaging in high-IQ conversation with the industry mavens on our thinkerNet blogosphere. The job comes with various perks, bags of kudos, and GIANT bragging rights. Interested?

Please email: moderators@internetevolution.com
Copyright © 2009 United Business Media Limited - All rights reserved.      About Us  |  Privacy Policy and Terms of Use  |  Contact Us
CMP Media LLC
Internet Evolution – not for thickies
Tom Nolle
Total Telephony Transcends Telepresence

11|20|09   |   2:11   |   2 comments


The problem with telepresence is that it's not universally accepted, because video calling isn't. While we can all do video calling, we also apparently worry too much about how we look. If we want HD telepresence in our future, we have to dress down, mess up our hair, and dive into our online life.
Full Nelson
SanFran.gov

11|19|09   |   8:51   |   No comments


Fritz has an exclusive talk with the mayor and CTO of San Francisco about that city's latest e-government efforts.
Robert D. Atkinson
America Has Much to Learn About Digital Piracy

11|18|09   |   2:09   |   No comments


The US loses about $20 billion a year on pirated software, movies, and music. But public policy can help stem the tide of digital theft. For example, France has recently passed a 'three strikes and you’re out' law, whereby if after two warning letters an individual continues to download pirated software then his Internet access will be cut off. US policy makers should consider adopting similar policies.
Not Dr. Phil
Internet & TV in Perfect Harmony?

11|16|09   |   2:22   |   3 comments


A new set-top offers an Internet experience without screwing up what we like about TV.
Tom Nolle
Meatballs & Monetization

11|16|09   |   2:00   |   2 comments


Meatballs aren't an obvious source of monetization for the Internet, but Cloudy With a Chance of Meatballs may well be just that, and might also represent a new dynamic in the complex world of Internet content delivery.
The Incredible Hultquist
Tweet Less, Get More Clicks

11|9|09   |   2:24   |   No comments


Evidence shows that you can tweet too much. Sites and services like Twitter and Facebook are a good place to reach your audience, but think quality over quantity.
Steve Saunders' Outernet
The New Information Divide: Part 3

Part 3 of 3   |  
See complete series
11|6|09   |   1:46   |   No comments


A digital content market is emerging. Only two things are known about it: the first is that at some point the Internet will primarily become a paid network. The second known factor is that there are innumerable variables in the digital content market that have yet to be worked out. It’s not known, for example, exactly how users will pay for content (micropayments, subscriptions, bartering of farm animals, other).
Steven Peterson
iPhone App Makes Sense of Public Transport

11|5|09   |   1:19   |   No comments


Routesy is an iPhone application that uses the phone’s GPS to let the user know where and when the next train or bus is coming. The application’s developer, Steven Peterson, talks about why a mobile application makes sense, especially given that this transportation information is already available on the Web.
Reiter's Block
Slobbering Over the N900

11|4|09   |   2:41   |   11 comments


Techies have been going crazy over the pending release of Nokia's N900 cellular phone, which incorporates a newly revised touch-screen operating system. Reiter's got one. Is the craziness justified?
Steve Saunders' Outernet
The New Information Divide: Part 2

Part 2 of 3   |  
See complete series
11|4|09   |   2:19   |   4 comments


Bad news! By eliminating the world’s digital divide we’re likely to create a new divide: the information divide, where we end up creating a two-tier Internet where access to 'quality' content is controlled and charged for by mega-corporations, and the gulf between information haves and have-nots is entirely dependent on how much money they have. This is, of course, an almost exact inversion of the current situation on the Internet – where access is expensive and content is free.
what.the.ferraro
Facebook Lacks Social Skills

11|20|09   |   1:53   |   No comments


Facebook's 'Suggestions' for users demonstrate how little social networking sites understand about true social relationships.
Singer at C-Level
Smart Grid Opportunities

11|20|09   |   2:49   |   No comments


Industry initiatives and government stimulus funds are giving enterprise software vendors a great opportunity to help build out and manage smart grid technologies.
Tom Nolle
Total Telephony Transcends Telepresence

11|20|09   |   2:11   |   2 comments


The problem with telepresence is that it's not universally accepted, because video calling isn't. While we can all do video calling, we also apparently worry too much about how we look. If we want HD telepresence in our future, we have to dress down, mess up our hair, and dive into our online life.
what.the.ferraro
ThinkerNet Wins Min's Award for Best Blogs!

11|19|09   |   1:13   |   4 comments


ThinkerNet wins the Min's award for 'Best Blogs' – Internet Evolution's fifth award this year!
Full Nelson
SanFran.gov

11|19|09   |   8:51   |   No comments


Fritz has an exclusive talk with the mayor and CTO of San Francisco about that city's latest e-government efforts.
Robert D. Atkinson
America Has Much to Learn About Digital Piracy

11|18|09   |   2:09   |   No comments


The US loses about $20 billion a year on pirated software, movies, and music. But public policy can help stem the tide of digital theft. For example, France has recently passed a 'three strikes and you’re out' law, whereby if after two warning letters an individual continues to download pirated software then his Internet access will be cut off. US policy makers should consider adopting similar policies.
Singer at C-Level
Connecting Stakeholders: Part 3

Part 3 of 3   |  
See complete series
11|18|09   |   2:09   |   No comments


Financial management planning does not need to include Voodoo economics, but it does help to tap into the knowledge base of your team through some sort of real-time system. We explore your options.
Reiter's Block
Tweeting for Customer Support

11|18|09   |   2:20   |   No comments


When Reiter gets incensed over incompetent Verizon FiOS order-taking and support, he broadcasts it via Twitter. Did it do any good? How should your company offer Twitter support? Watch this for all the answers.
what.the.ferraro
Dogster.com More Popular Than Gov 2.0

11|17|09   |   2:05   |   1 comment


A lot of attention is being paid to launching Gov 2.0 Websites, but these sites aren't attracting a lot of visitors.
Reiter's Block
Is the BlackBerry 9700 'Bold' Enough?

11|17|09   |   3:07   |   4 comments


The successor to the BlackBerry Bold 9000 – the Bold 9700 – will be available soon in the US. Is it worth upgrading? Reiter's got one, and offers advice.
TechWeb The Global Leader In Technology Media