The Right Search Tool

With a single dominant player, the Web search market hasn’t exactly been fertile ground for startups and emerging companies with innovative new technologies. But a flurry of venture capital investment, product development, acquisitions, and other industry activity seems to mark the beginning of a new era in search, one in which specialists play a bigger role.

Google (Nasdaq: GOOG) handles 62 percent of Web search queries, followed by Yahoo Inc. (Nasdaq: YHOO) and Microsoft Corp. (Nasdaq: MSFT), according to research firm Nielsen Online . Behind them is a proliferation of upstarts, promising nuanced forms of consumer-oriented Web searches, arcane new “semantic Web” and contextual search services, mobile search, and search results in other guises, such as NewsGator feeds.

The burst of activity over the past 12 months is more befitting a land rush than a market dominated by one powerhouse:

• Semantic Web search engine Hakia Inc. , founded in 2004 by a group of European scientists, debuted late last year a “social search” feature designed to connect users with people of similar interests. A $5 million investment in January brought its total to $21 million.

• In January, contextual search specialist Silobreaker Ltd. launched its new engine for relations-mapping and trend analysis on topics and people.

• A few weeks later, startup Surf Canyon released its Discovery Engine, a browser plug-in that disambiguates (i.e., reorders based on relevance) search results from the major engines in real time.

• In March, SearchMe Inc. launched its “visual search” capability and category suggestions for nontext searches. The three-year-old company landed $16 million in venture capital this year, bringing its total to $31 million from the likes of Sequoia Capital and Lehman Brothers.

• Ask.com , one of the original “plain English” search engines, said in May it will acquire Lexico, the owner of Dictionary.com and Thesaurus.com.

• Search aggregator Viewzi , which presents results from top search engines in innovative and user-friendly interfaces, introduced its new tools in June.

• Search startup Cuil Inc. , created by a pair of Google veterans and purporting to index billions more Web pages than Google, launched in July.

For business users, the flourishing of new search technologies could bring relief from a growing dissatisfaction with the information overload delivered by a typical Google search. At-work surfers (assuming they’re actually working and not looking for the latest Gnarls Barkley video) “are not just exploring and experiencing things on the Web,” says Surf Canyon founder Mark Cramer. “Their objective is more specific, and they need to get it in a more timely fashion.”

The big three aren’t standing still. In July, Yahoo announced Yahoo Search Boss, a Web-services platform that will let developers create new search tools based on Yahoo APIs -- tools that could compete with Yahoo itself. About the same time, Microsoft moved to bolster its Live Search with the acquisition of Powerset, a developer of semantic search.

And Google continues to plow ahead. Interviews with Google’s search gurus show that the company is devoting a chunk of its $16.5 billion in annual revenue to improving its search tools and investing in businesses that drive users to those tools. Yet Google is spread thinner than ever: It’s promoting its operating system for mobile devices, Android; putting out a browser to compete with Internet Explorer; pushing into cloud computing; dedicating resources to enterprise search; and developing an entirely new industry around clean, renewable energy sources.

That’s a full slate even for a company with a market cap around $146 billion. Like many technology leaders before it, Google could get distracted from its core business and miss the next wave of innovation. “I think there are major opportunities being created because there’s so much broad focus on how to be a search behemoth,” says Brad Bostic, co-founder and president of mobile search provider ChaCha Search Inc. “That makes it difficult to be really great at all those niche areas.”

Next Page: When One Search Doesn’t Fit All

What does all of this research and development add up to? We’re entering what Danny Sullivan, editor in chief of Website Search Engine Land , refers to as the Search 3.0 era.

Sullivan describes Search 1.0 as being the late-90s period of multiple search engines -- AltaVista, Excite, HotBot, Infoseek, Lycos, WebCrawler -- that used simple text-search criteria to find and rank pages. Search for “nerf ball” and you’d get a lot of pages that use the terms “nerf” and “ball” in close proximity to each other. As marketers decoded those Web crawlers, tactics to game the system proliferated, and the usefulness of such blunt-force search tools declined.

That gave rise to Search 2.0 -- a.k.a. the Age of Google. Moving beyond text search, Google’s PageRank system uses a sophisticated algorithm to rank pages according to a set of “off-the-page” criteria, particularly the number and quality of links to a given Web page. That’s the primary form of Web search today, and it works fairly well for an extraordinary number of people.

EVEN GOOGLE HAS ITS LIMITS
Google has been offering its next generation of Web search since May 2007, when it introduced its Universal Search capability to return content from videos, images, news, maps, and books alongside conventional links to Web pages.

At the time, Marissa Mayer, Google’s VP of search products and user experience, described it as “integrating its siloed search engines.” Yet most of us remain firmly rooted in our silos. “We’ve barely scratched the surface with universal search,” Mayer said in her The future of search blog post Sept. 10, where she promised a new interface and user experience related to new media in the coming months.

Marissa Mayer

Google’s VP of Search Products and User Experience looks ahead

Just to begin universal search was an engineering effort that took more than 100 engineers longer than two years, letting Google run a search query across multiple search indexes without multiplying the resulting computational load in a geometric progression.

In her blog post, Mayer cited four big areas search would improve upon in the coming years, of which Media is one, including universal search across types of media. Another is Mode, such as using simpler mobile devices by speaking instead of typing, or even by entering a picture instead of words. The other two are Personalization, including location-based and social connections, and Language, meaning translations across them.

Google isn’t the only company seeking to add features to its search results. It has unmatched resources, but that doesn’t guarantee it will deliver the breakthrough most relevant to business users, acknowledges Peter Norvig, Google’s director of research.

Take universal search. Google serves perhaps the ultimate mass market. It’s in the business of satisfying a high percentage of general search queries, not creating highly specialized tools for data-intensive forms of search such as video, images, and other content. Norvig uses the example of military analysis. A pure-play military research firm with clients searching for photos of nuclear missile sites can afford to spend a lot of time fine-tuning a specialized search app by hand. Even Google, with its 10,000 engineers, can’t lavish that kind of time on a single-application search tool, so it has to “learn by example,” as Norvig puts it. Here are some pictures, here are some words in a caption: Can you associate certain words with certain kinds of pictures and learn to generalize?

“There may be companies that can really focus on one thing, and do a better job,” says Norvig. “That’s up to them.”

— Richard Martin

Now, though, there’s growing realization of the limitations of Google’s PageRank system and its one-search-fits-all results page. Thus the fledgling Search 3.0 movement: the blending of vertical or specialized search results along with the horizontal across-the-Web results normally provided by Google or Microsoft’s LiveSearch.

Google has its own take (see sidebar), but the most interesting action is in startups developing tools that do things the big guys don’t. “That’s where the opportunity is going to be,” says IDC analyst Susan Feldman.

Take Silobreaker, a contextual search engine that “brings meaning and context to Web content,” according to Kristofer Mansson, CEO of the search and relational-analysis provider, who spoke at the Demo conference in Palm Desert, Calif., in January.

Silobreaker is particularly useful for tracking news makers and current-events topics. It provides search results with value-added elements including context extraction (how a person or topic fits with other people, institutions, or categories), mapping, trend tracking (graphing numbers of mentions in the world press), and relationship mapping. The engine draws content on global issues, science, technology, and business from 10,000 news, blog, research, and multimedia sources.

While Silobreaker is being introduced as a free Web tool, Mansson sees opportunity for big companies and government entities, including intelligence agencies. Mats Bjore, the company’s director of business development, helped set up the open-source intelligence efforts for the Swedish Armed Forces.

The coolest part of Silobreaker is its relational mapping, which displays in graphic form the web of people and topics in close proximity to the search term. By manipulating the nodes in the map, you can see how the points of interest relate to one another; clicking on the midpoint between two nodes reveals an article or other content that explains how the two are related.

Businesses could use Silobreaker to track market trends, map connections among news events and products or industries, and trace links between competing executives, their previous companies, and industry developments. Mansson foresees partnering with content publishers or leading companies in different industries to create highly targeted tools that draw in premium content as well as general Web pages, whether the topic is fuel cells, derivative trading instruments, or nanotechnology.

“When general news content is not enough, we could add on research and premium content, together with a domain expert, to create a must-have service for that specific sector,” Mansson said.

Mahalo.com Inc. , launched the same month as Google’s Universal Search, is a back-to-the-future engine that uses human editors to sift through automated search results to separate the wheat from the chaff. Founded by former tech blogger Jason Calacanis, Mahalo employs Wiki-like, community-generated content to improve search results. A search for Pablo Picasso, for example, brings up a quick Picasso biography, a Top 7 Web page ranking, plus links to Picasso works online and related videos.

Employing editors to sift the vast Web, though, seems a Herculean task, even if its goal is to supplement Google, not replace it. Though its investors include CBS Corp. (NYSE: CBS), News Corp. (NYSE: NWS), and Sequoia Capital, Mahalo isn’t destined to be the businessperson’s first stop.

Next Page: Media-Centric Search

Other new search engines are based on probing different forms of content. Viewzi, a visual search engine launched in June, presents results in viewing options based on the type of content being sought: If you’re looking for a video, you’ll see a “video wall” of clips from YouTube Inc. , Veoh Networks Inc. , and elsewhere. Web pages are displayed as thumbnail previews, rather than text links. Viewzi also uses a cloud-like 3D interface to display sites in relation to other relevant pages.

“We can’t outdo Google,” says Giovanni Gallucci, Viewzi’s “Social Media Ninja,” so Viewzi is focused on ways to “remake search” without reinventing what exists.

Giovanni Gallucci

Remake search, don’t reinvent it, says Social Media Ninja Gallucci

In this case, that means using results from the major search players but displaying the results in more creative ways. Especially enticing for business users is Viewzi’s capability of getting to know you: The system can recall past searches and which results were chosen, along with how a user chooses to have them displayed and how to interact with the results. So if you regularly search for stock photos of penguins in their native habitat, for instance, Viewzi will offer up waterfowl-related content as opposed to stuff related to Pittsburgh’s professional hockey franchise.

“We can learn on an individual basis through the entire engine, so the engine is tailored to your specific needs without you doing anything,” says Gallucci.

Even more individualized is Rollyo Inc. , a four-year-old company that lets people bundle up to 25 Websites into a custom search engine (or “roll your own” -- get it?). Users can make their customized search tools public and share them via the Rollyo interface. Google’s Custom Search Engine offers a similar capability, though the approach hasn’t caught on widely.

Then there are search engines targeted at specific audiences, like Quintura for Kids, which uses a “cloud search” feature to give children layers of tags (e.g., “television”) that let them quickly move to relevant sub-categories such as The Suite Life of Zack & Cody or Hannah Montana without typing in queries.

Finally, there are new tools that aren’t really search engines at all. Often these come embedded in other types of applications. Launched in February, Surf Canyon is a browser extension that uses “semantic real-time implicit personalization” to sort results from the large search engines and pull the most relevant to the fore. Surf Canyon watches and learns from user behavior signals to calculate “instantaneous relevancies” to select the most useful needles from a mountainous haystack of search results.

Another is Jodange , a startup that in September launched the second version of its Top of Mind tool for tracking sentiment and perceived value from key opinion holders. Jodange’s technology identifies influential figures on specific topics and monitors how their opinions shift over time, as well as how they drive sentiments across the media and the public. Founder and CEO Larry Levy offers a free version at www.jodange.com but plans to offer a subscription service to companies seeking to “understand who’s saying what about their market, their competitors, their company, and the executives,” he says.

“For too long, we’ve been forced to use this blunt instrument called 'keyword search,' ” notes Levy. Now, semantic Web search tools and other innovations are delivering “the ability to start to home in on what people are most interested in, and how to serve up information to them, based on the fact that we know their purview, their point of view, their interests.”

In their constant search for a productivity edge, business users have been waiting for that promised land for years. “Like many innovations, the concept was born out of frustration,” CEO Mark Cramer says of Surf Canyon’s tool.

As niche products take their place alongside mainstream tools, the frustration of having one imperfect search engine soon could be replaced by a new one -- having to constantly switch tools based on the type of work a person’s doing. Nevertheless, if new search technologies do a better job of finding the information people need, it’s a consequence many of us will accept.

— Richard Martin, Editor at Large, InformationWeek