Search TanglesInternet search is big business. But the drive for profits by search firms and the sites they index is taking its toll on the results in your browser.
Illustration by Joe Zeff
Now head over to Yahoo and type in "spas and hot tubs." Again, in early June, the first and fifth results pointed to the Watkins Manufacturing-owned HotSpring and Caldera spas Web sites. Watkins pays Yahoo an annual fee to guarantee that Yahoo will index these sites, and also pays a small sum when someone clicks on a link to the sites in Yahoo's search results.
Yahoo and Google, the King Kong and Godzilla of search engines, have long dedicated portions of their results pages to commercial links, clearly labeled as "sponsored" and displayed in separate blocks.
But on any search engine, the truly valuable real estate is the "actual results"--the place where you expect to find answers to your queries. And it is here that some search companies and commercial Web sites are using new tactics, and putting new spins on old tricks, to affect search results. Most of these techniques are legitimate; others are deceptive and unethical.
One thing is clear, however: Search results are being manipulated to a greater degree than ever. With Google about to become a publicly traded company, and with Microsoft preparing its own search service, search is a big business that is about to get even bigger. And the pursuit of profits--by search companies and by Web sites that depend on search engines to drive revenue-producing traffic--is affecting how your search queries are answered.
Pay for Play
One search engine practice that's become controversial in recent years is called paid inclusion. Because even the best search engines can't index the entire Internet, some--namely Yahoo and Ask Jeeves--allow Web site owners to pay a small fee to guarantee that their site will be included in the search engine's index. However, both firms insist paid submissions don't affect search rankings.
Google doesn't accept paid submissions. You can submit a site to Google for free, but there's no guarantee as to when or if the site will be added to Google's index.
Ask Jeeves' Site Submit program for its Ask.com search engine charges a one-time fee of $30 for the first page submitted and $18 for each subsequent page. A disclaimer next to Ask.com's search results says that some sites have paid to be indexed.
In March Yahoo rolled out its Site Match Exchange paid-inclusion program. Participants pay a $49 annual fee for the first URL, $29 per page for the next nine URLs, and $10 per page thereafter to guarantee not only that those pages will be indexed but that they will be recrawled (Web jargon for reindexed) every 48 hours, compared with a schedule of up to a month for nonpaying sites. Participants can fine-tune content with the latest products and prices, and they can ensure that their site is completely indexed (search engines normally index only up to 1000 pages per crawl, even on a large site--and they do not index a site's searchable databases at all).
Illustration by Gordon Studer
Site Match's impact on search results is highly debated. Critics--and even some supporters--believe participating sites are more likely to rank high in Yahoo's results than nonparticipants. But Keith Boswell, chief operating officer for Marketleap, a marketing firm that resells Site Match, says, "The more quality content you have in your Web site, the more likely [it] will be relevant and rise higher in search results."
Gary Ruskin, executive director of Commercial Alert, a nonprofit dedicated to keeping commercial and public interests separate, says programs like Site Match make "moneyed interests" more visible in Yahoo's search results than smaller firms that can't afford to pay for premium indexing. Ruskin believes that, as more companies pay so as to compete with Site Match customers, Yahoo's search engine will become little more than a searchable yellow pages directory masquerading as unbiased, noncommercial Web search results.
Search engine companies bristle at the notion that their search results are tainted by money. A Yahoo representative said that Yahoo's quality speaks for itself. "People wouldn't return to Yahoo if they didn't trust us," she said.
Similar to Ask Jeeves, Yahoo displays a "What's this?" link (just above the search results area) that goes to a disclaimer stating that about 1 out of every 100 sites in its index has paid to be included. But several other sites that use Yahoo's search results--including Dogpile.com, Excite.com, and MSN--do not display such a disclaimer.
MSN, however, plans to end its relationship with Yahoo and implement its own search technology. In the meantime, MSN and Ask Jeeves both say they plan to more clearly identify Yahoo-produced paid and commercial listings.
Google, the number one search engine, has its own problems, many caused by a cottage industry of so-called optimization firms that promise to boost rankings of customers' Web sites. Their efforts can reduce the quality of Google's search results.
Some techniques are legitimate. Marketleap and Position Technologies, for example, help site owners analyze content and better describe products so search engines can more easily identify and rank relevant content. These measures help explain why product searches mostly yield links to vendors rather than to reviews or news.
But some ratings-boosting techniques are controversial.
A practice called cloaking occurs when a page appears one way to a search engine's indexing technology but looks quite different to people who click on the link to it.
Cloaking can have perfectly legitimate uses. One example: National Public Radio creates cloaked pages with rough transcripts of its radio programs. People who click on search engine links to these pages get to the archived audio files of the broadcasts and don't see the transcripts. PC World similarly uses cloaking for articles in its database.
But the desktop advertising company WhenU ran afoul of Google in May over cloaked pages that Google believed had misled people who searched on "WhenU." The cloaked pages, which included media reports critical of WhenU's practices, placed high in Google's results, but when users clicked on those links, they saw positive reports only. Google banned WhenU from its index for violating its policies, a ban that was still in place at press time.
In an e-mail response to PC World's queries, WhenU said a search optimization firm created the pages without WhenU's knowledge, and that the company "instructed the outside firm to reverse their actions" as soon as it learned about the cloaked pages.
Another way companies try to change results is by using multiple domains to sell the same products. This increases the odds of being found by search engines and being listed multiple times.
Google tries to excise manipulated results, but it's easy to find pages that violate its policies. Google, in a quiet period before its initial public offering, did not comment.
It's easy to see why Google and Yahoo wield such power: They serve more than 90 percent of results at the top 25 search engines, says ComScore QSearch. And Jupiter Research projects that U.S. search engine ad sales will reach $2.1 billion in 2004, up from $1.6 billion last year.
But observers say consolidation of the search market hurts users. "Fewer voices in search mean fewer options for consumers," says Danny Sullivan of the Search Engine Watch online newsletter.
Web searching is clearly evolving. Microsoft's initiatives should heat up the competition (see "The Future of Search"), and new search tools are on the horizon. With any luck, the search engines that win this race will be the ones with the most relevant results.
The Future of Search: This Time It's Personal
Illustration by Joe Zeff
Yahoo says it is considering similar desktop search technology, but declined to provide details. Google was in a quiet period leading up to its initial public offering; published reports, however, suggest it too is working on desktop search software.
Hints of where things may be headed are online right now. Search for "Chinese food Houston" at Yahoo, for example, and the top results are not links but the names, addresses, and phone numbers of Chinese restaurants close to Houston's geographic center. Yahoo believes searchers prefer immediate answers to links. Over 140 million people have volunteered their personal information to Yahoo, and it plans to use that data to make search more relevant and personal to them, the company says.
MSN, meanwhile, says its upcoming search engine also will focus on offering personalization and delivering answers.
Amazon.com is pushing the search envelope with a trial search engine called A9.com. Launched in early 2004, A9 ties regular Web search results from Google (enhanced with additional details such as names of sites popular with other visitors to the link) to "Search Inside the Book" results from Amazon. It also stores the user's recent A9 search history.
Eurekster, launched earlier this year, combines social networking with personalized search. Results are based on the search preferences and behavior of your Eurekster contacts as well as your own search history. If you choose classical sites when you search for music, they will dominate your results the next time you search for music.
You can perform a search tied to a Eurekster special-interest group--either a public one such as Parents or Organic Gardening, or a private group. Results from these searches will include sites popular among other group members.