By MICHAEL LIEDTKE The Associated Press
Tuesday, March 2, 2004; 1:03 AM
SAN FRANCISCO -- Internet giant Yahoo! Inc. is adopting a new system for indexing Web pages that will charge businesses to include more material currently unlisted in its online search engine, marking the first volley in a duel with its former ally Google Inc.
Sunnyvale-based Yahoo is touting the approach, scheduled to be announced Tuesday, as a practical way to assure its search engine captures more of the so-called "Deep Web" - the billions of pages that aren't found during periodic crawls of the Internet.
The method, often called "paid inclusion," also will help Yahoo's search engine keep better tabs on the most current material on a Web page, company officials said.
More than 99 percent of Yahoo's search index will consist of Web links that don't pay fees, said Tim Cadogan, the company's vice president of search.
Search engine analysts generally applauded Yahoo's move, saying it could open a rich new vein of content that's lacking from all Internet search engines.
But the fees required to participate in the program are likely to raise worries about Yahoo creating an online caste system dividing the haves and have nots of the Internet.
To ease those concerns, Yahoo isn't charging nonprofit Web sites to add unlisted links to its search engine. The nonprofit sites initially participating in the new indexing system include National Public Radio and the Library of Congress.
While Yahoo's index will continue to include Web sites that don't pay the fees, there's no guarantee on how frequently those destinations will be visited nor how extensively the content will be analyzed, Cadogan said.
The fees won't buy Web sites a higher ranking in Yahoo's noncommercial search results, Cadogan said.
The fees under Yahoo's "Content Acquisition Program" will be based on the size of the participating Web sites, how many unlisted links are submitted and how frequently the links are clicked on by the users of Yahoo's search engine.
