washingtonpost.com
How you and Google are losing the battle against spam in search results

By Michael S. Rosenwald
Washington Post Staff Writer
Saturday, January 29, 2011; 1:11 PM

Earlier this month, my friend Rebecca Skloot replaced her hulking big-box TV - I can vouch for its girth, having moved it once - with a flat-screen no thicker than an iPad. She turned it on and, horror of horrors, the picture was lousy.

Displeased, she turned to Google for help. What the search engine delivered was a mess, a collection of spammy sites riddled with ads. So she turned to Twitter, posting: "Old TV died, got newfangled LED TV. Shocked how bad/fake movies look! . . . Others have this prob?"

Solutions to Skloot's technological melodrama rolled in. Fix this setting, turn this off, shazam! A few hours later, she posted: "Thx 4 fixing my TV today! It's example of how Google=in trouble. Googled 4 fix, got spam sites. On Twitter answer=asap."

Skloot's story seems ever more common these days. Google is facing withering criticism from tech bloggers and search engine experts who say the world's premier gateway to digital information is increasingly being gamed by spammers. Google, they say, is losing.

One tech blogger, the well-known iPhone app developer Marco Arment, wrote a post about "Google's decreasingly useful, spam-filled web search." Another blog offered a piece titled "On the increasing uselessness of Google." Yet another headline spoke of "Trouble in the House of Google."

Data seem to back them up. Google's success rate, as measured by the percentage of users visiting a Web site after executing a search, fell 13 percent last year, according to Experian Hitwise, which monitors Web traffic. Microsoft's Bing search engine increased its search efficiency by 9 percent over the same period.

Although there could be several reasons for the disparity, one is most certainly spam in Google's results, analysts said.

"It's clear that Google is losing some kind of war with the spammers," said tech guru Tim O'Reilly, who often cheers Google's technology. "I think Google has in some ways taken their eye off the ball, and I'd be worried about it if I were them."

(Your Take: Is Google getting less reliable?)

Google recently responded with its own blog post, acknowledging some problems and promising to fix them.

"Reading through some of these recent articles, you might ask whether our search quality has gotten worse," the statement said. "The short answer is that according to the evaluation metrics that we've refined over more than a decade, Google's search quality is better than it has ever been in terms of relevance, freshness and comprehensiveness. . . . However, we have seen a slight uptick of spam in recent months, and while we've already made progress, we have new efforts underway to continue to improve our search quality."

Google's predicament, analysts say, comes at a critical moment in the life of the Internet. The company generates billions of dollars in revenue from search ads. But social networks such as Twitter and Facebook offer people the ability to gather information online the way we always have offline - by asking people we know. Studies show we often give greater trust to information gathered from sources we know than from those we don't.

The pool of people who could answer our questions online is growing fast. Internet users spend more than 20 percent of their time online using social networks. Last year, with more than 500 million users, Facebook topped Google as the world's most visited Web site. (Washington Post Co. Chairman Donald Graham is on Facebook's board of directors.)

While millions of people still use Google every day with no problems, I now see requests in my Twitter and Facebook feeds for things I once Googled: opinions on new cars, the best home repair person, the best place to eat, how to find a developer for iPhone apps. When I asked a friend on Facebook why she turned to her friends for new-car ratings, she replied: "facebook makes me smile b/c everyone has opinions about everything. i thought that, in this case, the opinions would actually be helpful. i can't read big google search findings b/c i have no patience."

A good side, a dark side

No doubt, there is a lot to wade through, with more than a million new spam Web pages created every hour, according to Blekko, a new search engine trying to take on Google. Optimizing what search engines find on all those pages is a big business - with a good side and a dark side. On the good side, analysts say, there are legitimate search engine experts such as Jill Whalen, founder of the Boston-area firm High Rankings, who works with big companies to make sure important information appears high up in search results. These search engine optimizers are masters of key words, site design and other techniques.

"It's very frustrating for decent SEOs right now," said Whalen, who recently wrote a blog post headlined, "Dear Google, Stop making me look like a fool!" "I would think it would be very hard to get into this business now as a new SEO because there is just so much noise."

The number of links to a Web page is one measure Google uses to rank Web sites, and that brings us to the dark side of search engine optimization: There are services that sell links to content that spammers are promoting. The more links to a Web page, the better.

In a particularly devious way of goosing a page ranking, some spammers use Amazon's online marketplace for Web work, Mechanical Turk, to hire people at a nickel a click to follow links that would boost their site's presence on Google. Panos Ipeirotis, a New York University business school professor, studied Mechanical Turk's new work offerings late last year and found that more than 40 percent were for spamming work. "The results were disturbing," he wrote.

Then there are the content farms. Companies such as Demand Media, which raised $151 million in an initial public offering Wednesday, and AOL have created businesses that pay writers - or anyone who can reasonably string sentences together - to post content that answers popular search queries. AOL's Seed division offers low-paid writing assignments based on hot topics. Demand Media, particularly through its eHow Web site, specializes in how-to information. The company has told investors it relies heavily on revenue generated through Google's advertising program.

Content farms are masters at surfacing their content high in Google's search results. The content, critics say, is often less than masterful. A search on Google for "how do I calibrate flat screen tv" turns up two eHow sites in the first three links. One is written by someone named Jenna Johnson, whose other articles include "How to Organize a Small Bedroom" and "How to Use Leftover Meatloaf." (Make a meatloaf sandwich, she suggests.)

Demand Media declined to comment, citing the quiet period after its IPO, but company executives have in the past rejected the "content farm" label. AOL Senior Vice President David Mason also disputed the "content farm" term, saying the company pushes its authors to write deep, engaging content. Seed recently highlighted an item on its blog about cleaning eyeglasses with toothpaste.

My friend with the lousy TV picture found nothing in the eHow posts - or the other links that Google surfaced - that helped her.

"I was about to take my new TV back to the store because I thought it was just a kind of TV that wouldn't work the way we wanted it to," she said. Then Twitter, not Google, saved the day. Asked how her picture is now, Skloot said, "Amazing."

A step behind

Google, for its part, says that it has long been aware of link buying and other spamming tricks and that its algorithms take all of that into account. But many of Google's critics wonder how far ahead of the spammers the company really is.

"I have a hard time believing they are losing this badly," O'Reilly said. "I look at more and more categories of searches that are clearly spam" - medical information, product reviews, technology help, restaurant tips.

As for content farms, Google engineers said the company must come up with better ways to keep that content from showing up so high in search results.

"We are hearing the complaints about this type of shallow content, and we are going to address it," said Amit Singhal, the most senior engineer on Google's page-ranking team. "We have to make sure our users are getting the best information, and if they are not, we have failed."

The company, in its blog post, said it recently launched a new internal technological tool that makes it harder for "spammy" content to rise in results. Other tools and strategies are forthcoming, executives said.

But many Internet analysts wonder whether the new spam-fighting effort, although needed, is beside the point.

"We have kind of stretched the usefulness of search engine algorithms for surfacing the kind of specific content we are looking for," said William Tancer, general manager of global research for Experian Hitwise.

The future, many believe, is social search. Microsoft, which is a major investor in Facebook, can scour a searcher's own Facebook account on every query. Facebook, noting an increase in users who ask for information in their status updates, is testing a new function called Facebook Questions.

"If we notice a lot of people using a feature in a particular way, that's when we consider specializing a product for that need," said Bret Taylor, Facebook's chief technology officer.

"Anyone can answer your question, which means you can tap into the collective knowledge of the millions of people on Facebook," the company says on a Web page explaining the new tool. Answers that users rate as high quality can then be highlighted beyond a user's own social network, pushing answers from one network of people to many, many others.

A hot start-up called Quora is following a similar question-and-answer model. Meanwhile, Google is also working on social search, with co-founder Sergey Brin reportedly heading up the effort. He recently said the company has scratched just one percent of what social search could be. Google already searches Twitter alongside classic Web page searches. Users signed in on a Google account will also see results such as images or status updates from their contacts. Some information from Facebook is included, but nothing from a user's account.

Google spent $50 million last year to buy a company called Aardvark that lets users send questions to people in their social networks. If those friends can't answer the question, it is sent on to friends of friends. Answers are delivered, often immediately, via instant message or e-mail. I gave it a try last week.

I have a nasty case of sciatica and might need to start walking at my desk while I work. I asked Aardvark: "What is the best way to set up a treadmill desk? I want to put a treadmill under my desk sometimes." One smart aleck quickly answered, "Does your desk need some exercise?" But a few minutes later, I got an instant message from someone named Adam K. in Cranberry, Pa., offering a collection of Web links.

"Was Adam's answer helpful?" Aardvark asked.

I clicked yes.

View all comments that have been posted about this article.

© 2011 The Washington Post Company