Pushing Bad Data- Google’s Latest Black Eye

By:Eric Lester

Google stopped counting, or at least publicly displaying, the number of pages it indexed in September of 05, after a school-yard "measuring contest" with rival Yahoo. That count topped out around 8 billion pages before it was removed from the homepage. News broke recently through various SEO forums that Google had suddenly, over the past few weeks, added another few billion pages to the index. This might sound like a reason for celebration, but this "accomplishment" would not reflect well on the search engine that achieved it.

What had people buzzing was the nature of the fresh, new few billion pages. They were blatant spam- containing Pay-Per-Click (PPC) ads, scraped content, and they were, in many cases, showing up well in the search results. They pushed out far older, more established sites in doing so. A Google representative responded via forums to the issue by calling it a "bad data push," something that met with various groans throughout the SEO community.

How did someone manage to dupe Google into indexing so many pages of spam in such a short period of time? I'll provide a high level overview of the process, but don't get too excited. Like a diagram of a nuclear explosive isn't going to teach you how to make the real thing, you're not going to be able to run off and do it yourself after reading this article. Yet it makes for an interesting tale, one that illustrates the ugly problems cropping up with ever increasing frequency in the world's most popular search engine.

A Dark and Stormy Night

Our story begins deep in the heart of Moldva, sandwiched scenically between Romania and the Ukraine. In between fending off local vampire attacks, an enterprising local had a brilliant idea and ran with it, presumably away from the vampires... His idea was to exploit how Google handled subdomains, and not just a little bit, but in a big way.

The heart of the issue is that currently, Google treats subdomains much the same way as it treats full domains- as unique entities. This means it will add the homepage of a subdomain to the index and return at some point later to do a "deep crawl." Deep crawls are simply the spider following links from the domain's homepage deeper into the site until it finds everything or gives up and comes back later for more.

Briefly, a subdomain is a "third-level domain." You've probably seen them before, they look something like this: subdomain.domain.com. Wikipedia, for instance, uses them for languages; the English version is "en.wikipedia.org", the Dutch version is "nl.wikipedia.org." Subdomains are one way to organize large sites, as opposed to multiple directories or even separate domain names altogether.

So, we have a kind of page Google will index virtually "no questions asked." It's a wonder no one exploited this situation sooner. Some commentators believe the reason for that may be this "quirk" was introduced after the recent "Big Daddy" update. Our Eastern European friend got together some servers, content scrapers, spambots, PPC accounts, and some all-important, very inspired scripts, and mixed them all together thusly...

Five Billion Served- And Counting...

First, our hero here crafted scripts for his servers that would, when GoogleBot dropped by, start generating an essentially endless number of subdomains, all with a single page containing keyword-rich scraped content, keyworded links, and PPC ads for those keywords. Spambots are sent out to put GoogleBot on the scent via referral and comment spam to tens of thousands of blogs around the world. The spambots provide the broad setup, and it doesn't take much to get the dominos to fall.

GoogleBot finds the spammed links and, as is its purpose in life, follows them into the network. Once GoogleBot is sent into the web, the scripts running the servers simply keep generating pages- page after page, all with a unique subdomain, all with keywords, scraped content, and PPC ads. These pages get indexed and suddenly you've got yourself a Google index 3-5 billion pages heavier in under 3 weeks.

Reports indicate, at first, the PPC ads on these pages were from Adsense, Google's own PPC service. The ultimate irony then is Google benefits financially from all the impressions being charged to Adsense users as they appear across these billions of spam pages. The Adsense revenues from this endeavor were the point, after all. Cram in so many pages that, by sheer force of numbers, people would find and click on the ads in those pages, making the spammer a nice profit in a very short amount of time.

Billions or Millions? What is Broken?

Word of this achievement spread like wildfire from the DigitalPoint forums. It spread like wildfire in the SEO community, to be specific. The "general public" is, as of yet, out of the loop, and will probably remain so. A response by a Google engineer appeared on a Threadwatch thread about the topic, calling it a "bad data push". Basically, the company line was they have not, in fact, added 5 billions pages. Later claims include assurances the issue will be fixed algorithmically. Those following the situation (by tracking the known domains the spammer was using) see only that Google is removing them from the index manually.

The tracking is accomplished using the "site:" command. A command that, theoretically, displays the total number of indexed pages from the site you specify after the colon. Google has already admitted there are problems with this command, and "5 billion pages", they seem to be claiming, is merely another symptom of it. These problems extend beyond merely the site: command, but the display of the number of results for many queries, which some feel are highly inaccurate and in some cases fluctuate wildly. Google admits they have indexed some of these spammy subdomains, but so far haven't provided any alternate numbers to dispute the 3-5 billion showed initially via the site: command.

Over the past week the number of the spammy domains & subdomains indexed has steadily dwindled as Google personnel remove the listings manually. There's been no official statement that the "loophole" is closed. This poses the obvious problem that, since the way has been shown, there will be a number of copycats rushing to cash in before the algorithm is changed to deal with it.

Conclusions

There are, at minimum, two things broken here. The site: command and the obscure, tiny bit of the algorithm that allowed billions (or at least millions) of spam subdomains into the index. Google's current priority should probably be to close the loophole before they're buried in copycat spammers. The issues surrounding the use or misuse of Adsense are just as troubling for those who might be seeing little return on their adverting budget this month.

Do we "keep the faith" in Google in the face of these events? Most likely, yes. It is not so much whether they deserve that faith, but that most people will never know this happened. Days after the story broke there's still very little mention in the "mainstream" press. Some tech sites have mentioned it, but this isn't the kind of story that will end up on the evening news, mostly because the background knowledge required to understand it goes beyond what the average citizen is able to muster. The story will probably end up as an interesting footnote in that most esoteric and neoteric of worlds, "SEO History."

Digg del.icio.us Blink Stumble Spurl Reddit Netscape Furl

Article keywords: website hosting, google, spam, index, billions, pages, click, fraud

Article Source: http://www.articles2k.com

Mr. Lester served for 5 years as webmaster for ApolloHosting.com and previously worked in the IT industry an additional 5 years. Apollo Hosting provides website hosting, ecommerce hosting, vps hosting, and web design services to a wide range of customers.

Top Internet Marketing Articles
1). The Fly & The Window Of Opportunity By : Rick Davies Copyright 2006 1stPromotion.com One afternoon after going through a small backlog of emails, I was daydreaming about how it shouldn't be tough for absolutely anyone out there to make that first online sale and then to go on and make more and more sales. I feel bad for people that have paid money for some of the programs out there that promise to "stuff their wallets effortlessly" only to find that there is no free lunch.
2). Sign Our Guestbook – How Signing Guestbook’s Or Creating Your Own Helps Your Website By : Anton Cheranev Guestbook’s are some of the most popular areas on websites. Before a consumer does business with you, they are likely to check out your guestbook. Guestbook’s are somewhat like testimonials in that they offer real feedback on your site, your business, and your products. They are honest and are created from those that have had some sort of experience with your business or those who simply visit your site.
3). Where To Find A MySpace Friend Adder By : PaulG MySpace friend adders have become a great tool in maximizing the MySpace experience. Since MySpace has become popular, so has the importance of having tons of MySpace friends. This can benefit the causal MySpace user, the MySpace Music artist, and even the web publishers and marketers that use MySpace as a stepping stone toward success. Now MySpace friend adders have gone even further by adding other MySpace abilities to their programming.
4). How To Monetize A 1,000+ Page Site In 5 min. Or Less Without Violating Google's Adsense Policy! By : D Dean Hall As webmasters, writers and authors we all want to find ways to maximize the profitability of our content. We employ a variety of methods such as Adsense, Yahoo Ads, links to affiliate products of all types but there are a couple of problems. There is the sheer time involved in creating these links and then keeping them updated to the most relevant and profitable products.
5). 'Social Bookmarking' As An Aggressive and Acceptable Blog Marketing Tactic By : gregw Social bookmarking websites are becoming more and more popular. They allow you to save bookmarks online and Tag/Categorize them with keywords instead of saving them as bookmarks in the favorite's list of yourbrowser. This is particularly useful when your browser based bookmarks have become unwieldy. It's also help since you can access your bookmarks from any computer where you have an internet connection.
6). The Four Big Adsense Secrets By : Iszuddin Ismail Is there a BIG secret to the huge Adsense checks? Yes, you’ve seen those screen shots of 5-figure Adsense earning … so have I. And that made you thinking, why are some people earning those huge income, and you are not. Whatever people may say, I only see these are the only factors that make the difference between your earning and those ‘big-guns’. 1 - Adsense Placement Yes, putting your Adsense in a special way can be the difference between 1% click-through rate (CTR) and 10% CTR.
7). The Downsides Of Reciprocal Linking To Increase Search Engine Rankings By : Brian Gilley Reciprocal linking is still a highly sought after method for many companies and webmasters hoping to increase their web site’s ranking in the search engines. Back in 2002, when reciprocal linking firmly took its place as a real way to boost your rankings, many webmasters and sites started using it to quickly and frantically get the upper edge on their competition and the search engines.
8). One of the Quickest and Easiest Way to Dramatically Increase Your Website’s Page Rank By : Tim Wright This is one of the easiest and fastest ways that you can increase the number of one way links to your website and dominate your competition.
9). Spy on the Competition, Stay on Top By : Jay Chirino Just when you thought you had this whole online business thing mastered, it turns out that now you must become a secret super spy, break into your competitor’s databases and steal important documented information that you can put to use for your own benefit. So get your rope and mask ready because we’re going in! Oh, and by the way, I’m just kitting.
10). Let's Start! Create A Business Plan By : Helen Chow Let's Start! Create A Business Plan

New Internet Marketing Articles
1). 5 Tips for Making Mounds of Money on the Internet By : Michael Gulbin Make sure you are doing everything you can to make money on the web.
2). Are You Costing Yourself Money With This Common Article Marketing Mistake? By : Chris Ellington The Article Marketer editorial team reveals the worst mistake an internet marketer can make when submitting royalty free articles.
3). Learn To Market From A Child By : Clint Pollard A marketing lesson I learned from my grand daughter.
4). How To Maximize Your Profits From Backend Sales By : Jason Oickle The truth is with backend sales, you can gain additional or continuous profit without having to do any extra work.
5). Web Site Promotion Guides For Newbies By : Obinna Heche If you are new to the internet world you are not alone. There are a lot of people who are not too familiar with the internet but want to join in on the profits that are available.
6). Network Marketing for Overall Growth By : Sanjay Johari Network marketing provides tremendous opportunity to learn and earn. Those willing to learn have unlimited potential to earn and grow.
7). What Should You Know Before You Start A Home Business By : Arun Pal Singh As 2006 dawns on earth I am quite thoughtful. Many persons started their business in 2005. Some succeeded and some left disappointed.
8). Internet Marketing: Give Me Good Content By : Virginia Bola Have you ever considered how much boring, poorly written, superficial, and totally uninteresting stuff there is in those billions of web pages floating in the ether?
9). Internet Marketing: Give Me Something New By : Virginia Bola Give me freshness, variety, and a taste of greatness. I am totally frustrated by the lack of vitality found in the majority of websites.
10). Why Everyone Needs a Mentor. By : Gary Muller Why do you need a mentor and what could they possibly offer to help you grow your business? Discover why even successful marketers utilize the skills and expertise of an internet marketing mentor.