How to keep robots out of your web site - Search Engine Optimization Article

By:Dr. Bonomi

THE ROBOTS.TXT FILE

You know that search engines have been created to help people find information quickly on the Internet, and the search engines acquire much of their information through robots (also known as spiders or crawlers), that look for web pages for them.

The spiders or crawlers robots explore the web looking for and recording all kinds of information. They usually start with URL submitted by users, or from links they find on the web sites, the sitemap files or the top level of a site.

Once the robot accesses the home page then recursively accesses all pages linked from that page. But the robot can also check out all the pages that can find on a particular server.

After the robot finds a web page it works indexing the title, the keywords, the text, etc. But sometimes you might want to prevent search engines from indexing some of your web pages like news postings, and specially marked web pages (in example: affiliate´s pages), but whether individual robots comply to these conventions is pure voluntary.

ROBOTS EXCLUSION PROTOCOL

So if you want robots to keep out from some of your web pages, you can ask robots to ignore the web pages that you don´t want indexed, and to do that you can place a robots.txt file on the local root server of your web site.

In example if you have a directory called e-books and you want to ask robots to keep out of it, your robots.txt file should read:

User-agent: * Disallow: e-books/

When you don´t have enough control over your server to set up a robots.txt file, you can try adding a META tag to the head section of any HTML document.

In example, a tag like the following tells robots not to index and not to follow links on a particular page:

meta name="ROBOTS" content="NOINDEX, NOFOLLOW"

Support for the META tag among robots is not so frequent as the Robots Exclusion Protocol, but most of major web indexes currently support it.

NEWS POSTINGS

If you want to keep the search engines out of your news postings, you can create an an "X-no-archive" line in of your postings' headers:

X-no-archive: yes

But although common news clients, allow you to add an X-no-archive line to the headers of your news postings, some of them don´t permit you to do so.

The problem is that most search engines assume that all information they find is public unless marked otherwise.

So be careful because though the robot and archive exclusion standards may help keep your material out of major search engines there are some others that respect no such rules.

If you're highly concerned about the privacy of your e-mail and Usenet postings, you must use some anonymous remailers and PGP. You can read about it here:

http://www.well.com/user/abacard/remail.html http://www.io.com/~combs/htmls/crypto.html
http://world.std.com/~franl/pgp/

Even if you are not particularly concerned about privacy, remember that anything you write will be indexed and archived somewhere for eternity, so use the robots.txt file as much as you need it.

Written by Dr. Roberto A. Bonomi

Digg del.icio.us Blink Stumble Spurl Reddit Netscape Furl

Article keywords: robots, robots.txt, robots exclusion protocol, marketing internet marketing, home business

Article Source: http://www.articles2k.com

Dr. Roberto Bonomi is a successful e-book writer that shares his home business experience at: www.easy-home-business.com If you already have, or are looking for an Internet Home Business, you can't miss the free knowledge that you'll receive at his site, and you can post free your own articles at articles.drbonomi.com

Top Search Engine Optimization Articles
1). High Paying Lateral Keywords By : Matthew C. Keegan If you run AdSense on your site you know that some words pay more than others, much more in fact. More than likely you have also learned that terms like "structured settlements" and "mesothelioma" can produce incredibly high PPC revenue, if they show up on your site at all. Unfortunately hundreds of thousands of other webmasters are "on" to this practice judging by the number of sites created regularly to capitalize on the phenomenom.
2). Creating Sitemaps For Google, MSN AND Yahoo! - The Easy Way By : Philip Nicosia If you own or maintain a website or intend to own one, wouldn’t it be great if you get frequent visitors who find satisfaction in getting exactly the information they need from your page? While that satisfaction largely depends on the contents of your website, how you get to be accessed by website users is the most critical factor of website development.
3). How To Stand A Chance In The Search Engine Optimization Maze By : bluecharm Despite all the other methods available for obtaining a decent web ranking for your website, for instance, link exchanges, search engine optimization or SEO is still regarded as the way to go to ensure that your website is listed high in the search engine rankings. Search engine optimization is a method of analyzing your site and modifying it to allow search engines to read and index it.
4). Link Exchange and Search Engine Optimization By : Bill Boyd Here’s an overview of how exchanging links can improve our search engine ranking. I believe Exchanging Links is one of, if not “the” best way to market your site online! Read below to see why. Some people believe that in order to be successful online they have to direct thousands of targeted visitors to their site. TRUE They also believe that they have to spend all their time adjusting the scripts and contents of their pages, to make the search engines spider them and rank them well.
5). Black Hat SEO – What Never To Do Or Get Banned By : While there are many legitimate skills in SEO there are also those that can work but if you get caught using them the results can be disastrous. When search engine optimization became an issue many techniques were employed because at that point the search engines used a much simpler algorithm. As these tricks were used to exploit the simpler algorithms they were also served to make them more advanced.
6). An SEO Glossary - Common SEO Terms Defined By : Glenn Murray Search Engine Optimization (SEO) has become an essential weapon in the arsenal of every online business. Unfortunately, for most business owners and marketing managers (and even many webmasters), it's also somewhat of an enigma. This is partly due to the fact that it's such a new and rapidly changing field, and partly due to the fact that SEO practitioners tend to speak in a language all of their own which, without translation, is virtually impenetrable to the layperson.
7). The Best Web Marketing Secrets By : Allen Brown Are you stuck up with lower down the sales? Not improving or upgrading your sale to higher scale? Do not worry! You are at the right place to get the Right Solution. We have magic wand to boost your sell. Your profit and sales will surely increases to your wish as you apply it. Do not worry you may be marketing in any form, i.e. may be through web, or by printed form or in personal way.
8). Organic SEO And Link Building By : John Tourloukis Organic SEO or search engine optimization is a slow and steady process. Achieving top rankings takes time and link building is a big part of that effort. If you are to have a successful web site and obtain high search engine rankings you will need quality relevant web sites linking to your web site. The amount and quality of links you need will depend on the competitiveness of the keyword phrases for which you are optimizing.
9). Keyword Specific Ads Equals Higher Search Engine Rankings By : Joe Money Keywords are an essential component of producing a web site. These are the words which the search engines use to help categorize and rank your pages. For instance, if your website is about 'writing articles' then your keywords could include 'writing', 'articles', 'article writing' and so on. The advice from many web designers and search engine optimizers has been to find as many keywords as you can.
10). Is Your Website Invisible? The Google Sandbox Solution By : Brian Carter What is the Google Sandbox? How Do You Know if You’re Stuck in the Sandbox? Does the Sandbox Really Exist, or is it Just the Google Algorithm? Why Do You Get Sandboxed? Is There a Way to Trick the Sandbox Filter?

New Search Engine Optimization Articles
1). The SEO Benefits Of Interlinking All Your Web Pages By : James Woolley In this article I discuss how to fully optimise the internal linking structure of your website to achieve a greater pagerank and higher search engine rankings.
2). The Facts on Google Page Rank By : Joyce Jackson Learn what Page Rank really is and some of the techniques you can use on your web site to get your ranking up on Google.
3). Failing with Google Adwords Can Help You Master Internet Advertising By : Cliff Posey This article explains how failing at adwords can still result in high search engine placement, successful Internet advertising, and high search engine ranking
4). Top SEO Advice: Forget About Fooling Google By : Joyce Jackson Learn some tips on why trying to fool Google with your site SEO means disaster, not a faster way to the top of the search engines.
5). The Benefits of SEO By : Robby SEO stand for Search Engine Optimization. It's a fancy term that means a website is user friendly to search engines bots. When you use SEO on a site you fix the site's meta tags, title, discription, and even the body of the index page, among other things to give the bots the best possible idea what your page is about and how relevant the content is to everything else on the page.
6). The Future of Search Engine Rankings By : Christian Taylor How do you plan for future changes in search engine rankings? It would be great to have a crystal ball, prying into the plans of the search engines and how they may change weights of algorithms and the like. For those of little SE understanding, algorithms are formulas or rules set up by SE's to determine ranking. Each rule then has a different weight or percentage of importance assigned to it.
7). The Three Basic Keys of Search Engine Optimisation By : David Touri With search engine optimisation on the rise, there are many mind blowing theories that are circulating about what will get you high up in the search engines. From having your keywords in bold font, to having your text as close to the top of a page, along with the famous 3% - 7% keyword density rule, we can scrap them all since it comes down to three basic key points.
8). Striking out on your own: Do you really need a sitemap generator? By : Philip Nicosia There are so many responsibilities that are associated with starting a website. Each step towards the realization of a website requires a lot of hard work and utmost dedication to achieve results and to pay attention to detail. Only by doing this can one ensure that a website is set up successfully and able to meet the goals set for its creation. Aside.
9). Good And Bad SEO Practices By : Andreas Obermueller Search Engine Optimizing is a set of practices developed to help you get a high traffic and PageRank, while simultaneously building a useful and functional website that will respond to the needs of your potential users. While some website owners are getting their site indexed by following and enhancing these guidelines, others are looking for a way to sneak in and crawl up search engines by developing various strategies to cheat spiders.
10). The Elusive Practice of Search Engine Optimization By : Jay Stockman Search engine technology has completely revolutionized the way we retrieve relevant information. Mastering the algorithms that drive these marvels of technology is the challenge, and assures the webmaster complete marketing success. In 1990, Alan Emtage at McGill University developed the first search engine, named Archie. Its purpose was to create a database of web filenames that could be queried, and retrieved by any user.