How search engines work: Crawling + Indexing - Search Engine Optimization Article

By:temp

A search engine operates, in the following order: 1) Crawling; 2) Deep Crawling Depth-first search (DFS); 3) Fresh Crawling Breadth-first search (BFS); 4) Indexing; 5) Searching.

Web search engines work by storing information about a large number of web pages, which they retrieve from the WWW itself. These pages are retrieved by a web crawler (also known as a spider) — an automated web browser which follows every link it sees, exclusions can be made by the use of robots.txt. The contents of each page are then analyzed to determine how it should be indexed. Data about web pages is stored in an index database for use in later queries. Some search engines, such as Google, store all or part of the source page (referred to as a cache) as well as information about the web pages, whereas some store every word of every page it finds, such as AltaVista. This cached page always holds the actual search text since it is the one that was actually indexed, so it can be very useful when the content of the current page has been updated and the search terms are no longer in it. This problem might be considered to be a mild form of linkrot, and Google's handling of it increases usability by satisfying user expectations that the search terms will be on the returned web page. This satisfies the principle of least astonishment since the user normally expects the search terms to be on the returned pages. Increased search relevance makes these cached pages very useful, even beyond the fact that they may contain data that may no longer be available elsewhere.

When a user comes to the search engine and makes a query, typically by giving keywords, the engine looks up the index and provides a listing of best-matching web pages according to its criteria, usually with a short summary containing the document's title and sometimes parts of the text. Most search engines support the use of the boolean terms AND, OR and NOT to further specify the search query. An advanced feature is proximity search, which allows you to define the distance between keywords.

The usefulness of a search engine depends on the relevance of the results it gives back. While there may be millions of Web pages that include a particular word or phrase, some pages may be more relevant, popular, or authoritative than others. Most search engines employ methods to rank the results to provide the "best" results first. How a search engine decides which pages are the best matches, and what order the results should be shown in, varies widely from one engine to another. The methods also change over time as Internet usage changes and new techniques evolve.

Most web search engines are commercial ventures supported by advertising revenue and, as a result, some employ the controversial practice of allowing advertisers to pay money to have their listings ranked higher in search results.

The vast majority of search engines are run by private companies using proprietary algorithms and closed databases, the most popular currently being Google, MSN Search, and Yahoo! Search. However, Open source search engine technology does exist, such as ht://Dig, Nutch, Senas, Egothor, OpenFTS, DataparkSearch and many others.

-----------------------------------------------------------

Digg del.icio.us Blink Stumble Spurl Reddit Netscape Furl

Article keywords: search engine, internet, technology, science history

Article Source: http://www.articles2k.com

David is the owner of Article Post Robot, www.articlepostrobot.com, the software which can post articles to hundreds of article sites and mail lists automatically. Demo is available upon request at help(at)articlepostrobot.com

Top Search Engine Optimization Articles
1). High Paying Lateral Keywords By : Matthew C. Keegan If you run AdSense on your site you know that some words pay more than others, much more in fact. More than likely you have also learned that terms like "structured settlements" and "mesothelioma" can produce incredibly high PPC revenue, if they show up on your site at all. Unfortunately hundreds of thousands of other webmasters are "on" to this practice judging by the number of sites created regularly to capitalize on the phenomenom.
2). Creating Sitemaps For Google, MSN AND Yahoo! - The Easy Way By : Philip Nicosia If you own or maintain a website or intend to own one, wouldn’t it be great if you get frequent visitors who find satisfaction in getting exactly the information they need from your page? While that satisfaction largely depends on the contents of your website, how you get to be accessed by website users is the most critical factor of website development.
3). How To Stand A Chance In The Search Engine Optimization Maze By : bluecharm Despite all the other methods available for obtaining a decent web ranking for your website, for instance, link exchanges, search engine optimization or SEO is still regarded as the way to go to ensure that your website is listed high in the search engine rankings. Search engine optimization is a method of analyzing your site and modifying it to allow search engines to read and index it.
4). Link Exchange and Search Engine Optimization By : Bill Boyd Here’s an overview of how exchanging links can improve our search engine ranking. I believe Exchanging Links is one of, if not “the” best way to market your site online! Read below to see why. Some people believe that in order to be successful online they have to direct thousands of targeted visitors to their site. TRUE They also believe that they have to spend all their time adjusting the scripts and contents of their pages, to make the search engines spider them and rank them well.
5). Black Hat SEO – What Never To Do Or Get Banned By : While there are many legitimate skills in SEO there are also those that can work but if you get caught using them the results can be disastrous. When search engine optimization became an issue many techniques were employed because at that point the search engines used a much simpler algorithm. As these tricks were used to exploit the simpler algorithms they were also served to make them more advanced.
6). An SEO Glossary - Common SEO Terms Defined By : Glenn Murray Search Engine Optimization (SEO) has become an essential weapon in the arsenal of every online business. Unfortunately, for most business owners and marketing managers (and even many webmasters), it's also somewhat of an enigma. This is partly due to the fact that it's such a new and rapidly changing field, and partly due to the fact that SEO practitioners tend to speak in a language all of their own which, without translation, is virtually impenetrable to the layperson.
7). The Best Web Marketing Secrets By : Allen Brown Are you stuck up with lower down the sales? Not improving or upgrading your sale to higher scale? Do not worry! You are at the right place to get the Right Solution. We have magic wand to boost your sell. Your profit and sales will surely increases to your wish as you apply it. Do not worry you may be marketing in any form, i.e. may be through web, or by printed form or in personal way.
8). Organic SEO And Link Building By : John Tourloukis Organic SEO or search engine optimization is a slow and steady process. Achieving top rankings takes time and link building is a big part of that effort. If you are to have a successful web site and obtain high search engine rankings you will need quality relevant web sites linking to your web site. The amount and quality of links you need will depend on the competitiveness of the keyword phrases for which you are optimizing.
9). Keyword Specific Ads Equals Higher Search Engine Rankings By : Joe Money Keywords are an essential component of producing a web site. These are the words which the search engines use to help categorize and rank your pages. For instance, if your website is about 'writing articles' then your keywords could include 'writing', 'articles', 'article writing' and so on. The advice from many web designers and search engine optimizers has been to find as many keywords as you can.
10). Is Your Website Invisible? The Google Sandbox Solution By : Brian Carter What is the Google Sandbox? How Do You Know if You’re Stuck in the Sandbox? Does the Sandbox Really Exist, or is it Just the Google Algorithm? Why Do You Get Sandboxed? Is There a Way to Trick the Sandbox Filter?

New Search Engine Optimization Articles
1). The SEO Benefits Of Interlinking All Your Web Pages By : James Woolley In this article I discuss how to fully optimise the internal linking structure of your website to achieve a greater pagerank and higher search engine rankings.
2). The Facts on Google Page Rank By : Joyce Jackson Learn what Page Rank really is and some of the techniques you can use on your web site to get your ranking up on Google.
3). Failing with Google Adwords Can Help You Master Internet Advertising By : Cliff Posey This article explains how failing at adwords can still result in high search engine placement, successful Internet advertising, and high search engine ranking
4). Top SEO Advice: Forget About Fooling Google By : Joyce Jackson Learn some tips on why trying to fool Google with your site SEO means disaster, not a faster way to the top of the search engines.
5). The Benefits of SEO By : Robby SEO stand for Search Engine Optimization. It's a fancy term that means a website is user friendly to search engines bots. When you use SEO on a site you fix the site's meta tags, title, discription, and even the body of the index page, among other things to give the bots the best possible idea what your page is about and how relevant the content is to everything else on the page.
6). The Future of Search Engine Rankings By : Christian Taylor How do you plan for future changes in search engine rankings? It would be great to have a crystal ball, prying into the plans of the search engines and how they may change weights of algorithms and the like. For those of little SE understanding, algorithms are formulas or rules set up by SE's to determine ranking. Each rule then has a different weight or percentage of importance assigned to it.
7). The Three Basic Keys of Search Engine Optimisation By : David Touri With search engine optimisation on the rise, there are many mind blowing theories that are circulating about what will get you high up in the search engines. From having your keywords in bold font, to having your text as close to the top of a page, along with the famous 3% - 7% keyword density rule, we can scrap them all since it comes down to three basic key points.
8). Striking out on your own: Do you really need a sitemap generator? By : Philip Nicosia There are so many responsibilities that are associated with starting a website. Each step towards the realization of a website requires a lot of hard work and utmost dedication to achieve results and to pay attention to detail. Only by doing this can one ensure that a website is set up successfully and able to meet the goals set for its creation. Aside.
9). Good And Bad SEO Practices By : Andreas Obermueller Search Engine Optimizing is a set of practices developed to help you get a high traffic and PageRank, while simultaneously building a useful and functional website that will respond to the needs of your potential users. While some website owners are getting their site indexed by following and enhancing these guidelines, others are looking for a way to sneak in and crawl up search engines by developing various strategies to cheat spiders.
10). The Elusive Practice of Search Engine Optimization By : Jay Stockman Search engine technology has completely revolutionized the way we retrieve relevant information. Mastering the algorithms that drive these marvels of technology is the challenge, and assures the webmaster complete marketing success. In 1990, Alan Emtage at McGill University developed the first search engine, named Archie. Its purpose was to create a database of web filenames that could be queried, and retrieved by any user.