Let’s begin with a bit of history; Search Engine Optimization initially started its debut on the Internet during the 1990’s, the term “Search Engine Optimization” was introduced by an Internet Marketing Group during the year of 1997 in an effort to define the steps needed to improve Search Engine Positioning. The term, refers to the practice of optimizing content in Website pages in an effort to gain improved positioning in the Search Engines by conforming to the preferences of the major Search Engines, namely Google and Bing.
SEO, Search Engines and the Indexing Website Pages.
Who remembers the days of the card catalog at your friendly neighborhood Library when the only way to find information, whether it be a book or magazine, was to manually scroll through thousands of index cards, which incidentally were manually created and placed inside wooden index card filing type cabinets. Search Engines operate the same way, well almost, they index all of the webpage’s on the web, the only difference is, the days of the manual card catalogs have disappeared, and nowadays everything is indexed in digital format. The process is a fairly simple one, performed by a computer program (namely a robot or simply a bot), which is a program, typically known as a web crawler or spider, it’s main purpose is to crawl the Internet for website pages (new or modified), the website crawler program typically copies the website pages for later indexing by the Search Engines to add into the Search Engines Library of Website Pages for faster and improved searching by the end user.
Robots and Spiders are crawling around my website, what should I do, should I be concerned?
How can you insure website pages are being crawled by a Web Bot or Spider for the Search Engines, or how can you prevent your files from being crawled by Web bot. Which brings up the question why would anyone want to hide folders or files from a Web Bot or Search Engine? There are several reasons; one would be to exclude files that include images, as most images do not provide details regarding the content found in a website since their file names are typically identifiable only by numbers or letter, (i.e. 23405.png), which would not provide any relevant information to the Search Engine regarding the content in the Website. To insure a Website is crawled, it may be beneficial to include a file called Robots.txt, (this file is typically placed inside the root folder of the Website, notice the “bot” J), which is a file specifically used to communicate with the Web Bot Crawlers to provide instructions as to which files/folders should be crawled and which files/folders should be excluded.
