A Robot in Search engine optimization services terminology
is a search engine software program which visits a page on
a website and follows all the links of the website from that
page and indexes some or all of the pages in the website.
Everyday search engines receive hundreds of new website submissions.
It is quite cumbersome and time consuming for a human to review
the whole of the website and judge whether that particular
website meets the search engine optimization standards and
index the same. Here is where our friend robot comes into
picture. Robots are highly intelligent software programs which
crawl the entire website, checking the relevancy, consistency
and significance of the site thereby effectively indexing
the website into the search engine database reducing the amount
of time consumed per site. In this way a robot can quickly
index more sites per day. Though a robot is not a very critical
aspect of search engine optimization services technology,
it is advisable to include it.
Normally, a robot visits the home page of the site and follows
the links present in the page, scanning each link and page.
Sometimes we do not prefer a robot to index a particular page(s)
in our site. For instance, you might want a series of pages
to be viewed in sequence and would like to index only the
page one. To achieve this, we have a special kind of Meta
tag known as Robots Meta tag. The robots Meta tag is similar
to other Meta tags and is placed in the head of the document.
This tag dictates the robot which pages to be indexed, which
pages not be indexed, which links should be followed and which
links should not be followed.
A typical meta robots tag would resemble as follows,
<meta name="robots" content ="index, nofollow">
The meta tag describes us that the robot should index the
page visited but to not follow the links in the page.
The other most and significant part of controlling robots
is the robots.txt file. The robots.txt file is used primarily
to control areas or portions of the website by excluding those
portions being visited by the robot. Whenever a robot visits
a site, it first checks the robots.txt file.
The robots.txt file,
- is a text file created in notepad or any text editor
- should be placed in the top level directory or root of
the website or server space
- should include all lower case letters
Through the robots.txt file we can,
- Exclude all robots from visiting the server
- Allow complete access to all robots
- Exclude robots from accessing a portion of the server
- Exclude a specific robot
- Exclude certain type of files from accessing by specifying
the file extensions.
Finally, we can conclude that the robots.txt file basically
acts as a filter thereby providing total control over the
search engine robot.
While talking of robots I would like to mention the revisit
tag. This tag is an important tag in the realm of search engine
optimization services technology. The revisit tag tells the
search engine the time duration after which it should visit
your site again. If you change your site's contents frequently
then your revisit time should be say a week else it can be
higher. Your search engine rankings dip if the search engine
visits you a second time and finds that the content has not
been altered significantly. Though not all search engines
honor the Revisit tag, it is advisable to include the tag.
If you are keen on top search engine rankings use the revisit