Most search engines like Google, MSN and Yahoo rely heavily on the use of robots to search the internet and index (or include) new website/blog pages. Every time you make a change to your website or blog and publish it online, your site will eventually be visited by a robot, which is an automated program designed to search your site and index it accordingly in the search engine.

Robots search out keywords in each web page and also assess the page rank and popularity of the site. Robots, like any other automated program are designed to work with stringent parameters and there are ways in which we as bloggers and website owners can help them read our site pages more easily. Many of these revolve around ensuring that we have the following meta tags in place on each page: Title, Keywords, Description.

In addition we can also upload a file called robots.txt and register it with google. This will help your site to be visited more reglaly by the Google Robots and thus get your pages indexed more quickly.

robots.txt files sound complicated but they are not here's our step by step guide;

1. Open notepad on your pc and cut and paste the follow text into it exactly as it is shown below:

User-agent: *
Disallow:
sitemap: http://thewebsiteurlofyoursitemap.xml

2. Save this file as robots.txt (all lower case)

3. Upload this file to the highest level in your web hosting server, usually called htdocs, but these can vary.

4. Look up google webmaster tools in google and create an account. (It's free and easy to do so) Then register the website address of your robots file with google (they take you through it step by step).

This simple robots file allow robots to visit any part of your site and points the robot to a copy of your site map which will help to get your website searched more ferquently by google robots.