The Robots.txt file should be included in your website's directory. This file helps the search engine to find out about website accessibility and crawl. You can also block search engines from viewing certain pages of your site if you wish. Robot.txt generator helps you create this file. Robots.txt Generator

Search engine optimization plays a major role in increasing the visibility of your website. The methods given above are essential and essential to any website. There will be other ways to make good use of it. The key point, however, is that any optimization process will be successful if done with the user in mind, not just the search engines. 

Search engines index millions of websites to search results that have returned keywords. They do this using "spiders".

Many search engines have their own spiders crawling through the web looking for web pages. Spiders also know as “robots” because they are simply small programs that automatically work, searching for web pages and then navigating through embedded text links to identify them.
Many robots look for a robots.txt file in a high-quality index of your website, also known as “root” where your home page is located on a web server. Robots.txt Generator
A robots.txt file is a simple text file creat in a basic text editor, such as Notepad. Allows you to control what a spider is allowed to access and what is not allowed to access or index.
The basic robots.txt file format is very simple:
User-Agent: [Spider Name]
Do not allow: [Filename]
For example, to allow ALL robots complete access to your website, your robots.txt file will look like this:
User-Agent: *
Disallow:
The asterisk is a "wild card" character that represents ALL robots. Leaving the Disallow line blank shows robots, that nothing on the site prohibits.
The following example blocks all robots from the cgi-bin (where your text is found), image guides, and portfolio directions:
User Agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /portfolio/

Note: You must use a different Do Not Allow individual directory or individual file.
In this example, you may be wondering why you do not want to allow the robot to point to your portfolio index. Robots.txt Generator

Not many webmasters spend time using the robots.txt file on their website. For search engine spiders that use robots.txt to see which clues to search for, a robots.txt file can be very helpful in keeping spiders indexing your actual pages and not other information, such as looking at your stats!

The robots.txt file helps keep your spiders out of access to component folders and files in your hosting directory that is not directly related to your web content. You can choose to keep spiders out of areas that contain search engine optimization programs and keep them out of the web statistics of your site.

Many search engines are unable to view dynamic content, especially those made in programming languages, such as PHP or ASP. If you have an online store set up on your hosting account, and it is in a different location, it is best to block the spiders in this guide so it only gets relevant information. Robots.txt Generator

The robots.txt Generator file should place in the directory where your key hosting files are found. You will therefore be advised to create a blank text file, save it as robots.txt, and upload it to your web host in the same directory your index.htm file is located on. 

more tools here XML SITEMAP GENERATOR