Robots.txt is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.
How to Edit WordPress Robots.txt File
1.First of all Install WP Robots Txt WordPress Plugin.
2.Now go to your WordPress Admin Panel >> Settings >> Reading >> Robots.txt Content.
3.Here you can simply edit your Robots.txt File.
4.When you edit your file successfully than simply click on “Save Changes” Button and you are done.