What is the robots.txt File
If you have spent any time browsing through folders on your webserver you will likely have come across a file called robots.txt.
This video covers the basics of what the file is, how to properly create one, and what the alternatives are.
Code GIST: https://gist.github.com/prof3ssorSt3v...
