SEO Tutorials | Tips & Tricks | Troubleshooting

Robots Exclusion Protocol

By on Aug 1, 2011 in Dreamweaver, SEO, XHTML & CSS | 1 comment

Preventing search engines from indexing – pages… or an entire site Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to visit a URL, say: Before it does so, it firsts checks for...

Read More