Skip to content

mounicmadiraju/robot.txt-changes

BranchesTags

Repository files navigation

robot.txt-changes

C# code to check changes in robot.txt file with 3 methods

About

💧 Test your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS and JavaScript) are disallowed!. Robots.txt files help you guide how search engines crawl your site, and can be an integral part of your SEO strategy.

Topics

monitoring robots-txt robots-parser

Resources

Readme

License

Apache-2.0 license
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages