RankforBot
Rankfor.AI Web Analysis Crawler
What is RankforBot?
RankforBot is the official web crawler for Rankfor.AI, an AI Territory Analysis platform. Our bot helps businesses understand how AI systems perceive and represent their content across the web.
RankforBot analyzes publicly accessible web content to provide insights into content territories, competitive positioning, and AI visibility optimization opportunities.
Technical Details
User Agent String:
Crawl Rate:
We employ adaptive crawl rates to minimize load on your servers. We respect standard crawl-delay directives in robots.txt.
What We Analyze
-
Structured Data JSON-LD, Schema.org markup, Open Graph tags, and other machine-readable metadata that helps AI systems understand your content.
-
Content Territories Topics, keywords, and semantic clusters that define your content landscape and competitive positioning.
-
Navigation & Site Structure How your content is organized and linked, which influences how AI systems prioritize and reference your pages.
-
LLM-Specific Files We check for llm.txt and llms.txt files that provide AI assistants with context about your brand and services.
Privacy & Data Usage
We only analyze publicly accessible content. RankforBot does not:
- Access password-protected or private areas
- Submit forms or POST data
- Execute JavaScript or interact with dynamic content beyond initial page load
- Store personal user data or PII (Personally Identifiable Information)
- Share your content with third parties
Analyzed data is used solely to provide AI Brand DNA insights to our platform users. We respect your content ownership and comply with all applicable data protection regulations.
How to Control RankforBot
To block RankforBot entirely:
Disallow: /
To allow access to specific sections only:
Allow: /blog/
Disallow: /
To set a custom crawl delay:
Crawl-delay: 10
Contact & Support
If you have questions about RankforBot, need to report an issue, or want to request specific crawl behavior for your domain, please contact us: