What is Crawl Budget?
TL;DR
Crawl budget is the number of pages a search engine or AI crawler will crawl and process on your website within a given timeframe.
Definition
Crawl budget is the number of pages a search engine or AI crawler will crawl and process on your website within a given timeframe. It is determined by the crawler's capacity (crawl rate limit) and the site's demand (crawl demand).
Why It Matters
If your crawl budget is wasted on low-value pages, your important content may not be crawled and indexed efficiently. Optimizing crawl budget ensures search engines and AI models discover and understand your most important content.
Examples
- A site with 10,000 auto-generated tag pages wasting crawl budget that should go to 50 high-value service pages
- Proper use of robots.txt and canonical tags to direct crawlers to priority content
Frequently Asked Questions
Do small businesses need to worry about crawl budget?
Crawl budget is primarily a concern for large sites (10,000+ pages). Small businesses should focus on site structure, internal linking, and ensuring important pages are crawlable - crawl budget is rarely a bottleneck for sites under 1,000 pages.
Related Terms
