Glossary · Crawl Budget
Crawl Budget
The number of pages a search engine will crawl on a site within a given period — a constraint for very large sites.
Crawl budget matters for sites with thousands of pages. If a search engine bot only crawls 500 pages per day on a 10,000-page site, deep pages may take weeks to be re-crawled after updates.
Optimizations that improve effective crawl budget: faster server response, fewer redirect chains, removing thin or duplicate pages, strong internal linking that surfaces deep pages, and accurate sitemaps.