SEO & GEO glossary

Crawl budget

Crawl budget is the number of URLs a search engine can and wants to crawl on your site in a given period. Google describes it as the combination of crawl capacity, how much crawling your server can handle, and crawl demand, how much Google wants to crawl.

Also known as: Crawl capacity, Crawl demand

Why it matters

For most sites, crawl budget is not a concern. Google’s crawl budget guide (opens in a new tab) is aimed at very large sites, with a million or more pages that change weekly, medium sites of 10,000 or more pages that change daily, and sites with many URLs listed as “Discovered – currently not indexed” in Search Console.

Budget is wasted on duplicate and low-value URLs: faceted filters, session IDs, infinite calendars, long redirect chains and soft 404 pages. Consolidating duplicates, keeping crawlers out of pure parameter spaces, keeping sitemaps clean and keeping the server fast all let crawlers spend their time on pages that matter.

Example

An online fashion store sells 20,000 products but exposes 2 million crawlable URLs, because every combination of size, colour and sort order creates a new address. Crawlers spend their visits on filter pages while new products wait to be found. Blocking the filter parameters frees crawling for the product pages.

Sources

Put the theory to work on your own site

DraftSEO is an AI SEO writer that researches keywords, writes SEO- and GEO-optimised articles with images, internal links and FAQs, and auto-publishes them to your website on a schedule.

Start for free