1. Home
  2. Glossary
  3. Crawl budget
SEO

Crawl budget

The number of URLs a search engine is willing and able to crawl on a site in a given period. It mainly matters for very large sites. Most small sites do not need to manage it.

Why it matters

On a very large site, crawlers may not reach every URL quickly. Removing duplicates, errors and endless URL variations helps them spend time on pages that matter.

Where you will see it

The free XML sitemap validator looks at this on a live page and explains what it finds in plain English: find the sitemap, validate the XML and limits, and spot-check that listed URLs really load.

Related terms

  • XML sitemap: A file that lists the URLs you want search engines to know about, optionally with last-modified dates.
  • Redirect chain: Several redirects in a row before the final page, for example http to https to www.
  • Duplicate content: The same or very similar content available at more than one URL. It is not a penalty in itself, but search engines have to pick one version.

Go deeper

  • XML sitemaps: How an XML sitemap works, its format and limits, which URLs belong in it, the common mistakes that make search engines ignore it, and how to check yours.

← All 66 glossary terms

↑ ↓ to moveEnter to openEsc to closeAll results