Crawl Budget: When It Matters, and How to Stop Wasting It

Crawl budget is the number of URLs Googlebot can and wants to crawl on your site over a period of time. Google describes it as a combination of two things.

• Crawl capacity limit: how much crawling your server can handle without problems. If your site responds quickly and reliably, the limit goes up. If it slows down or returns errors, Googlebot backs off.
• Crawl demand: how much Google wants to crawl your URLs, based on things like how popular pages are, how often they change, and how stale Google's copy is.

Put simply: capacity is how much Google can crawl, demand is how much it wants to. Your effective crawl budget is where those meet.

Here's the part most crawl budget articles bury. Google's own guidance says crawl budget mainly matters for very large sites, like those with around a million or more unique pages that change moderately often, or sites with tens of thousands of pages that change very frequently. Sites with a large share of URLs stuck as discovered but not indexed can also be affected.

If you run a 300-page business site and new pages get crawled within days, you don't have a crawl budget problem. You might have a quality, internal linking or indexing problem instead.

Signs you might have one

• New important pages take weeks to be crawled.
• Search Console shows a large, growing number of URLs as "Discovered, currently not indexed."
• Log files show Googlebot spending most of its requests on parameter URLs, filters or old junk.
• Updates to existing pages take a long time to appear in search.

Search Console Crawl Stats report

Found in Search Console settings, it shows total crawl requests, download size, average response time, and breakdowns by response code, file type, purpose and Googlebot type. Watch response time and server error trends especially.

Server log files

Logs are the ground truth. They show every request Googlebot made, to which URL, and what your server returned. Group requests by URL pattern and you'll see exactly where crawling goes. The log file analysis play walks through the process.

• Faceted navigation. Filter combinations can create near-infinite URLs. See the faceted navigation play.
• URL parameters. Sorting, tracking and session parameters create duplicate URLs of the same content.
• Infinite spaces. Calendars with endless next-month links, and internal search result pages.
• Soft 404s. Pages that return a 200 status but show "no results" or empty content.
• Redirect chains. Every hop is an extra request.
• Duplicate content. Printer versions, HTTP and HTTPS, trailing slash variants, uppercase variants. See the duplicate content play.
• Low-value pages at scale. Thin tag pages, auto-generated pages with little unique value.

1. 1Make the server fast and stable. Reduce response times and eliminate 5xx errors. Google crawls more when your site can handle it.
2. 2Block infinite and useless URL spaces in robots.txt. Filter combinations, internal search, sort parameters. Disallowed URLs aren't crawled.
3. 3Stop generating junk URLs. Fix the templates and links that create them, rather than only blocking them.
4. 4Return proper status codes. Real 404 or 410 for removed content, not soft 404s.
5. 5Flatten redirect chains to single hops, and update internal links to final URLs. See the redirects play.
6. 6Strengthen internal linking to important pages, so they're discovered and prioritized.
7. 7Keep XML sitemaps clean, listing only canonical, indexable URLs, with accurate lastmod dates.

• Noindex to save crawl budget. Google has to crawl a page to see the noindex tag, so it doesn't reduce crawling. It controls indexing, not crawling.
• Canonical tags to save crawl budget. Same issue. Canonicalized URLs still get crawled.
• Nofollow on internal links. An unreliable way to control crawling. Better to not link to junk URLs at all.
• Crawl-delay in robots.txt. Googlebot doesn't follow the crawl-delay directive.
• Blocking pages that contain noindex. If you block a URL in robots.txt, Google can't see its noindex tag, so the URL may stay indexed from links.

Noindex is a sign on the door. Googlebot still has to walk to the door to read it. Robots.txt stops the walk.Shmul

Crawl demand isn't something you control directly, but you can make it easier for Google to spend crawling where it matters.

Keep XML sitemaps honest

• List only canonical, indexable URLs that return 200.
• Split large sitemaps by type, like products, categories and articles, so you can see indexing per section.
• Remove URLs that redirect, 404 or carry noindex.

Use lastmod accurately

Google has said it can use the lastmod value when it's consistently accurate. Update it when content meaningfully changes, not every time the page is regenerated. A sitemap where every URL claims to have changed today teaches search engines to ignore the field.

Signal real freshness

Pages that genuinely change, such as prices, stock levels and new reviews, naturally attract more crawling. Pages that never change don't need frequent recrawls, and that's fine.

1. 1Confirm the problem with Crawl Stats, logs and indexing reports.
2. 2Map URL patterns and quantify where Googlebot requests go.
3. 3Classify patterns as valuable, duplicate or useless.
4. 4Fix server performance and errors.
5. 5Stop generating useless URLs, then block remaining infinite spaces.
6. 6Clean sitemaps and internal links to point at canonical URLs.
7. 7Re-measure crawl distribution after a few weeks and iterate.