Blog
Crawl budget: when it is worth investigating
Crawl budget describes the set of URLs Google can and wants to crawl on a site. It combines crawl capacity, related to resources and server responses, with demand, related to content Google considers worth visiting or updating. It is not a page allowance you can manually allocate.
Google’s crawl-budget guide primarily addresses large sites or rapidly changing content. If a site does not fit that profile and pages are crawled normally, maintaining its sitemap and checking indexing may be adequate. A missing search result does not, by itself, establish a budget shortage.
Two sites needing different investigations
These cases are fictional. The first is a small class website; the second is a catalog exposing many filter combinations. Their symptoms sound similar when described as “Google does not show new pages”, but the evidence changes the decision.
| Scenario observation | What it establishes | Next task |
|---|---|---|
| A class page is reported as crawled but not indexed | Google obtained the page at the recorded time | Review content, duplication and indexing signals; do not attribute that state to a lack of crawling |
| Many filter URLs receive requests while the server responds slowly or with errors | There may be an efficiency or availability problem | Review logs, URL families and capacity before assigning a cause |
| The new page was never linked from the catalog | An intended discovery route is missing | Add a relevant link and check access |
In the first row, removing a hundred sitemap entries does not explain why an already fetched page was not selected. In the second, declaring every filter useless is inadequate too: some collections may serve legitimate searches. Separate URL families and their consequences.
What to request from the website team
Ask for logs containing dates, URLs, statuses and response times, together with verification that requests attributed to Googlebot are authentic. A user-agent string containing its name is insufficient. Google’s crawler-verification documentation describes available methods.
Group requests into actionable families: current products, filters, redirects and errors. Look for alignment with template changes, URL growth or server incidents. One daily total does not identify which cause is involved.
Search Console supplies crawl statistics and inspection information with different scopes. Server logs allow investigation of requests for specific paths. For discovery, crawling and indexing, start with the distinction between those stages.
Which changes make sense after gathering evidence
If repeated combinations appear that do not need search visibility, review their generation and faceted navigation. If the server fails, repair availability and observe its responses again. If the page never received a link, address that access before starting a broad crawl-budget project.
Google cautions that blocking or hiding pages from crawling does not automatically shift resources elsewhere when the site is not hitting its serving limits. Making a low-value page faster does not force the crawler to visit it more often either.
A useful report ends with a supported cause or a bounded hypothesis and a test that could confirm it. For the small website, the decision may be not to start a crawl-budget project. For the catalog, it may be to correct one filter family and observe whether requests and errors change, without promising an indexed-page count.
Plan your next post in HeyMark.
Keep the idea, review the draft with your team, and see how it performed in the accounts you connected.