Skip to content
Innopulse Consulting
SEO

What is crawl budget?

Short definition

Crawl budget is the extent to which a search engine fetches a site in a given period. It only becomes a bottleneck on large sites — and then usually not because of volume but because it is spent on worthless URLs.

Crawl budget is the extent to which a search engine fetches a site in a given period. It follows from two quantities: the load the server tolerates, and the search engine’s interest in that site’s content.

When it matters at all

For most sites crawl budget is not an issue. A site with a few hundred URLs is captured completely without anyone having to think about it. It becomes relevant on large corpora — many thousands of URLs — on frequently changing content, and anywhere URLs are generated automatically. Working on crawl budget for a small site means optimising a problem you do not have.

What actually wastes it

The bottleneck rarely arises from the number of good pages. It arises from URLs with no value of their own that are fetched anyway: parameter variants of the same page, endless filter combinations, internal search results, calendar views with unbounded futures, sort variants. Individually harmless, in aggregate they are why important pages get visited less often.

Redirect chains and error pages

Two further consumers are underestimated. Every redirect costs a fetch; a chain of several costs correspondingly more and is abandoned beyond a certain length. And URLs that permanently return errors, or sit in the sitemap without existing, keep getting fetched. Both are clean-up jobs with immediate effect.

What works and what does not

What works is not generating worthless URLs in the first place, or excluding them from fetching via robots.txt. Also effective are a clean sitemap without dead or de-indexed entries, short internal paths to important pages, and fast server responses. What does not work is using noindex to steer budget: a page with noindex is still fetched, it is merely not indexed. To save fetches you must prevent the fetch, not the indexing.

How often a page is visited also depends on how easily it is reached. A page with a single inbound link four clicks from the home page is fetched less often than a well-connected one. Internal linking is therefore not purely a user matter but acts directly on discovery — and is usually the more effective lever than any budget optimisation.

Practical consequence

The useful order is: first check whether a bottleneck exists at all; then eliminate worthless URLs rather than reducing good ones; then clear up redirect chains and error pages; and only then consider finer control. On most sites the work ends after the first step.

SEO is our specialty

Innopulse doesn't just explain terms — we put them into practice for DACH companies.