What bot traffic costs you, and when to block bots

Bots were 51 per cent of web traffic in 2024, and a share of them now reach the AI features organisations have put on their websites, where every request has a price attached. The costs land in four separate places, and only one of them appears on an invoice. 

The starting point is not blocking anything. It is finding out what is actually reaching your site, because the analytics most teams rely on count a substantial amount of automation as people.

How much of my website traffic is automated?

Slightly more than half, on the most recent broad measurement. Imperva’s 2025 Bad Bot Report, published by Thales, found that automated traffic accounted for 51 per cent of all web traffic in 2024, the first time in a decade it had passed human activity.

The breakdown is the more useful figure. The report puts the 2024 profile at 49 per cent human, 37 per cent bad bot and 14 per cent good bot. Bad bot traffic rose from 32 per cent in 2023, the sixth consecutive annual increase.

Your own site will differ, sometimes dramatically. A site that ranks well for commercially valuable terms attracts more scraping than one that does not, and a site with an exposed API attracts more still. The sector figure tells you the question is worth asking, not what your answer is.

What is ClaudeBot, GPTBot, or the crawler in my logs?

ClaudeBot, GPTBot and their equivalents are the user agents AI companies use to fetch web pages, and they do two quite different jobs that are worth separating.

Some crawling gathers content for model training. Some fetches a page in real time because a user has asked a question and the assistant is retrieving a source to answer from and cite. The first takes your content and returns nothing directly. The second is closer to a search engine referral, and for most organisations it is traffic they want.

Treating both as one thing is the reason so much of the advice on this subject is unhelpful. They arrive through similar mechanisms and they have opposite value to a site owner.

A third category matters more than either: user agents that claim to be a known crawler and are not. Verifying a crawler by its published address ranges, rather than by the string it announces, is the first thing to do before drawing conclusions from any log analysis.

Do AI crawlers respect robots.txt?

The major ones publish that they do, and compliance across the wider population of crawlers is inconsistent. robots.txt is a request rather than a control, and it has never been anything else.

The practical consequence is that robots.txt is the right place to state your position and the wrong place to rely on for enforcement. Anything that must not be fetched needs a control at the edge or in the application, not a line in a text file that operates on the honour system.

This changed materially on 1 July 2025, when Cloudflare announced that it would ask every new domain whether AI crawlers should have access, moving the default from opt-out to opt-in, and launched a private beta of a pay-per-crawl mechanism allowing site owners to charge for access. For a large share of the web, the enforcement question moved from a text file to a network control.

The numbers in this area disagree, and it is worth knowing why

There is a widely quoted statistic about how many pages AI companies crawl for every visitor they send back, called the crawl-to-refer ratio. It is worth understanding both because it is genuinely interesting and because the published figures are unreliable.

Cloudflare published the metric on 1 July 2025, covering 19 to 26 June 2025, putting Anthropic at 70,900 crawl requests per referral and Mistral at 0.1. The metric is defined as HTML requests from a platform’s user agents divided by HTML requests carrying that platform’s hostname in the Referer header, normalised to one referral.

Search for the same figure and you will find seven different numbers spanning more than two orders of magnitude, most with no period attached, most on blogs citing each other. The differences come from the measurement window, from which view of the data is selected and from normalisation, none of which usually gets stated.

Cloudflare notes a limitation in its own post that almost nobody repeats: traffic referred by Claude’s native application does not carry a Referer header at all, and the same is likely true of other native apps, so the published ratios probably overstate the imbalance.

The lesson generalises well beyond this metric. Before acting on any bot statistic, check what it counts, over what period, and who benefits from the number being large.

What does bot traffic actually cost?

Four costs, and organisations usually only look for the first.

Infrastructure. Requests consume bandwidth, origin capacity and, on a metered platform, money. High-volume crawling of a large site is a real load, and it arrives without any of the caching benefits that human traffic patterns bring.

Distorted measurement. This is the one that does the most damage and appears on no invoice. Automated traffic inflates sessions and users, deflates average engagement time, and depresses conversion rate because the denominator is wrong. Decisions get made on those numbers. A team can spend a quarter trying to fix a conversion rate that was never broken.

Content taken without traffic returned. Whether this is a cost depends entirely on your business model, and it matters less than the volume of debate suggests.

Metered AI features consumed by automation. This is the newest and the one growing fastest. An AI-powered search box, chat assistant or summarisation feature exposed on a public page costs money per request, and automation does not get bored.

How do you stop automated traffic running up an AI bill?

The failure mode has a name. OWASP’s Top 10 for LLM Applications 2025 lists unbounded consumption as the tenth risk, and names denial of wallet among its attack vectors: driving high-volume operations against a pay-per-use service specifically to impose unsustainable cost.

You do not need anybody to be attacking you for the controls to be worth having, because the same measures protect against a misconfigured integration or an enthusiastic user.

At the edge, before requests reach the application: verify crawlers properly and challenge those failing verification, rate limit by path with the AI endpoints treated separately from ordinary pages, and apply a stricter policy to any endpoint that costs money per request than to one serving a cached page.

In the application, which is where the controls that actually cap cost live: a token ceiling per request covering both input and output, rate limits tied to an authenticated identity rather than an address, timeouts, a spend cap that halts the service rather than merely warning about it, and a named person who can switch the feature off. All five are cheap to add before launch and awkward to retrofit once a bill has already arrived.

The specific thing to check today is whether any AI-powered feature on your site is reachable without authentication and without a per-user ceiling. If it is, your monthly cost has no upper bound, and the first you will know is the invoice.

Should we block AI crawlers?

Not as a blanket policy, and the framing of the question is the problem.

Blocking everything also blocks the crawlers that fetch your pages to answer user questions and cite you as the source. For most organisations, being findable and quotable by AI assistants is becoming as important as ranking in search results, and a blanket block removes that.

The decision worth making is more granular. Allow the crawlers that drive discovery and citation. Decide separately, and deliberately, about crawling for model training, where the answer depends on whether your content is a marketing asset or the product itself. Rate limit everything so that no crawler, permitted or not, can affect availability. And protect anything that costs money per request regardless of who is asking.

That decision belongs to the site owner rather than to a default, and since July 2025 the tooling has existed to make it properly.

Where the analytics problem lands

Fix the measurement before anything else, because it is cheap and it changes decisions immediately.

Check your analytics for the tells. A large share of direct traffic with no referrer. Very low average engagement time, measured in seconds rather than minutes. And the clearest one, which takes thirty seconds to check: open the city report and see whether any of your top three are data centre locations rather than places your customers live. A well-known cloud region sitting above your actual largest market is not a marketing result, and it is more common than most teams expect.

Then filter data centre traffic out, keep the unfiltered view alongside it for comparison, and re-baseline your reporting against the filtered numbers.

Expect the totals to fall and the engagement metrics to improve. Neither is a change in performance. It is the first accurate picture the organisation has had.

This work belongs in a maintenance routine rather than in a project, alongside the other checks in our guide to what a website maintenance service should cover, and the edge controls described here are configured at the same layer as the HTTP security headers that should already be in place.

Before your next reporting cycle

Pull a week of server logs and count what share of requests came from verified crawlers, unverified user agents claiming to be crawlers, and everything else. Then check whether any AI-powered feature on your site can be reached without authentication and without a spend ceiling.

Those two checks take an afternoon between them and they usually change what somebody does next. We run them as part of our maintenance work and are happy to run them as a standalone piece if it would be useful.

Frequently asked questions

What percentage of web traffic is bots?

Imperva’s 2025 Bad Bot Report found automated traffic reached 51 per cent of all web traffic in 2024, split as 37 per cent bad bots and 14 per cent good bots against 49 per cent human. Individual sites vary widely, so the figure is a reason to measure your own rather than a substitute for doing so.

Will blocking AI crawlers hurt our search visibility?

It can, and the risk is broader than traditional search. Crawlers that fetch pages to answer user questions are what allow an AI assistant to cite you as a source. Blocking them removes that visibility, so the decision should distinguish between crawling for citation and crawling for model training.

Can bots cost us money on an AI feature?

Yes. Any AI feature charged per request and reachable without authentication or a spend ceiling can be driven into a large bill, whether deliberately or accidentally. OWASP names this unbounded consumption, and the controls are token limits per request, rate limits per identity, and a hard budget ceiling with alerting. 

References

• Imperva, a Thales company: 2025 Bad Bot Report, covering 2024 data
• Cloudflare: Introducing pay per crawl, 1 July 2025: https://blog.cloudflare.com/introducing-pay-per-crawl/
• Cloudflare: The crawl before the fall of referrals, 1 July 2025, data for 19 to 26 June 2025: https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/
• OWASP Top 10 for LLM Applications 2025, LLM10 Unbounded Consumption: https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/