Bot traffic and DoS attacks

A constant slice of every website's traffic is automated. Some of it is welcome: search-engine crawlers, link previews from Slack and WhatsApp, uptime monitors. A lot of it isn't: vulnerability scanners, scrapers, credential-stuffing tools, AI training crawlers, small DoS attempts.

This article explains what you can do yourself in Core to keep a site stable under bot pressure, what Cyberfusion already does at the infrastructure level, and the limits of what any automated defence can catch.

What you can do, most effective first

1. Cap what one site can consume — 'Max Children', 'CPU Limit (cores)' and 'Memory Limit (MB)' on the FPM pool. A bot wave then stays inside that site's budget.
2. Restrict who can reach admin and staging domains — firewall groups on the domain router.
3. Put basic auth in front of pre-launch sites — bots get a 401 and PHP never runs.

Rate limiting and tarpitting are already on for every site, and you don't configure those. See What Cyberfusion already does.

For elaborate bot traffic none of this stops, see What's still hard.

Where bots hurt the most

Bots are usually not trying to take a site down. The damage is a side effect of behaviour a real visitor would not have:

• Hitting search and filter URLs that bypass the page cache, for example /?s=... or /?filter=colour%2Bsize%2Bprice....
• Hammering login endpoints with username and password lists.
• Crawling every paginated link, including ones a sitemap would have skipped.
• Following dead URLs and generating thousands of 404s.

The result: CPU, database time and cache that should be serving real visitors goes to automated traffic instead.

Webshops feel this first. Search, filters and checkout bypass the page cache and hit the database directly, so a bot crawling filter combinations can send the database so many queries that real customers see slow pages or errors at checkout. Any site with a dynamic backend (PHP, NodeJS) is exposed to the same pattern.

Cap the resources one site can use

You cannot stop bots from sending requests. You can put a hard limit on what one site is allowed to consume.

On the FPM pool:

• 'Max Children' — concurrent PHP requests. Once reached, further requests queue instead of starting more workers.
• 'CPU Limit (cores)' — total CPU the pool can use. Once reached, the pool slows down but cannot starve other sites on the node.
• 'Memory Limit (MB)' — total RAM the pool can use. Once reached, the pool cannot push the node into swap or get killed for running out of memory.

These do not reduce bot traffic. They contain its impact: a bot wave on one site cannot slow down the other sites sharing the same server.

Restrict who can reach a domain

On the domain router, the 'Firewall Groups' section limits the domain to specific IP ranges.

Most common use: admin and staging domains. When only your team needs to reach them, an IP allow-list cuts bot traffic on those domains to zero.

Put a login wall in front of pre-launch sites

Basic authentication on a site, or a path inside one, blocks unauthenticated requests at the web server, before PHP runs.

A bot crawling pre-launch URLs gets a 401 and moves on. The FPM pool never starts a worker.

What Cyberfusion already does

On for every site by default:

• Rate limiting — the load balancer tracks request rates per source IP and slows or drops clients above reasonable thresholds. Identifiable good bots, such as major search engines, get a higher allowance than anonymous traffic.
• Tarpitting — clients matching a known abuser pattern are held in slow connections instead of being answered. The attack costs the attacker more CPU and bandwidth; real visitors are unaffected.
• Transport-layer DoS protection through the upstream network.

What's still hard

The mechanisms above handle most casual nuisance and small DoS attempts. They do not catch:

• Botnets rotating across thousands of IPs, so no single IP looks abusive.
• Low-and-slow scrapers sending a few requests per minute, staying under any reasonable per-IP threshold.
• AI training crawlers that imitate browser behaviour closely enough to look like real visitors.

Cyberfusion is currently experimenting with two options for these cases:

• Blackwall — a managed bot and web application firewall service that classifies traffic using continuously updated threat data.
• Anubis — an open-source proof-of-work challenge gateway. Real browsers solve a small cryptographic puzzle invisibly, while scrapers and AI crawlers spend so much CPU on the puzzle that crawling at scale stops being worth it.

Neither is on by default. Contact Cyberfusion to discuss enabling one for a specific domain.

The limit of any automated defence

No setting in Core, and no external product, catches every kind of nuisance. A determined attacker eventually finds a new pattern — freshly rented IPs, a different headless-browser fingerprint, a slightly different request shape. Someone then has to read the logs, recognise the pattern, and put a targeted block in place.

For Cyberfusion-managed clusters, that is part of the support relationship. Contact Cyberfusion and it will be investigated.

What to set, by situation

• Many client sites on one server — set sensible 'Max Children' and 'Memory Limit (MB)' on every site, so a bot wave on one client's site cannot slow the others down. For pre-launch sites, attach basic auth.
• Webshops — sizing matters most. Combine a generous 'Max Children' for real peaks with a 'CPU Limit (cores)' so a bot wave cannot starve other sites. Add firewall groups on admin and staging domains.
• One tenant per site — with each on its own account, one tenant's bot problem stays contained to that tenant. For admin endpoints, pair basic auth with an IP allow-list.
• Sites that attract AI crawlers — developer tools, public APIs, doc sites. These often outgrow per-IP rate limiting; contact Cyberfusion about Blackwall or Anubis.