Bot traffic, nuisance, and DoS attacks

A constant slice of every website's traffic is automated. Some of it is welcome — search-engine crawlers, link previews from Slack and WhatsApp, uptime monitors. A lot of it isn't — vulnerability scanners, scrapers, credential-stuffing tools, AI training crawlers, small DoS attempts.

This article explains what you can do yourself in Core to keep a site stable under bot pressure, what Cyberfusion already does at the infrastructure level, and the limits of what any automated defence can catch.

Where bots hurt the most

Bots aren't usually trying to take a site down. The damage is a side effect of behaviour a real visitor wouldn't have:

• Hitting search and filter URLs that bypass the page cache (e.g. /?s=..., /?filter=colour%2Bsize%2Bprice...).
• Hammering login endpoints with username/password lists.
• Crawling every paginated link, including ones a sitemap would have skipped.
• Following dead URLs and generating thousands of 404s.

The result is the same: CPU, database time, and cache that should be serving real visitors is spent on automated traffic instead.

Webshops feel this first. Search, filters, and checkout bypass the page cache and hit the database directly, so a bot crawling filter combinations can send the database so many queries that real customers see slow pages or errors at checkout. Any site with a dynamic backend (PHP, NodeJS) is exposed to the same pattern — webshops just see it most.

What you can do in Core

You can't stop every bot from sending requests. But you can put hard limits on what one virtual host is allowed to consume, so a bot wave on one site doesn't slow the rest of the node down.

Cap the resources one site can use

On the FPM pool:

• 'Max Children' — concurrent PHP requests. Once it's reached, further requests queue instead of starting more workers.
• 'CPU Limit (cores)' — total CPU the pool can use. Once reached, the pool slows down but can't starve other sites on the node.
• 'Memory Limit (MB)' — total RAM the pool can use. Once reached, the pool can't push the node into swap or get killed for OOM.

These don't reduce bot traffic, but they contain its impact: a bot wave on one site stays inside that site's budget.

Restrict who can reach a domain

On the domain router, the 'Firewall Groups' section limits the domain to specific IP ranges. The most common use is on admin or staging domains — when only your team needs to reach them, an IP allow-list cuts bot traffic on those domains to zero.

Put a login wall in front of pre-launch sites

Basic authentication on a virtual host (or a path inside one) blocks unauthenticated requests at the web server, before PHP runs. A bot crawling pre-launch URLs gets a 401 and moves on; the FPM pool never starts a worker.

What Cyberfusion does at the infrastructure level

These are on for every site by default — you don't configure them.

• Rate limiting — the load balancer tracks request rates per source IP and slows or drops clients that exceed reasonable thresholds. Identifiable good bots (e.g. major search engines) get a higher allowance than anonymous traffic.
• Tarpitting — clients that already match a known abuser pattern are held in slow connections instead of being answered. The attack costs the attacker more CPU and bandwidth, while real visitors are unaffected.
• Standard transport-layer DoS protection through the upstream network.

What's still hard — and what we're working on

The mechanisms above handle most casual nuisance and small DoS attempts. They don't catch elaborate attacks:

• Botnets that rotate across thousands of IPs, so no single IP looks abusive.
• Low-and-slow scrapers — a few requests per minute, staying under any reasonable per-IP threshold.
• AI training crawlers that imitate browser behaviour closely enough to look like real visitors.

For these cases, Cyberfusion is currently experimenting with:

• Blackwall — a managed bot and web application firewall service that classifies traffic using continuously-updated threat data.
• Anubis — an open-source proof-of-work challenge gateway. Real browsers solve a small cryptographic puzzle invisibly; scrapers and AI crawlers spend so much CPU on the puzzle that crawling at scale becomes uneconomic.

Neither is on by default yet. If you're seeing elaborate bot traffic on a specific site, contact Cyberfusion — we can discuss enabling one of these for that domain.

The honest limit

No setting in Core, and no external product, catches every kind of nuisance. A determined attacker eventually finds a new pattern — a freshly-rented set of IPs, a different headless-browser fingerprint, a slightly different request shape — and someone has to look at the logs, recognise the pattern, and put a targeted block in place.

For Cyberfusion-managed clusters, that's part of the support relationship: contact Cyberfusion and we'll investigate.

Use cases by target group