Skip to content

Separating services across nodes

Wondering whether your database, web server and mail should each get their own node?

There are two separate reasons to split them up — security and performance — and they don't automatically point the same way. Separating services also has a cost beyond the extra nodes on your invoice: every query then has to cross the network.

This article describes what you gain and what you give up on both counts.

The short version

Where separating is clearly worth it:

  • Mail. A website has no legitimate reason to read anyone's mail, so separate it even on a small cluster.
  • Many websites, or many customers, on one cluster. One compromised site otherwise becomes every site's files and every site's database.

Where it usually isn't: one website with one database, on a node that isn't short of memory or CPU.

Before moving a database to its own node, count the queries your pages run. A page running 300 queries pays a network round trip 300 times, which adds roughly 90 ms to every page load without any single query showing up as slow. See what separating costs you.

Unsure which side your cluster falls on? Contact Cyberfusion support with what the cluster runs.

What decides which services share a node

A node runs the software of the node groups it has. A node with 'nginx', 'PHP' and 'MariaDB' runs the web server, PHP and the database server on one machine. Give 'MariaDB' to a second node instead, and the database server runs there.

Small clusters usually have one node with every group on it. That is a deliberate choice, not an oversight.

There are two separate reasons to split services up — security and performance — and they do not automatically point the same way. Decide them separately.

Security: what an attacker gains when everything is on one node

A break-in happens in steps, each needing its own vulnerability:

  1. Getting in. Someone exploits an out-of-date plugin, theme or CMS on one of your sites. They can now run commands as that site's UNIX user — read and change that site's files, and nothing else.
  2. Becoming root. They use a Local Privilege Escalation (LPE) vulnerability: a bug in the operating system that lets a normal user become root, who may do anything on the machine.
  3. Taking everything on the node. As root, they read and change every file on that node, not just the site they came in through.

Step 3 is where sharing a node matters. With the database server on the same node, root gets:

  • Every database on that node, not just the one belonging to the site they broke into. Database passwords and grants are enforced by MariaDB or PostgreSQL when a client connects. Someone reading the database's files straight off the disk never connects to the database server, so none of that applies. They can copy every database and change the contents of any of them.
  • The encryption key as well. With 'MariaDB encryption keys' enabled the files are encrypted on disk, but the key that decrypts them sits on the same node, because MariaDB needs it to serve queries. Encryption at rest protects against someone taking the physical disks out of the data centre, not against someone on the running machine. See Security and auditing.

The same applies to everything else on the node: mailboxes if it also handles mail, TLS private keys, and the files of every other UNIX user.

What separating does NOT prevent

Your website has to reach its own database. The username and password are in wp-config.php or .env, which the site's UNIX user can read.

So an attacker who only manages step 1, on a website whose database lives on a completely separate node, still has that database's contents. They log in with the credentials they found, exactly as the website does.

If the only thing on the database node is the one database that site uses, separating the two buys you very little.

What separating DOES prevent

Everything the website has no business reaching:

  • Other databases. The application's database user has grants on its own database only. root on a database node has all of them. Separating means an attacker has to settle for the access the application already had.
  • Services the website never talks to. Mailboxes are the clearest case. A website has no legitimate reason to read anyone's mail, so with mail on a different node, a website break-in does not reach it. There is no "the credentials were in the config file" here.
  • The next step of the attack. Reaching a separate node needs a network path to it and a further vulnerability on it. On a shared node, step 3 hands the attacker everything at once.

What blocks the attack wherever the services run

Step 2 — becoming root — is the step Cyberfusion blocks by default. UNIX user namespacing disables SUID binaries for namespaced UNIX users, and most published LPE exploits need one to do the escalation.

An attacker who got code execution as a UNIX user usually cannot get past it, even when the underlying operating-system bug is present and unpatched.

That is what makes sharing a node defensible. Keep namespacing on. See when to opt out — and note that turning it off is what makes step 2 possible again.

Performance: what separating gains you

Services stop competing for the same memory and CPU

A database server and PHP on one node draw from the same memory and the same processors, and get in each other's way in both directions: a heavy report query slows page loads, and a traffic peak slows queries.

Memory bites hardest, because both want as much as they can get for different reasons. PHP wants memory for worker processes, to handle more visitors at once. MariaDB wants memory for the InnoDB buffer pool, to keep your tables out of disk reads. On a shared node you divide one fixed amount between them, and giving one more means giving the other less.

Give the database its own node and all of its memory goes to the database. See InnoDB buffer pool for what that memory buys you.

A node that does one job does it better

Every time a network packet arrives or a disk finishes a read, the hardware interrupts the processor: it stops what it was doing, handles the event, then picks its old work back up. A node running nginx, PHP and MariaDB gets all three services' worth of these.

Each one costs a small amount of time, and it also throws away part of the processor's short-term memory of what it was working on, so resuming is slower than it would otherwise be. The work still gets done, but a busy node spends a real share of its time switching between jobs instead of doing them.

There is a second gain: settings that apply to the whole machine can be chosen for one job instead of compromised between two. A machine tuned for many short-lived web connections and a machine tuned for a database are not tuned the same way.

You can grow one part

If the database needs more memory but the web server does not, a separate database node lets you grow only that one. On a shared node, more memory for the database means paying for a bigger machine the web server does not need.

Maintenance affects less at a time

Rebooting a node for a kernel update takes down everything on it. With services split, a database reboot does not stop the web server from serving cached and static content.

Performance: what separating costs you

Every query has to cross the network

This is the trade-off that matters most for the database, and the reason separating is not automatically faster.

On one node, a query never leaves the machine. On separate nodes, every query is a real network round trip: out to the database node, and back. Cyberfusion's cluster networking is fast, but fast is not free.

A round trip inside a data centre typically costs a fraction of a millisecond. For a page running five queries, that is nothing. It becomes visible when a page runs many:

  • 20 queries → the cost is paid 20 times.
  • 300 queries → 300 times. Not unusual for a CMS with a lot of plugins, or an application that loads related records one at a time in a loop instead of fetching them together.

At a third of a millisecond per round trip, 300 queries add roughly 90 milliseconds to every page load. The awkward part: none of it shows up as a slow query. Every individual query is still fast. There are just a lot of them, and each one is now further away.

What to do about it:

  1. Count the queries your pages run before you move the database, not after. A performance monitoring tool shows this per request. See PHP performance monitoring with Tideways.
  2. Reduce the number of queries. Loading related records in one query instead of one query per record is usually the biggest change available, and it makes the application faster on a shared node too.
  3. Cache what you can. A result served from Redis is a query that does not happen at all.
  4. Accept that some applications should not be separated. If an application runs hundreds of queries per page and you cannot change its code, keeping the database on the same node may be the right call — and the security case has to be weighed against that.

More machines to keep running

Every node is an operating system that needs updates and reboots, and the site now depends on two machines plus the network between them. Cyberfusion handles the maintenance, but more moving parts is still more that can go wrong.

What it costs in money

Every node costs money every month. A cluster with the web server, the database and mail each on their own node needs at least three, plus more for redundancy.

For a handful of low-risk sites that are not short of memory or CPU, that can be more than the situation justifies. Separating services is worth paying for when one break-in would reach more than you are comfortable with, or when the services are genuinely competing for the same machine. Not on the general principle that separate is better.