Separating services across nodes

Wondering whether your database, web server and mail should each get their own node?

There are two separate reasons to split them up — security and performance — and they don't automatically point the same way. Separating services also has a cost beyond the extra nodes on your invoice: every query then has to cross the network.

This article describes what you gain and what you give up on both counts.

A node runs the software of the node groups it has. A node with 'nginx', 'PHP' and 'MariaDB' runs the web server, PHP and the database server on one machine. Give 'MariaDB' to a second node instead, and the database server runs there.

Small clusters usually have one node with every group on it. That's a deliberate choice, not an oversight, and the rest of this article is about what you're choosing between.

The two questions are independent. A cluster can have a strong security reason to separate its services and no performance reason at all, or the other way round. It's worth deciding them separately rather than looking for one answer that covers both.

Security: what an attacker gains when everything is on one node

A break-in usually happens in steps, and each step needs its own vulnerability:

Getting in. Someone exploits an out-of-date plugin, theme or CMS on one of your sites. They can now run commands as that site's UNIX user. At this point they can read and change that site's files, and nothing else.
Becoming root. They use a Local Privilege Escalation (LPE) vulnerability, a bug in the operating system that lets a normal user become the root user, who may do anything on the machine.
Taking everything on the node. As root, they read and change every file on that node. Not just the site they came in through.

Step 3 is where sharing a node matters. If the database server runs on the same node, root gets:

Every database on that node, not just the one belonging to the site they broke into. Database passwords and grants are enforced by MariaDB or PostgreSQL when a client connects. Someone reading the database's files straight off the disk never connects to the database server, so none of that applies. They can copy every database and change the contents of any of them.
Encryption at rest doesn't stop this. With 'MariaDB encryption keys' enabled the database files are encrypted on disk, but the key that decrypts them sits on the same node, because MariaDB needs it to serve queries. Someone with root on that node has the key too. Encryption at rest protects against someone taking the physical disks out of the data centre, not against someone who is on the running machine. See Security and auditing.

The same reasoning applies to anything else on the node: mailboxes if it also handles mail, TLS private keys, and the files of every other UNIX user on it.

What separating services does not prevent

Your website has to reach its own database. The username and password are in wp-config.php or .env, which the site's UNIX user can read. So an attacker who only manages step 1, on a website that talks to a database on a completely separate node, still has that database's contents. They log in with the credentials they found, exactly as the website does.

Moving the database to its own node does nothing about that. If the only thing on the database node is the one database that site uses, separating the two buys you very little.

What separating services does prevent

What changes is everything the website has no business reaching:

Other databases. The application's database user has grants on its own database only. root on a database node has all of them. Separating the database server means an attacker has to settle for the access the application already had, instead of getting every database on the machine.
Services the website never talks to at all. Mailboxes are the clearest case. A website has no legitimate reason to read anyone's mail, so if mail runs on a different node, a website break-in doesn't reach it. There's no equivalent of "the credentials were in the config file" here.
The next step of the attack. Reaching a separate node requires a network path to it and a further vulnerability on that node. On a shared node, step 3 hands the attacker everything at once.

What blocks the attack wherever the services run

Step 2 above, becoming root, is the step Cyberfusion blocks by default. UNIX user namespacing disables SUID binaries for namespaced UNIX users, and most published LPE exploits need one to do the escalation. An attacker who got code execution as a UNIX user usually can't get past it, even when the underlying operating-system bug is present and unpatched.

That's what makes sharing a node defensible. Keep namespacing on. See when to opt out for the one case where it's turned off, and note that turning it off is what makes step 2 possible again.

Where the security case is strongest

Weigh it by what else lives on the node:

A cluster running one website, with one database. The gap between "attacker has the website's own database credentials" and "attacker has root on the database server" is small, because there's nothing else on it to take.
A cluster running many websites, or many customers. One compromised site becomes every site's files and every site's database. The sites have nothing to do with each other, and the owner of the site that got hacked isn't the only one affected.
Anything holding data of a different kind. Mail, and databases belonging to other applications, are worth separating even on a small cluster, because a website compromise otherwise reaches data the website never needed.

Performance: what separating gains you

Services stop competing for the same memory and CPU

A database server and PHP on one node draw from the same pool of memory and the same processors. They get in each other's way in both directions: a heavy report query slows down page loads, and a traffic peak slows down queries.

Memory is where this bites hardest, because both services want as much of it as they can get, for different reasons. PHP wants memory for worker processes, so it can handle more visitors at the same time. MariaDB wants memory for the InnoDB buffer pool, so it can keep your tables out of disk reads. On a shared node you're dividing one fixed amount between them, and giving one more means giving the other less. Give the database its own node and its whole memory goes to the database. See InnoDB buffer pool for what that memory actually buys you.

A node that does one job does it better

Beyond dividing up memory and CPU, a machine running fewer different things is simply quicker at what's left.

Every time a network packet arrives or a disk finishes a read, the hardware interrupts the processor: it stops what it was doing, handles the event, then picks its old work back up. A node running nginx, PHP and MariaDB gets all three services' worth of these. Each one costs a small amount of time, and it also throws away part of the processor's own short-term memory of what it was working on, so resuming the interrupted work is slower than it would otherwise be. The work all gets done, but a busy node spends a real share of its time switching between jobs instead of doing them.

Give the web server its own node and it isn't competing with database work for any of that. There's a second, less obvious gain: settings that apply to the whole machine can be chosen for one job instead of compromised between two. A machine tuned for many short-lived web connections and a machine tuned for a database are not tuned the same way, and when both run on one node, neither gets what it wants.

You can grow one part

If the database needs more memory but the web server doesn't, a separate database node lets you grow only that one. On a shared node, more memory for the database means paying for a bigger machine that the web server doesn't need.

Maintenance affects less at a time

Rebooting a node for a kernel update takes down everything on it. With services split up, a database reboot doesn't stop the web server from serving cached and static content.

Performance: what separating costs you

Every query has to cross the network

This is the trade-off that matters most for the database server, and it's the reason separating services isn't automatically the faster choice.

When the application and the database are on one node, a query never leaves the machine. When they're on separate nodes, every query is a real network round trip: out to the database node, and back. Cyberfusion's cluster networking is fast, but fast is not the same as free.

A round trip inside a data centre typically costs a fraction of a millisecond. That sounds like nothing, and for a page that runs five queries it is. It becomes visible when an application runs a lot of queries per page:

A page that runs 20 queries pays that cost 20 times.
A page that runs 300 pays it 300 times. That's not an unusual number for a CMS with a lot of plugins, or for an application that loads related records one at a time in a loop instead of fetching them together.

At a third of a millisecond per round trip, 300 queries add roughly 90 milliseconds to every single page load. The awkward part is that none of it shows up as a slow query anywhere. Every individual query is still fast. There are just a lot of them, and each one is now further away.

What to do about it:

Count the queries your pages run before you move the database, not after. A performance monitoring tool shows you this per request. See PHP performance monitoring with Tideways.
Reduce the number of queries. Loading related records in one query instead of one query per record is usually the single biggest change available, and it makes the application faster on a shared node too.
Cache what you can. A result served from Redis is a query that doesn't happen at all.
Accept that some applications shouldn't be separated. If an application runs hundreds of queries per page and you can't change its code, keeping the database on the same node may genuinely be the right call, and the security case has to be weighed against that.

More machines to keep running

Every node is an operating system that needs updates and reboots, and the site now depends on two machines and the network between them rather than one. Cyberfusion handles the maintenance, but more moving parts is still more that can go wrong.

What it costs in money

Every node costs money every month. A cluster with the web server, the database and mail each on their own node needs at least three, plus more if you want any of them redundant.

For a cluster running a handful of low-risk sites that aren't short of memory or CPU, that can be more than the situation justifies. Separating services is worth paying for when one break-in would reach more than you're comfortable with, or when the services are genuinely competing for the same machine. It isn't worth paying for on the general principle that separate is better.

There's no single right answer that fits every cluster. If you're unsure which side your cluster falls on, contact Cyberfusion support with what the cluster runs.

InnoDB buffer pool: why a database node's memory matters, and what happens when the data no longer fits in it.
UNIX user namespacing: the SUID restriction that blocks most privilege escalation, and the one reason to opt out.
Security and auditing: what's encrypted at rest, what a node records on SSH login, and why root access is the limit of a node's own logs.
Add a node group: how to move a service onto a node.
All about HAProxy listens: how applications reach a database on another node, including on another cluster.