Server Setup 2020

Server setup overview:

  • Cheapest Hetzner Cloud VM. 1 CPU, 2 GiB RAM, 20 GiB disk. Make sure to set up an SSH key in the Hetzner account so that when the VM is first installed, SSH access will already be key-only.
  • Debian stable.
  • Wireguard on the VM. All access for administration/debug/monitoring purposes go through the wireguard interface.
  • Admin access via SSH. Public-key auth only. Follow Mozilla's OpenSSH config guide.
  • sshd listening only on the Wireguard interface. This creates 2 layers of strong key based authentication and encryption (exploits in one are unlikely to affect the other). This also hides sshd from port-scans.
  • Root login disabled completely.
  • One user with sudo access. Strong random password. Password auth is accessible "locally" via Cloud-provider's machine console. (2 layers of authentication here: Cloud provider login, plus machine console login using strong password). You could disable "local" console login, but if you break SSH access somehow then you'll be out of options and need to wipe and reinstall the machine (which is ok if you're set up for that!). Note that you have to really trust your cloud provider since if you go through this kind of machine console then the cloud provider can see your password in the clear. But you have to trust your cloud provider anyway, right?
  • nftables for packet filtering. Manually configured:
    • Accept but rate limit packets on network control protocols (ICMP etc).
    • Accept packets on established connections.
    • Split to a chain for the public interface and a chain for the Wireguard interface.
    • Public interface: Accept new connections on 80,443 TCP, 443 UDP (for HTTP/3 protocol), plus UDP on the wireguard external port (otherwise you'll never be able to connect to the VPN!).
    • Wireguard interface: Accept new connections on 80,443 TCP, 443 UDP, plus 22 TCP (SSH). Open other ports as needed (eg, for prometheus metrics collection etc). This could be configured to accept-all since anything appearing on the wireguard interface has already been authenticated. But filtering is a little better since it protects the server from receiving traffic that has "leaked" from my home LAN (a culprit here is Spotify, which broadcasts service discovery and mdns requests on every interface it can find to try to find smart speakers on the local network; this means when the VPN is active there can be service discovery requests and other "consumer network" type traffic reaching the server).
    • Default-drop policy.
    • Default-drop for forwarding chain.
    • Default-accept for outgoing. (This could be restricted further but I consider the configuration complexity and maintenance cost high relative to the benefit)
    • Don't use 'ct related' rules; these are for Application-level Gateway magic intended to support protocols that operate on multiple ports where the remote side will connect back to the calling side on some port different to the port that the original connection was established on. For example, in traditional FTP the client connects to the server for the control part of the protocol, but in order to transfer data the server connects back to the client on a client-specified port; supporting this in the firewall requires the firewall to understand enough FTP to extract the client port and then accept an incoming connection on this port which it would not normally do (ie, temporarily open the port). This is a massive PITA because the firewall needs to parse FTP. And that's just one example; there are a bunch of protocols like this. Luckily I don't care about any of them, so a much safer way is to just disable all the application protocol parsers and don't bother with any "related" connection-state rules. See also Secure use of iptables and connection tracking helpers and other articles.
  • Enable rp_filter in /etc/sysctl.conf, setting net.ipv4.conf.all.rp_filter = 1. This is return path filtering which drops outgoing packets that have source addresses that wouldn't route back to the machine. Under normal circumstances nothing on the server should generate packets like that, but it could happen due to some attack or exploit, and filtering that out in the kernel helps avoid the machine being used in part of some broader (D)DoS attack somehow.
  • Disable nf_conntrack_helper. As part of not caring about multi-port protocols that require the firewall to parse the application level protocol, it's nice to disable nf_conntrack_helper in the netfilter conntrack settings. This way the "helpers" (which do the parsing) won't be used without explicitly configuring use within the netfilter ruleset.
  • Unique non-privileged user for each running internet-facing service.
  • Services managed via systemd units to come up on boot.
  • systemd journal configuration set to rotate the journal and limit total journal size to avoid excessive disk usage from accumulating system logs.

Services:

  • Caddy to provide a general frontend, and terminate web-facing TLS. Caddy deals with getting Let's Encrypt TLS certificates so there's no extra certificate management. Caddy does support live config reload, but I've disabled this for now. Caddy can serve files directly which makes it easy to serve a "This site is down for maintenance" type holding page. (Other options: nginx, haproxy, though haproxy can't serve files easily as far as I can tell)
  • Local Prometheus collector. Monitoring and metrics are useful. Logs are ok but real-time monitoring is better for some purposes. A local collector is a little odd but is the simplest approach for a single-VM setup.
  • Miscellaneous application servers.

Why wireguard?

SSH should be strong if properly configured (and the intention is to keep the SSH configuration strong), so why put it behind wireguard? A few reasons:

  • Layered security: For someone to get in, they'll need both a vulnerability in wireguard, and one in SSH. This prevents remote exploit problems in SSH from being immediately catastrophic to the system's overall security, so that there's time to upgrade things.
  • The VPN access can be used for direct access to other services on the machine. Of course, this works against the layered security, but it's probably still reasonable for some things (e.g., read only access to monitoring data or something). Alternatively I could use an SSH tunnel (within wireguard).
  • Hiding the SSH port and any other services running on the machine. The wireguard protocol is authenticated from the first packet, and wireguard won't respond to packets that don't come from a known & configured peer. This means connection attempts don't reach the SSH service, so port scans should show the machine as being dark except for genuinely public services (HTTP/HTTPS). Commonly discussed or used alternatives and related options:
    • fail2ban: Doesn't hide SSH at all, but dynamically adds IP filtering rules in response to patterns of access that look like attacks. Lots of people seem to swear by fail2ban, and lots of others say it's a terrible idea and you should just make sure you've got SSH configured correctly so that it can't be brute forced.
    • SSH on a non-standard port: Doesn't hide SSH from full port scanning but hides it from people who are only scanning common service ports. Comes with the nominal disadvantage that if the machine is compromised without you realising, then potentially the attacker can inject some MiTM running on the machine as a non-privileged user and use it to intercept connection attempts and thereby escalate to gain privilege. Frankly that particular threat seems a little outlandish to me - if someone has been able to log into my VM as a non-privileged user then they can almost certainly own the machine even if they can't MiTM future SSH connections, and they certainly can perform resource attacks to deny access to me. Anyway, from anecdotes I've read online, in practice using a non-standard SSH port (but otherwise nothing special) does significantly cut down on attacks.
    • Port knocking: This is a classic. This involves some complicated packet filter rules which drop connection attempts, but track them and once the 'correct' sequence of packets has been seen, then a final connection attempt (presumably to a non-standard SSH port) will be accepted. This is very fun and will hide the presence of SSH unless an attacker is able to watch all the traffic to the machine. However, I dislike complexity in my security configuration. Port knocking seems much more fiddly than wireguard, and doesn't come with clear and strong cryptographic properties.

Why nftables?

nftables is used as a firewall. There are three questions here. Let's start from the easiest:

  • Why nftables instead of iptables? nftables is newer, and supposed to be better. In particular, it's supposed to be possible to achieve higher performance, and the configuration is more flexible and perhaps easier to understand and work with. Ultimately though, I'm using nftables because Debian 10 uses nftables by default and Debian 10 is the easiest OS option I had on Hetzner since it's one of the prepackaged images they provide.
  • Why should the server have a firewall? Well, everyone knows you should have a firewall, right? Actually I don't like that answer because it doesn't explain anything. What actual value does a firewall provide?
    • Layered security: You should only be running properly configured and secure services on your system, any services that are not supposed to be public should either not be running or should be limited to only listening on localhost (or possibly on the local VPN interface). So even without a firewall it shouldn't be possible to connect to the machine on the publicly routable address and make anything bad happen. However, it's easy to make a mistake in configuration and run some service listening on all interfaces where it should be restricted, or leave a database using no password or a weak password or something, and services might have exploitable vulnerabilities, and so on. With a firewall you have an extra layer preventing access to services that are intended to be internal-only.
    • Packet level monitoring and rate limiting: You may want to rate limit various things like network control type packets or even new connections to your web server. To do that at the packet level you need a firewall.
    • Flexibility to add temporary rules: If your server comes under attack (and you actually notice which may be the hard part) then you might be able to add temporary rules to mitigate the attack; e.g., just dropping everything from a certain IP. You don't want to have to install and learn to configure a firewall from scratch in that situation, you want to have the whole thing set up and running, preferably with a few pre-tested rule templates that you can put into play quickly.
  • Why plain nftables instead of one of the firewall manager tools? There are a few well known firewall management tools available on Linux, in particular ufw and firewalld (and others but those are the two that show up most easily in searches). However, I just don't like them. ufw seems to provide only a command line based system for configuration where you execute a series of ufw commands to configure your firewall. E.g., ufw default deny incoming and stuff. This is absurd - no one wants to configure a firewall by running individual CLI commands. Firewall configuration should obviously be in a configuration file that you can write and understand and keep in version control, so it's all in one place. Secondly, ufw is built around iptables not nftables. It can use nftables - as all iptables-based systems can - because there is an nftables backend for iptables that does conversion. But that results in a really horrible and hard to follow nftables config. The other one, firewalld might be better (and in fact I use it on my home machines), but in general I don't like the way the configuration works. Again, I just want a single config file that I can write and define the entire ruleset, I don't want to go through a list of 100 predefined options for allowing particular services. In the end, the plain nftables config language is pretty easy to follow and use, and gives me a single config file that fits my needs well. Plus it means I can easily set up counters and rate limits and other things that don't fit directly into the simple world of ufw and firewalld.

Why Hetzner?

It's the cheapest non-sketchy Cloud/VM provider I could find. Since I'm not looking to build anything with "High Availability" and I'm not interested in using the many scalable but expensive "managed services"/Cloud SaaS providers, running one or two cheap VMs that I manage myself is what I want.

Hetzner provides a few premade OS images to use; when you create the VM in the first place the only ones it presents you with are Debian (currently Debian 10), Fedora, CentOS, or Ubuntu. I have no prior experience with Fedora, so that's probably not a great idea for me. Ubuntu seems to always have its own weird variants of what everyone else is using (e.g., "Snap" based installs, and stuff), plus I don't really trust them not to have telemetry or advertising by default, so Ubuntu is out. I don't know why anyone would pick CentOS, I suppose there must be some reason, but it's not for me. So that leaves Debian. Hooray Debian! You may be old and stuck in your ways, but at least I can trust you to be sane and stable.

(Hetzner has some install ISOs for a bunch of other distros and also for some BSDs and stuff, but let's not worry about that).

Hetzner uses cloud-init to inject an SSH key you provide into the VM when it's first set up, so the thing is initialised with a no-password root account that can only be accessed via pubkey SSH. Good.