Running an IPv6-only host — redux

I have previously blogged about why you should consider IPv6 only hosting and setting up Apps on Kubernetes IPv6 to run my WordPress blog.

Kubernetes is not really designed for a single server (but is great for scaling and enterprise system), and although it was good experience learning how to set it up on IPv6, the overhead was too much and I eventually ended up with a crashed blog.

I'm still running IPv6 only, but with a much simpler set up.

This consists of docker, configured to run with IPv6, with docker-compose to run the different components and systems.

If you are planning on setting up your own server, read my notes on Securing your IPv6-only docker server before starting.

On my server there are currently three instances of WordPress for different websites, and 3 corresponding databases, as well as a Matrix Synapse server and plugins.

Read on for my notes on initial setup of the server with IPv6 and connectivity testing, including addressing schemes, docker configuration, IPv6 network address translation, and the Network Discovery Protocol Proxy Daemon.

The cost of IP addresses

With the cost of IPv4 addresses (at March 2022) currently USD 40.00, a few colleagues at Telstra Purple, where I work in IoT consulting, have asked me what is the cost of an IPv6 address. (It has also been pointed out that a Raspberry Pi Zero, or other IoT device, is cheaper to purchase than the IPv4 address you might use to run it.)

Well, IPv4 has around 4 billion addresses, and a single IPv4 address is a /32 (32-bit length), and costs USD 40.00, so a C-class /24 subnet has 256 addresses and costs USD 10,000.

Going the other way, the IPv6 subnets have a fixed size of /64, which is 4 billion × 4 billion addresses, or about 16 quadrillion addresses.

An ISP will typically allocate a /56 or sometimes a /48, although my hosted solution is only allocated a single /64 subnet.

If a /32 is USD 40.00, then a /40 will be about USD 0.16, and a /48 (which some ISPs allocate, enough for 65,000 subnets) would be about 1/16th of 1 US cent. My home ISP (Internode, Australia) only gives me a /56, which is only enough for 256 subnets, and would only cost 1/4,000th of 1 US cent.

That is the cost for 256 subnets, each with 16 quadrillion addresses.

Suffice to say, the current cost of an IPv6 address is effectively zero.

Addressing scheme

The first thing to think about for setting up docker with IPv6 only is a basic addressing scheme. My hosting company, Mythic Beasts, allocates a /64 subnet for each machine, which effectively costs them zero, but provides more than enough addresses to use.

For outgoing addresses, there are various privacy extensions that use random outbound addresses, but for a server you also need static addresses for inbound connections.

With a whole /64, it is relatively easy to use an address for the node itself, and then allocate a subrange of /112 for docker services to publish on.

For internal communications within docker, you can pick a Unique Local Range (ULA) e.g. by using a generator such as https://dnschecker.org/ipv6-address-generator.php

This range will be unique but, without anything special set up, not routeable over the Internet, so will by default be private.

Within your ULA network, you can allocate a subnet for docker on the machine, perhaps the same subnet as the public address, or something similar.

Then within the subnet you will want to allocate a range of addresses, e.g. /80 to the default bridge network, and then additional ranges for other internal networks, e.g. a /112.

Note that a /112 still allows 65,000 addresses to be allocated, e.g. for services and other containers.

IP address plan example

Plan Range	Description
`2001:db8:1234:a1::1`	Main public address of the host.
`2001:db8:1234:a1:1::/112`	Public CIDR for `demo_public`. Additional address to publish ports on.
`fd12:3456:789a::/56`	ULA network.
`fd12:3456:789a:8101::/64`	ULA subnet allocated for this docker.
`fd12:3456:789a:8101::/80`	Default bridge network address range. Use ULA to avoid exposing. Uses NAT66 for external connections.
`fd12:3456:789a:8101:1::/112`	Internal network `demo_internal`. Used to allocate internal container addresses. Does not need routing. Uses NAT66 for external connections.

Docker configuration

Install Docker CE:

You can use specific version, for consistency, or latest if you want.

On Ubuntu, https://docs.docker.com/engine/install/ubuntu/:

# Prerequisite packages
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg lsb-release

# Add Docker's official GPG key:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add the Docker apt repository:
echo deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker CE
sudo apt-get update
sudo apt-get install -y containerd.io=1.4.9-1 docker-ce=5:20.10.8~3-0~ubuntu-$(lsb_release -cs) docker-ce-cli=5:20.10.8~3-0~ubuntu-$(lsb_release -cs)

Install docker compose v2:

See https://docs.docker.com/compose/cli-command/:

DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
mkdir -p $DOCKER_CONFIG/cli-plugins
curl -SL https://github.com/docker/compose/releases/download/v2.2.3/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose

Enable IPv6 for OS and Docker

First of all you need to ensure IPv6 forwarding is enabled in your OS and the docker daemon is configured to use IPv6.

Enable IPv6 forwarding:

# Setup required sysctl IPv6 params, these persist across reboots.
cat <<EOF | sudo tee /etc/sysctl.d/90-ipv6.conf
net.ipv6.conf.all.forwarding        = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

Enable docker IPv6:

You also need to configure the default IPv6 network range (see IP address plan above). You can use the private range, which ensures no routing is possible. A ULA range is always available, even if you have only a single IPv6 address.

Also, disable userland-proxy, and use NAT66 instead for external access from the ULA range access (see below).

# Set up the Docker daemon
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  ipv6: true,
  fixed-cidr-v6: fd12:3456:789a:8101::/80,
  userland-proxy: false
}
EOF

systemctl restart docker

For more details see: https://medium.com/@skleeschulte/how-to-enable-ipv6-for-docker-containers-on-ubuntu-18-04-c68394a219a2

Docker compose configuration

Docker networks

Create a docker-compose.yml file for your machine (you may want to store it in source control), and set up the networks based on your plan:

version: '3.7'

networks:
  internal:
    enable_ipv6: true
    ipam:
      config:
        - subnet: fd12:3456:789a:8101:1::/112
  public:
    enable_ipv6: true
    ipam:
      config:
        - subnet: 2001:db8:1234:a1:1::/112

By default the public range, like any other public address range, is open to all traffic.

The public network should be protected by a firewall that has default deny incoming and default deny routing (forwarding), and only routes the traffic that you explictly want. If you don't have access to an external firewall, you can use Uncomplicated Firewall (UFW) to set up protection for your IPv6-only server. ### External access for ULA networks using NAT66

For external access, when using ULA, some form of proxying is required; rather than userland-proxy (which has issues), I recommend using IPv6 network address translation (NAT66) instead.

While you could argue that in this case NAT is not required — you could just use addresses from the public subnet, and block them from unwanted external connection via a firewall — there are situations, such as running a docker machine on a laptop, where the external range may change as the machine moves around, where having a stable internal range and using NAT66 can be useful.

There is a existing IPV6 nat container that you can easily run for this service, provided by Robbert Klarenbeek, details at https://github.com/robbertkl/docker-ipv6nat

Add the IPv6 NAT service to your docker-compose.yml file as a service:

services:

  ipv6nat:
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - SYS_MODULE
    cap_drop:
      - ALL
    image: robbertkl/ipv6nat
    network_mode: host
    restart: always
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock:ro'

This container doesn't do much itself, but use the permissions to set up the host ip6tables networking. Note that you could also set up rules manually in ip6tables, but using the container is easier.

Address translation is particularly useful if you only have a single IPv6 address, as it allows you to run containers on an internal network where outgoing calls are proxied to the Internet (although the economics of why you would only be given a single address seems strange). For incoming requests in this setup you need to use published ports on the main IPv6 address.

If you have a larger public range, then you can use public ranges (with appropriate firewall rules).

Neighbor Discovery Protocol Proxy Daemon

You will also need to advertise the public range you are using, so network discovery will know to route the addresses to the host machine.

You can do this manually for each individual address, but it is easier to install the Neighbour Discover Protocol Proxy Daemon by Daniel Adolfsson, https://github.com/DanielAdolfsson/ndppd/tree/1.0-devel, and have it advertise the entire range automatically.

# Install
sudo apt update
sudo apt install -y ndppd

# Create config
cat <<EOF | sudo tee /etc/ndppd.conf
proxy eth0 {
    rule 2001:db8:1234:a1:1::/112 {
        auto
    }
}
EOF

# Restart
sudo systemctl daemon-reload
sudo systemctl restart ndppd

# Set to run on boot
sudo systemctl enable ndppd

Side note on outbound access to IPv4

The above configuration is for enabling access to the rest of the IPv6 Internet, outbound through NAT66 or inbound through NDPPD. Your services running in docker may also need access to IPv4 resources.

This is no different from the IPv6 only host needing access to IPv4 and requires a solution such as a combination of DNS64 and NAT64, offered by your hosting company. The containers will then use IPv6 to connect to the DNS64 and NAT64 services, which will then connect to IPv4.

Initial testing

With IPv6 configured, your networks set up, and both NAT66 and NDPPD running, you can add a simple testing instance like busybox to your docker-compose.yml file to check things are working.

Pick a random address (4 hex characters) for the container, and for simplicity use the same value in both ranges, configuring the container on both the internal and public networks.

You can configure busybox to run a simple httpd daemon, and also publish to a port on the host machine.

You may also need to set extra_hosts with the address for host.docker.internal — as you are using IPv6 you know the unique value you are using is not going to conflict with any other addresses.

In your docker-compose.yml file this will look like:

  busybox:
    command: sh -c 'echo Hello world! > index.html && httpd -f -v'
    extra_hosts:
      - 'host.docker.internal:fd12:3456:789a:8101:1::1'
    image: busybox
    networks:
      internal:
        ipv6_address: 'fd12:3456:789a:8101:1::ab53'
      public:
        ipv6_address: '2001:db8:1234:a1:1::ab53'
    ports:
      - '8081:80'

Start the containers:

User docker compose to start the two containers in daemon mode, so that you can test connections.

docker compose -p demo up -d ipv6nat busybox

Testing connections

Test outbound connections:

docker exec demo-busybox-1 ping6 -c 4 google.com

If you are following best practice and have a firewall enabled (whether a separate device or running on the server) then you will need to make sure the relevant addresses and ports are allowed for incoming connections. See Securing your IPv6-only docker server for more details if you are running Uncomplicated Firewall (UFW) on the server. Once the firewall is configured, use a browser on another machine to test inbound connections direct to the container http://[2001:db8:1234:a1:1::ab53]. And also to the published port on the host http://[2001:db8:1234:a1::1]:8081. Note that you will need an IPv6 connection (dual stack is okay) to test these.

Test connection between containers on the internal ULA network by creating another temporary container:

docker run --rm -it --network demo_internal busybox wget -O - http://[fd12:3456:789a:8101:1::ab53]

For all of the above you can check the requests against the container logs:

docker logs demo-busybox-1

Troubleshooting

If ipv6nat is failing, check the logs:

docker logs demo-ipv6nat-1

It may show the error: exit status 3: modprobe: can't change directory to '/lib/modules': No such file or directory ip6tables v1.8.7 (legacy): can't initialize ip6tables table 'filter': Table does not exist (do you need to insmod?)

If so, try loading the module on the host via modprobe ip6table_filter, and if that works check cat /etc/modules and if it is not present, add it to load on reboot:

echo ip6table_filter | sudo tee -a /etc/modules

Next steps

One you have docker running services on your IPv6-only machine, with connections to and from the Internet, you can add the application services you want to run on the server, e.g. WordPress, and any back end supporting components, e.g. databases.

For web-based services, you may want to add a proxy service, such as Caddy, that will automatically take care of TLS certificate registration.

To allow access to your IPv6 only machine from IPv4 only clients, my hosting company Mythic Beasts offers an incoming reverse proxy service, or you can use a content distribution network like Cloudflare that includes reverse proxy services (I use this for https://www.gryphontechnology.com, hosted on the same box as my blog).

(Cloudflare can also be used to provide IPv6 access to your existing IPv4 only servers.)

Note that some containers may not be set up to work out of the box with IPv6, they may be set up for IPv4 only by default, or be expecting dual-stack, even if the underlying software does support IPv6. This means you may often need to modify configuration files and troubleshoot to get services working.

Running an IPv6-only host — redux(11 min read)