Apps on Kubernetes IPv6 – Kubeapps, WordPress

Once you have Kubernetes running on IPv6 only the next step is to install some apps.

This is my first post written on my new WordPress instance, hosted on Kubernetes IPv6 only. If you are reading it, then it is working

Of course apps have their own issues not being configured by default to work with IPv6, so for each app you need to test and work out what configuration details need to be tweaked (assuming the app supports IPv6 in the first place).

To start off with, I installed Kubeapps, to get an application management dashboard, and then used that to install WordPress.

With WordPress installed, I exported the content from my old blog and then imported it into the new instance, and tweaked a few WordPress settings.

The final step was to configure the Mythic Beasts reverse proxy, to make my blog available for legacy IPv4 users.

Installing Kubeapps

If you follow the quick start instructions, or the Getting Started walk through, then, of course, it doesn't work for IPv6, although it is a good place to start. https://github.com/kubeapps/kubeapps#quick-start

You have to click through to the detailed instructions and check the values.yaml file to see that one of the first configuration settings is enableIPv6 with a default value of false

You can just pass that on the command line, --set enableIPv6=true, but there were a few other settings I wanted to tweak, so I created a configuration override file and used that to install Kubeapps.

The instructions below use existing services Helm, Nginx Ingress, and cert-manager, already installed in the cluster.

Using HTTPS is best practice, so the first step was to set up a host name in DNS, pointing to my ingress service. With IPv6 I could point directly to the service, if I configure firewalls correctly, but I would need to know the IPv6 address first.

apps.example.com CNAME ingress.k1.gryphontechnology.biz.
ingress.k1.gryphontechnology.biz AAAA 2a00:1098:80:a1:8:3:0:d3ef

Using an already deployed ingress service, I already knew the IPv6 address to use, and the ingress service can co-ordinate with the certificate manager to issue a free Let's Encrypt certificate.

As well as configuring IPv6 and HTTPS ingress, I also reduced the minimum CPU requirements as I have a small server that only has limited use and want to also deploy other apps. You can check resource allocation with kubectl describe node.

Once DNS is set up, e.g. kubeapps.k1.example.com, add the repo, create a config, and use Helm to install Kubeapps:

helm repo add bitnami https://charts.bitnami.com/bitnami
kubectl create namespace kubeapps
 
cat <<EOF | tee kubeapps-config.yaml
enableIPv6: true
ingress:
  enabled: true
  certManager: true
  hostname: apps.example.com
  tls: true
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
postgresql:
  resources:
    requests:
      cpu: 100m
EOF
 
helm install kubeapps --namespace kubeapps -f kubeapps-config.yaml bitnami/kubeapps

Once deployed you can check that the Kubeapps components are running, and that cert-manager has created the needed certificate.

You can then access https://apps.example.com

To log in, you can generate a token:

kubectl create serviceaccount kubeapps-operator
kubectl create clusterrolebinding kubeapps-operator --clusterrole=cluster-admin --serviceaccount=default:kubeapps-operator
kubectl describe secret `kubectl get secret | grep kubeapps-operator | awk '{print $1}'`

Installing WordPress

With Kubeapps up and running, the Kubeapps console can be used to install WordPress, although, again, IPv6 does not work by default.

The issue was not with WordPress itself, but with the default configuration of the included MariaDB (MySQL), which was configured to only listen on IPv4. This means the components would install and each work independently, but not talk to each other.

Similar to the above, the first step is to set up DNS, using a CNAME to the same ingress:

sgryphon.gamertheory.net CNAME ingress.k1.gryphontechnology.biz.
ingress.k1.gryphontechnology.biz AAAA 2a00:1098:80:a1:8:3:0:d3ef

There are a lot of configuration options for WordPress, including the blog name, admin email, password, etc, and some need to be set in the advanced YAML view.

NOTE: Make sure to also give the deployment a meaningful name, which needs to be set in the main form, not the YAML.

wordpressUsername: xxxxxxxx
wordpressPassword: yyyyyyyy
wordpressEmail: sly@gamertheory.net
wordpressFirstName: Sly
wordpressLastName: Gryphon
wordpressBlogName: Software / Wetware

Similar to the above, best practice is to enable ingress with HTTPS and use cert-manager to automatically provision certificates.

ingress:
  enabled: true
  certManager: true
  hostname: sgryphon.gamertheory.net
  annotations:
    {
      cert-manager.io/cluster-issuer: letsencrypt-production
    }
  tls: true

As well as setting up ingress for HTTPS, I also configured the default scheme to HTTPS, and then also configured the liveness and readiness probes accordingly, so the check to port 8080 has a header indicating the protocol is HTTPS (otherwise the check will be rejected).

wordpressScheme: https
.
.
.
livenessProbe:
  httpGet:
    httpHeaders: [ { name: X-Forwarded-Proto, value: https } ]
readinessProbe:
  httpGet:
    httpHeaders: [ { name: X-Forwarded-Proto, value: https } ]

Configure MariaDB (MySQL) to list on IPv6

The important part for IPv6 was to ensure MySQL was listening on all addresses, not just IPv4.

To do this you have set the mariadb > primary > configuration property in the deployment settings to have the default configuration settings (for bitnami) except change bind-address to be :: (or *). You need to specify the whole file, even though only one line is changed from what we be generated.

mariadb:
  primary:
    ## Configure MariaDB Primary with a custom my.cnf file
    ## ref: https://mysql.com/kb/en/mysql/configuring-mysql-with-mycnf/#example-of-configuration-file
    ##
    configuration: |-
      [mysqld]
      skip-name-resolve
      explicit_defaults_for_timestamp
      basedir=/opt/bitnami/mariadb
      plugin_dir=/opt/bitnami/mariadb/plugin
      port=3306
      socket=/opt/bitnami/mariadb/tmp/mysql.sock
      tmpdir=/opt/bitnami/mariadb/tmp
      max_allowed_packet=16M
      bind-address=::
      pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid
      log-error=/opt/bitnami/mariadb/logs/mysqld.log
      character-set-server=UTF8
      collation-server=utf8_general_ci
      [client]
      port=3306
      socket=/opt/bitnami/mariadb/tmp/mysql.sock
      default-character-set=UTF8
      plugin_dir=/opt/bitnami/mariadb/plugin
      [manager]
      port=3306
      socket=/opt/bitnami/mariadb/tmp/mysql.sock
      pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid

As well as the WordPress, ingress, and MariaDB IPv6 settings, I also reduced the CPU minimum resource requirements:

resources:
  requests:
    cpu: 100m

I also added a storageClass to both the WordPress and MariaDB persistence sections, that I used to connect the persistent volume storage. In my case I was using some local volumes (see below), but this will vary depending on your deployed solution.

persistence:
  storageClass: local-storage
.
.
.
mariadb:
  primary:
    persistence:
      storageClass: local-storage

WordPress Storage

There are many storage options for Kubernetes. To start off with I used a basic local storage configuration.

Once WordPress was deployed (above), the database and web pods will be in a paused state waiting for storage to be available, with the status "pod has unbound immediate PersistentVolumeClaims".

I did not have automatic storage provisioning set up, so needed to manually add storage.

You can check the pod status (kubectl get pods --all-namespaces), specific pods (kubectl describe pod <pod-name>), and the two waiting storage claims (kubectl get PersistentVolumeClaim).

I used simple local storage, so first I created some folders:

mkdir /var/opt/k8s-data
mkdir /var/opt/k8s-data/wordpress-sgryphon-mariadb-pv
mkdir /var/opt/k8s-data/wordpress-sgryphon-pv

I then created a PersistentVolume matching the local-storage claim for MariaDB, which detected the change and completed the deployment. You need to configure the nodeAffinity with the hostname where you have the local storage.

cat <<EOF | tee wordpress-sgryphon-mariadb-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: wordpress-sgryphon-mariadb-pv
spec:
  capacity:
    storage: 8Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /var/opt/k8s-data/wordpress-sgryphon-mariadb-pv
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kube001.example.com
EOF
 
kubectl apply -f wordpress-sgryphon-mariadb-pv.yaml

To confirm MySQL is accessible, I used netcat to check the service port was open (this was part of the debugging to work out that it wasn't listening on IPv6).

1	`nc` `-z -v` `2001:db8:1234:5678:8:3:0:3456 3306`

Once the database is up and running, you can do similar for the front end storage:

cat <<EOF | tee wordpress-sgryphon-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: wordpress-sgryphon-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /var/opt/k8s-data/wordpress-sgryphon-pv
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kube001.example.com
EOF
 
kubectl apply -f wordpress-sgryphon-pv.yaml

Once both storage volumes are assigned and both pods are running you should be able to access WordPress admin on the configured URL via HTTPS, https://sgryphon.gamertheory.net/wp-admin/.

Migrating WordPress

Once WordPress was up and running, it was not too difficult to export from my old instance, and import to the new instance, via the WordPress management dashboard.

I also reconfigured my theme and customisations, and a few other plug ins, including:

ActivityPub – Support for the ActivityPub federation protocol, allowing others to follow (subscribe to) your blog.
Webmention – Support for the modern W3C standard for Pingbacks and Trackbacks.
Semantic-Linkbacks – Advanced formatting of webmentions, pingbacks, trackbacks, etc.
SyntaxHighlighter Evolved – For formatting code blocks.

IPv4 reverse proxy

Mythic Beasts provides an IPv4 reverse proxy, that will allow IPv4 only users to access my IPv6 only site.

In the hosting control panel I am able to register the third party domains I own (with several authentication methods available), and then configure services with the host name and the endpoint address (of my ingress service).

Then in my DNS I configure IPv4 A records from my blog host name to the Mythic Beasts reverse proxy address.

ingress.k1.gryphontechnology.biz A 93.93.129.174
ingress.k1.gryphontechnology.biz A 46.235.225.189

Once configured, I can test with an IPv4 only client (turn off IPv6) and make sure that it can display my blog (with the translation from IPv4 to IPv6 being done by the Mythic Beasts reverse proxy).

If you have an IPv4 address available you could do similar using either an external reverse proxy, or by assigning the address to your ingress service.

You are currently accessing this page from IP address 2a00:1098:0:80:1000:3b:1:1. If it is an IPv6 address, then it shows the IPv6 only deployment is working; if it is an IPv4 address, then it shows the reverse proxy is working. You can also test your IPv6 support at https://test-ipv6.com/.

Apps on Kubernetes IPv6 – Kubeapps, WordPress(8 min read)