Deploying OpenEverest on a 3-Node K3s Cluster

Part 1 of 2: From bare Ubuntu servers to a running MySQL cluster

OpenEverest is an open source platform for running databases such as MySQL, PostgreSQL and MongoDB on Kubernetes. You get a web console and an API, and Kubernetes operators handle provisioning, scaling, backups and failover behind the scenes.

This post builds a small but realistic lab from scratch: a 3-node K3s Kubernetes cluster on Ubuntu 26.04, OpenEverest on top of it, and a 3-node Percona XtraDB Cluster (PXC) for MySQL that you can connect to from your laptop. Every step explains what the command does and why it is needed.

Part 2 of this series looks inside the MySQL cluster: how to read Galera’s health indicators, and a live failover test where we kill the primary database pod while writes keep running.

What You Will Build

• K3s: a lightweight, CNCF-certified Kubernetes distribution. A single binary includes containerd (the container runtime), Flannel (pod networking), CoreDNS (service discovery) and the local-path storage provisioner.
• Helm: the Kubernetes package manager, used to install OpenEverest.
• OpenEverest: the control plane (everest-server for the UI/API, everest-operator for reconciling database clusters).
• Percona XtraDB Cluster: a 3-node, synchronously replicated MySQL cluster with HAProxy in front of it, created from the OpenEverest UI.

Server Inventory

Each node needs a unique hostname and a unique IP address. Kubernetes identifies nodes by hostname, so two VMs cloned from the same image with the same hostname will conflict.

The nodes must be able to reach each other on these ports:

Phase 1: Prepare All Three Nodes

Where to run: all three servers. Open three terminal windows and SSH into each one.

Step 1.1: Connect to Each Server

Open one SSH session per server so you can run the preparation commands side by side.

# Terminal 1 - Control plane ssh <USER>@<CONTROL_PLANE_IP> # Terminal 2 - Worker node 1 ssh <USER>@<WORKER1_IP> # Terminal 3 - Worker node 2 ssh <USER>@<WORKER2_IP>

Step 1.2: Install Packages and Disable Swap

This refreshes the package index and installs the tools the rest of the guide needs:

• curl and wget download the K3s and Helm installers.
• open-iscsi and nfs-common are not needed by local-path storage, but they are prerequisites for replicated storage such as Longhorn or NFS volumes. Installing them now makes it easy to switch storage later.

Swap is turned off because the kubelet expects it to be disabled by default. With swap on, memory limits on database pods become unpredictable. The sed command comments out the swap line in /etc/fstab so swap stays off after a reboot.

sudo apt-get update && sudo apt-get install -y curl wget apt-transport-https open-iscsi nfs-common sudo swapoff -a sudo sed -i '/ swap / s/^/#/g' /etc/fstab

Step 1.3: Enable Kernel Modules and IP Forwarding

Kubernetes networking relies on two kernel modules and three kernel settings:

• overlay: the filesystem driver containerd uses to stack container image layers.
• br_netfilter: lets iptables see traffic crossing the Linux bridge, which is how Kubernetes Services and network policies work.
• net.bridge.bridge-nf-call-iptables and ip6tables: send bridged IPv4/IPv6 traffic through iptables.
• net.ipv4.ip_forward: allows the node to route packets between pods on different nodes.

modprobe loads the modules now. The file in /etc/modules-load.d/ loads them again on every boot, and the file in /etc/sysctl.d/ does the same for the kernel settings.

sudo modprobe overlay sudo modprobe br_netfilter sudo tee /etc/modules-load.d/k8s.conf <<EOF overlay br_netfilter EOF sudo tee /etc/sysctl.d/kubernetes.conf <<EOF net.bridge.bridge-nf-call-iptables  = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward                 = 1 EOF sudo sysctl --system

Phase 2: Install the K3s Control Plane

Where to run: control plane only (10.50.55.155).

Step 2.1: Install the K3s Server

The official install script downloads K3s, installs it as a systemd service named k3s, and starts the Kubernetes control plane. INSTALL_K3S_EXEC passes extra flags to the server:

• –disable traefik: K3s ships with the Traefik ingress controller. This lab doesn’t use ingress, so we skip it to save resources.
• –node-ip 10.50.55.155: pins the address Kubernetes advertises for this node. This matters on servers with more than one network interface.
• –flannel-backend=vxlan: pod traffic between nodes is wrapped in VXLAN over UDP 8472. This is already the default; it is set explicitly for clarity.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik --node-ip 10.50.55.155 --flannel-backend=vxlan" sh -

Step 2.2: Let Your User Run kubectl

K3s writes its admin kubeconfig to /etc/rancher/k3s/k3s.yaml, which only root can read. These commands copy it to your home directory, make you the owner, and point KUBECONFIG at it, now and in every future shell.

mkdir -p $HOME/.kube sudo cp /etc/rancher/k3s/k3s.yaml $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config export KUBECONFIG=$HOME/.kube/config echo "export KUBECONFIG=$HOME/.kube/config" >> ~/.bashrc

Note: This is the only server where kubectl is configured. All later kubectl and helm commands in this series run on the control plane.

Step 2.3: Get the Node Join Token

Worker nodes need a shared secret to join the cluster. K3s stores it in node-token.

sudo cat /var/lib/rancher/k3s/server/node-token

Output (format)

K10<ca-hash>::server:<secret>

The token has two parts. K10<ca-hash> lets the joining node verify the server’s TLS certificate. The part after ::server: is the actual password.

Security: Treat the whole token as a credential. Never publish the part after ::server:. Anyone who has it can join a machine to your cluster.

Phase 3: Join the Worker Nodes

Where to run: each worker, in its own terminal. Replace <TOKEN> with the full token from Step 2.3.

When K3S_URL is set, the same install script runs in agent mode. The node connects to the API server at https://10.50.55.155:6443, authenticates with the token, and starts running pods. –node-ip again pins the address each node advertises.

Step 3.1: Join Worker Node 1

curl -sfL https://get.k3s.io | K3S_URL=https://10.50.55.155:6443 \   K3S_TOKEN="<TOKEN>" \   INSTALL_K3S_EXEC="--node-ip=10.50.55.156" sh -

Step 3.2: Join Worker Node 2

curl -sfL https://get.k3s.io | K3S_URL=https://10.50.55.155:6443 \   K3S_TOKEN="<TOKEN>" \   INSTALL_K3S_EXEC="--node-ip=10.50.55.157" sh -

Phase 4: Install OpenEverest

Where to run: control plane only (10.50.55.155).

Step 4.1: Verify All Nodes Are Ready

Ready means the kubelet on that node is healthy and has working pod networking. Workers show <none> under ROLES; that is normal for K3s agents.

kubectl get nodes

Output

NAME                              STATUS   ROLES           AGE     VERSION arunjithopeneverestworkernode1    Ready    <none>          4m6s    v1.36.5+k3s1 arunjithopeneverestworkernode2    Ready    <none>          3m18s   v1.36.5+k3s1 arunopeneverestcontrolplanenode   Ready    control-plane   9m3s    v1.36.5+k3s1

Note: In K3s, the control plane node also runs normal pods, including one of the database pods. This is fine for a test lab. In production, it is better to run database pods only on worker nodes.

kubectl get pods -n everest -o wide | grep pxc mysql-vjg-pxc-0                                    1/1     Running   0               75m   10.42.2.17   arunjithopeneverestworkernode2    <none>           <none> mysql-vjg-pxc-1                                    1/1     Running   0               10d   10.42.1.7    arunjithopeneverestworkernode1    <none>           <none> mysql-vjg-pxc-2                                    1/1     Running   0               10d   10.42.0.10   arunopeneverestcontrolplanenode   <none>           <none>

Step 4.2: Install Helm

Helm installs applications from versioned packages called charts. This script detects your OS and architecture and installs the helm binary to /usr/local/bin.

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Note: Piping a script from the internet into bash is convenient for a lab. For repeatable builds, pin a specific Helm release instead.

Step 4.3: Confirm the Default StorageClass

Databases need persistent volumes. When OpenEverest creates a database, it requests storage from the default StorageClass, which on K3s is local-path.

kubectl get storageclass

Output

NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  11m

What these columns mean for a database:

• WaitForFirstConsumer: the volume is created only after the pod is scheduled, so the data directory lands on the same node as the pod.
• Node-local data: each volume is a directory on one node’s disk. The pod is pinned to that node for its whole life. If the node is lost, the pod cannot move; Galera rebuilds the data on a new node from the surviving replicas.
• ALLOWVOLUMEEXPANSION false: you cannot grow these volumes later, so size them generously.
• RECLAIMPOLICY Delete: deleting the volume claim deletes the data.

If local-path is not marked (default), set it:

kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=true --overwrite

Step 4.4: Install OpenEverest with Helm

helm repo add registers the OpenEverest chart repository, and helm repo update downloads its index. helm install creates a release named everest in a new namespace, everest-system, which holds the OpenEverest control plane.

helm repo add openeverest https://openeverest.github.io/helm-charts/ helm repo update helm install everest openeverest/openeverest \   --namespace everest-system \   --create-namespace

Step 4.5: Verify the OpenEverest Pods

Two pods should reach Running within 1 to 2 minutes:

• everest-server: serves the web console and REST API.
• everest-operator: watches OpenEverest resources and turns them into database clusters by driving the Percona operators.

kubectl get pods -n everest-system

Output

NAME                                READY   STATUS    RESTARTS   AGE everest-operator-589d68c4cf-4hbss   1/1     Running   0          115s everest-server-846b99fcd-2jtd9      1/1     Running   0          115s

Step 4.6: Retrieve the Admin Password

OpenEverest stores its user accounts in a Kubernetes Secret. Secret values are base64-encoded, so the command extracts the users.yaml key and decodes it. The output contains the admin user’s initial password.

kubectl get secret everest-accounts -n everest-system -o jsonpath='{.data.users\.yaml}' | baseCommand (on your laptop) ssh -L 8080:127.0.0.1:8080 arunjith.aravindan@10.50.55.155

Phase 5: Open the OpenEverest Web Console

The everest Service is only reachable inside the cluster. To use it from your laptop, chain two forwards: an SSH tunnel from your laptop to the control plane, and a kubectl port-forward from the control plane into the cluster.

Step 5.1: Open an SSH Tunnel from Your Laptop

-L 8080:127.0.0.1:8080 means: anything that connects to port 8080 on my laptop is sent through SSH to port 8080 on the control plane’s loopback interface.

ssh -L 8080:127.0.0.1:8080 arunjith.aravindan@10.50.55.155

Step 5.2: Port-Forward the UI Service

In the SSH session you just opened, forward local port 8080 on the control plane to the everest Service. Leave this running.

kubectl port-forward svc/everest 8080:8080 -n everest-system

Security: kubectl port-forward listens on 127.0.0.1 by default, which is exactly where the SSH tunnel delivers traffic. Avoid –address 0.0.0.0: it exposes the UI to every host that can reach the control plane.

Step 5.3: Log In

Open http://localhost:8080 in your browser and log in as admin with the password from Step 4.6.

Step 5.4: Create a MySQL Database

In the console, create a new MySQL database with 3 nodes. OpenEverest creates a Percona XtraDB Cluster with three PXC pods and HAProxy in front of them, in the everest namespace. It generates a name for the cluster; in this lab it is mysql-vjg, so substitute your own cluster name in the commands that follow.

Phase 6: Connect to MySQL

Step 6.1: Understand the Kubernetes Services

Before connecting, look at what OpenEverest created. Each Service is a stable network endpoint in front of a group of pods.

kubectl get svc -n everest | grep mysql-vjg

Output

mysql-vjg-haproxy            ClusterIP   10.43.69.73   <none>   3306/TCP,3309/TCP,33062/TCP,33060/TCP,8404/TCP   44m mysql-vjg-haproxy-replicas   ClusterIP   10.43.2.25    <none>   3306/TCP                                         44m mysql-vjg-pxc                ClusterIP   None          <none>   3306/TCP,33062/TCP,33060/TCP                     44m mysql-vjg-pxc-unready        ClusterIP   None          <none>   3306/TCP,33062/TCP,33060/TCP                     44m

Inside the cluster, CoreDNS resolves mysql-vjg-haproxy (or the full form mysql-vjg-haproxy.everest.svc.cluster.local) to the Service. That name does not resolve from your laptop, which is why the next steps use port forwarding.

Step 6.2: Get the MySQL Root Password

OpenEverest stores the database credentials in a Secret named everest-secrets-<cluster-name>. This command decodes the root key into a shell variable, so the password never appears on screen or in your shell history.

export ROOT_PW=$(kubectl get secret everest-secrets-mysql-vjg -n everest -o jsonpath='{.data.root}' | base64 --decode) echo "$ROOT_PW"   # print it once if you need to type it on your laptop

Step 6.3: Port-Forward HAProxy on the Control Plane

This forwards port 3306 on the control plane’s loopback interface to the HAProxy Service. Keep this terminal open.

kubectl port-forward svc/mysql-vjg-haproxy 3306:3306 -n everest

Output

Forwarding from 127.0.0.1:3306 -> 3306 Forwarding from [::1]:3306 -> 3306 Handling connection for 3306

Step 6.4: Tunnel from Your Laptop

Port 3307 here is just a local port on your laptop, chosen so it doesn’t clash with a MySQL server you may already run on 3306. The tunnel sends it to port 3306 on the control plane, where the port-forward from Step 6.3 picks it up.

ssh -L 3307:127.0.0.1:3306 arunjith.aravindan@10.50.55.155

Step 6.5: Connect with the MySQL Client

In another laptop terminal, connect to the local end of the tunnel. -p with no value makes the client prompt for the password, so it is not saved in your shell history.

mysql -h 127.0.0.1 -P 3307 -u root -p

The full path of the connection is: laptop port 3307, then SSH, then control plane port 3306, then kubectl port-forward, then HAProxy, then the current writer pod.

Step 6.6: Check the Cluster Status

These queries show Galera’s replication status. Part 2 explains every value; for now, look for Synced, Primary and a cluster size of 3.

SQL

SHOW STATUS LIKE '%wsrep%stat%'; SHOW STATUS LIKE '%wsrep%siz%';

Output

+------------------------------+--------------------------------------+ | Variable_name                | Value                                | +------------------------------+--------------------------------------+ | wsrep_local_state_uuid       | c1319eb6-d354-11f1-8a79-0ff80b37e565 | | wsrep_monitor_status (L/A/C) | [ (20, 20), (44, 44), (44, 44) ]     | | wsrep_flow_control_status    | OFF                                  | | wsrep_local_state            | 4                                    | | wsrep_local_state_comment    | Synced                               | | wsrep_ist_receive_status     |                                      | | wsrep_evs_state              | OPERATIONAL                          | | wsrep_cluster_state_uuid     | c1319eb6-d354-11f1-8a79-0ff80b37e565 | | wsrep_cluster_status         | Primary                              | +------------------------------+--------------------------------------+

+------------------------+-------+ | Variable_name          | Value | +------------------------+-------+ | wsrep_cert_index_size  | 1     | | wsrep_gcache_pool_size | 15624 | | wsrep_cluster_size     | 3     | +------------------------+-------+

Step 6.7: See Which Node You Are Connected To

@@hostname and @@wsrep_node_name show which PXC pod HAProxy picked for this connection.

SQL

SELECT @@hostname, @@wsrep_node_name, @@read_only;

Output

+-----------------+-------------------+-------------+ | @@hostname      | @@wsrep_node_name | @@read_only | +-----------------+-------------------+-------------+ | mysql-vjg-pxc-0 | mysql-vjg-pxc-0   |           0 | +-----------------+-------------------+-------------+

Notice @@read_only = 0. In Percona XtraDB Cluster every node accepts writes; there are no read-only replicas as in asynchronous replication. So why does HAProxy send all writes to one node?

Galera checks each transaction for conflicts at commit time (certification). If two nodes change the same row at the same moment, one of the transactions fails certification and the application gets ERROR 1213 (40001): Deadlock found when trying to get lock, even though no real lock wait happened. Sending all writes to a single node avoids these conflicts. The -replicas endpoint is meant for reads only by convention, not because the nodes are read-only.

Step 6.8: Open a MySQL Shell Inside a Specific Pod

To talk to one particular node without HAProxy, run the MySQL client inside that pod. A PXC pod has more than one container, so -c pxc selects the database container. -it gives you an interactive terminal, and the client prompts for the root password.

percona@arunopeneverestcontrolplanenode:~$ kubectl exec -it mysql-vjg-pxc-2 -n everest -c pxc -- mysql -u root -p

SQL

SELECT @@hostname, @@wsrep_node_name, @@read_only;

Output

+-----------------+-------------------+-------------+ | @@hostname      | @@wsrep_node_name | @@read_only | +-----------------+-------------------+-------------+ | mysql-vjg-pxc-2 | mysql-vjg-pxc-2   |           0 | +-----------------+-------------------+-------------+

Summary and What’s Next

You now have a 3-node K3s cluster running OpenEverest and a 3-node Percona XtraDB Cluster behind HAProxy, reachable from your laptop. Along the way you saw how K3s nodes join a cluster, why local-path storage pins database pods to nodes, and how HAProxy, headless Services and CoreDNS fit together.

In Part 2, we read Galera’s health indicators in detail and run a live failover test: we delete the writer pod while a client writes every second, then watch HAProxy switch writers, the pod rejoin with an incremental state transfer, and all three nodes end up with identical data.

The post Deploying OpenEverest on a 3-Node K3s Cluster appeared first on Percona.