Kubernetes Cluster Installation, Certificate Management and Backup/Restore Operations with RKE 2

Hello friends,

In our earlier article "Rancher Longhorn Distributed Storage Installation, Backup and Restore Operations", we had set up a Kubernetes cluster with RKE 1. The name RKE 2 may give us the impression that it's the next version of RKE 1, but it's a completely different deployment tool. If we look at the differences

One of the most important differences is that it doesn't use Docker. RKE 1 does the deployment by building the whole setup on Docker.
If you look at the CNCF website, you can see that its other name is RKE Goverment. It offers more security security compared to RKE 1. It's actually a deployment tool first developed for US government agencies.
With its default configuration, it's able to pass the CIS (Center for Internet Security) Kubernetes benchmark.
It's scanned with Trivy on every build.
It's coded in accordance with US government security standards (FIPS 140-2).

Suse's goal is to offer a tool for migrating from RKE 1 to RKE 2 once enough development has been completed. You can also read about the topic on this page.

Kubernetes Installation with RKE 2

 # -*- mode: ruby -*- # vi: set ft=ruby : # enter a number between 1-9 MASTER_COUNT = 3 WORKER_COUNT = 3 MASTER_IP_PREFIX = "192.168.56.4" WORKER_IP_PREFIX = "192.168.56.5" PROVIDER = "virtualbox" BASTION_IP = "192.168.56.31" ROOT_PASS = "rkeadmin" ENV['VAGRANT_DEFAULT_PROVIDER'] = PROVIDER $allscript = <<-ALLSCRIPT MASTER_COUNT=$1 WORKER_COUNT=$2 MASTER_IP_PREFIX=$3 WORKER_IP_PREFIX=$4 BASTION_IP=$5 ROOT_PASS=$6 echo master count : $MASTER_COUNT echo worker count : $WORKER_COUNT echo master ip prefix : $MASTER_IP_PREFIX echo master worker prefix : $WORKER_IP_PREFIX echo bastion ip : $BASTION_IP echo root pass : $ROOT_PASS sudo apt update && sudo apt upgrade -y echo "===================== root user is being activated =====================" sudo su echo -e "$ROOT_PASS\n$ROOT_PASS" | passwd root echo "export TERM=xterm" | tee -a /etc/bash.bashrc echo "===================== swap is being disabled ==========================" sed -i "/swap/d" /etc/fstab swapoff -a systemctl disable --now ufw echo "=================== sysctl is being updated for the kubernetes network ==============" modprobe overlay modprobe br_netfilter cat <<EOF | sudo tee -a /etc/sysctl.d/kubernetes.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF sysctl --system echo "============= ssh settings are being made =============" sed -i "s/#PasswordAuthentication yes/PasswordAuthentication yes/" /etc/ssh/sshd_config echo "PermitRootLogin yes" | sudo tee -a /etc/ssh/sshd_config systemctl reload sshd echo "===================== host files are being edited ===================" for i in $(seq 1 $MASTER_COUNT); do echo "$MASTER_IP_PREFIX$i master$i" >>/etc/hosts done for i in $(seq 1 $WORKER_COUNT); do echo "$WORKER_IP_PREFIX$i worker$i" >>/etc/hosts done echo "$BASTION_IP bastion" >>/etc/hosts echo "$BASTION_IP rke.kubernetes.local" >>/etc/hosts ALLSCRIPT $bastionscript = <<-BASTION MASTER_COUNT=$1 WORKER_COUNT=$2 MASTER_IP_PREFIX=$3 WORKER_IP_PREFIX=$4 BASTION_IP=$5 ROOT_PASS=$6 echo master count : $MASTER_COUNT echo worker count : $WORKER_COUNT echo master ip prefix : $MASTER_IP_PREFIX echo master worker prefix : $WORKER_IP_PREFIX echo bastion ip : $BASTION_IP echo root pass : $ROOT_PASS sudo apt install sshpass -y sudo su ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa for i in $(seq 1 $MASTER_COUNT); do sshpass -p $ROOT_PASS ssh-copy-id -i ~/.ssh/id_rsa.pub -o StrictHostKeyChecking=no root@$MASTER_IP_PREFIX$i echo "yes \n" | scp ~/.ssh/id_rsa root@$MASTER_IP_PREFIX$i:.ssh/ echo key copied: $MASTER_IP_PREFIX$i done for i in $(seq 1 $WORKER_COUNT); do sshpass -p $ROOT_PASS ssh-copy-id -i ~/.ssh/id_rsa.pub -o StrictHostKeyChecking=no root@$WORKER_IP_PREFIX$i echo "yes \n" | scp ~/.ssh/id_rsa root@$WORKER_IP_PREFIX$i:.ssh/ echo key copied: $WORKER_IP_PREFIX$i done BASTION Vagrant.configure("2") do |config| # master (1..MASTER_COUNT).each do |i| config.vm.define "master#{i}" do |master| master.vm.box = "generic/ubuntu2004" master.vm.network "private_network", ip: MASTER_IP_PREFIX+"#{i}" master.vm.hostname = "master#{i}" master.vm.provider :PROVIDER do |lv| lv.memory=2048 lv.cpus=2 end end end # worker (1..WORKER_COUNT).each do |i| config.vm.define "worker#{i}" do |worker| worker.vm.box = "generic/ubuntu2004" worker.vm.network "private_network", ip: WORKER_IP_PREFIX+"#{i}" worker.vm.hostname = "worker#{i}" worker.vm.provider :PROVIDER do |lv| lv.memory=2048 lv.cpus=2 lv.storage :file, :size => '10G' end end end # bastion config.vm.define "bastion" do |bastion| bastion.vm.box = "generic/ubuntu2004" bastion.vm.network "private_network", ip: BASTION_IP bastion.vm.hostname = "bastion" bastion.vm.provider :PROVIDER do |lv| lv.memory=2048 lv.cpus=2 end bastion.vm.provision "shell", run: "once", inline: $bastionscript, args: [MASTER_COUNT,WORKER_COUNT, MASTER_IP_PREFIX,WORKER_IP_PREFIX, BASTION_IP, ROOT_PASS] end # globally defined shell provision. It will run on all vms config.vm.provision :shell, run: "once" do |s| s.args = [MASTER_COUNT,WORKER_COUNT, MASTER_IP_PREFIX,WORKER_IP_PREFIX, BASTION_IP, ROOT_PASS] s.inline = $allscript end end

Nginx External Load Balancer Installation

We'll do all our installation operations on our bastion server.This server will also act as the load balancer. For this, we'll install Nginx. After logging in to the bastion server, we switch to the root user and start the installation.

The ports used by the services.

sudo su # apt update && apt upgrade -y apt install nginx mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak cat << EOF > /etc/nginx/nginx.conf worker_processes 4; worker_rlimit_nofile 40000; #to be able to use stream, we load the strem module among the dynamic modules include /usr/share/nginx/modules/*.conf; load_module /usr/lib/nginx/modules/ngx_stream_module.so; events { worker_connections 8192; } # we can use it because we loaded the stream module. stream { upstream master_backend { least_conn; server master1:9345 max_fails=3 fail_timeout=5s; server master1:9345 max_fails=3 fail_timeout=5s; server master1:9345 max_fails=3 fail_timeout=5s; } server { listen 9345; proxy_pass master_backend; } upstream kube_api { least_conn; server master1:6443 max_fails=3 fail_timeout=5s; server master2:6443 max_fails=3 fail_timeout=5s; server master3:6443 max_fails=3 fail_timeout=5s; } server { listen 6443; proxy_pass kube_api; } upstream rancher_web { least_conn; server master1:80 max_fails=3 fail_timeout=5s; server master1:80 max_fails=3 fail_timeout=5s; server master1:80 max_fails=3 fail_timeout=5s; } server { listen 80; proxy_pass rancher_web; } upstream rancher_web_https { least_conn; server master1:443 max_fails=3 fail_timeout=5s; server master1:443 max_fails=3 fail_timeout=5s; server master1:443 max_fails=3 fail_timeout=5s; } server { listen 443; proxy_pass rancher_web_https; } } EOF systemctl status nginx

RKE2 Server Installation

When we look at the RKE 2 documentation, the server concept refers to Kubernetes Master nodes, while the agent concept refers to Kubernetes worker nodes.

First, we'll install one master and one worker node each, and then add the other nodes.

While still as the root user on the bastion server, we switch to the master1 node and run the server installation script.

Source

We'll deliberately install an old version and upgrade it later.

You can follow the versions on the official Github page.

For server configuration references, visit this page.

 ssh root@master1 curl -sfL https://get.rke2.io --output install.sh chmod +x install.sh INSTALL_RKE2_CHANNEL="stable" INSTALL_RKE2_TYPE="server" INSTALL_RKE2_VERSION="v1.22.4+rke2r2" sh install.sh mkdir -p /etc/rancher/rke2 cat << EOF > /etc/rancher/rke2/config.yaml write-kubeconfig-mode: "0644" tls-san: - rke.kubernetes.local - 192.168.56.31 cni: "calico" cluster-domain: "rke.kubernetes.local" node-ip: "192.168.56.41" EOF # first enable, then start, like this systemctl enable rke2-server systemctl start rke2-server

We can find the kubectl tool in the "/var/lib/rancher/rke2/bin" directory. Also, to completely remove RKE 2 from the system, we can use the "rke2-killall.sh" and "rke2-uninstall.sh" scripts in the "/usr/local/bin" directory.

export PATH=$PATH:/var/lib/rancher/rke2/bin export KUBECONFIG=/etc/rancher/rke2/rke2.yaml source ~/.bashrc

you can use the command below to follow the logs

journalctl -u rke2-server -f

When we test it, we can see that the master1 node has come up. Our version number also appears as v1.22.4.

kubectl get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master1 Ready control-plane,etcd,master 112s v1.22.4+rke2r2 192.168.56.41 <none> Ubuntu 20.04.5 LTS 5.4.0-132-generic containerd://1.5.8-k3s1

To be able to add the other master and worker nodes, we'll need the node-token information.

cat /var/lib/rancher/rke2/server/node-token # you should see a token like the one below K1033e47d23de84c5fc84e596090feac30fca35c2a31d89b609ace433720bcc3692::server:47ae5521c9ba374917118eefb9f66c50

RKE2 Agent Installation

As we also mentioned above, in the RKE 2 installation the agent concept means a worker node in Kubernetes.

While as the root user on our bastion server, we ssh to the worker1 node as root. Then we move on to the installation.

 ssh root@worker1 curl -sfL https://get.rke2.io --output install.sh chmod +x install.sh INSTALL_RKE2_CHANNEL="stable" INSTALL_RKE2_VERSION="v1.22.4+rke2r2" INSTALL_RKE2_TYPE="agent" sh install.sh mkdir -p /etc/rancher/rke2 cat << EOF > /etc/rancher/rke2/config.yaml server: https://rke.kubernetes.local:9345 token: K1033e47d23de84c5fc84e596090feac30fca35c2a31d89b609ace433720bcc3692::server:47ae5521c9ba374917118eefb9f66c50 node-ip: "192.168.56.51" EOF systemctl enable rke2-agent systemctl start rke2-agent

you can use the command below to follow the logs

journalctl -u rke2-agent -f

if you happen to get an error and want to start the operations over, we can use the "rke2-killall.sh" and "rke2-uninstall.sh" scripts in the "/usr/local/bin" directory. But if the worker node has been registered to kubernetes, that is, if you can see it in the node list, then you also need to delete it on the Kubernetes side.

kubectl delete node worker1

Other Server and Agent Installations

From now on, we do all the server and agent installations we'll do in the same way as the agent installation I did above. Only the configuration file will be different.

Installation script for the server

Don't forget to change the node ip address. Unlike the first server we installed, we added the token.

ssh root@master2 curl -sfL https://get.rke2.io --output install.sh chmod +x install.sh INSTALL_RKE2_CHANNEL="stable" INSTALL_RKE2_TYPE="server" INSTALL_RKE2_VERSION="v1.22.4+rke2r2" sh install.sh mkdir -p /etc/rancher/rke2 cat << EOF > /etc/rancher/rke2/config.yaml server: https://rke.kubernetes.local:9345 tls-san: - rke.kubernetes.local - 192.168.56.31 token: K1033e47d23de84c5fc84e596090feac30fca35c2a31d89b609ace433720bcc3692::server:47ae5521c9ba374917118eefb9f66c50 cni: "calico" cluster-domain: "rke.kubernetes.local" node-ip: "192.168.56.42" EOF # first enable, then start, like this systemctl enable rke2-server systemctl start rke2-server

Installation script for the agent

ssh root@worker2 curl -sfL https://get.rke2.io --output install.sh chmod +x install.sh INSTALL_RKE2_CHANNEL="stable" INSTALL_RKE2_VERSION="v1.22.4+rke2r2" INSTALL_RKE2_TYPE="agent" sh install.sh mkdir -p /etc/rancher/rke2 cat << EOF > /etc/rancher/rke2/config.yaml server: https://rke.kubernetes.local:9345 token: K1033e47d23de84c5fc84e596090feac30fca35c2a31d89b609ace433720bcc3692::server:47ae5521c9ba374917118eefb9f66c50 node-ip: "192.168.56.52" EOF systemctl enable rke2-agent systemctl start rke2-agent

After the installations are finished, let's get the list of nodes on our master1 server

kubectl get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master1 Ready control-plane,etcd,master 10h v1.22.4+rke2r2 192.168.56.41 <none> Ubuntu 20.04.5 LTS 5.4.0-135-generic containerd://1.5.8-k3s1 master2 Ready control-plane,etcd,master 31m v1.22.4+rke2r2 192.168.56.42 <none> Ubuntu 20.04.5 LTS 5.4.0-135-generic containerd://1.5.8-k3s1 master3 Ready control-plane,etcd,master 22m v1.22.4+rke2r2 192.168.56.43 <none> Ubuntu 20.04.5 LTS 5.4.0-135-generic containerd://1.5.8-k3s1 worker1 Ready <none> 7h37m v1.22.4+rke2r2 192.168.56.51 <none> Ubuntu 20.04.5 LTS 5.4.0-135-generic containerd://1.5.8-k3s1 worker2 Ready <none> 108m v1.22.4+rke2r2 192.168.56.52 <none> Ubuntu 20.04.5 LTS 5.4.0-135-generic containerd://1.5.8-k3s1 worker3 Ready <none> 96m v1.22.4+rke2r2 192.168.56.53 <none> Ubuntu 20.04.5 LTS 5.4.0-135-generic containerd://1.5.8-k3s1

Let's publish an application to test it.

 kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 2 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 EOF kubectl expose deployment nginx-deployment --type=NodePort --port=80

we run the command below to get the port number

 kubectl get svc # nginx-deployment NodePort 10.43.159.62 <none> 80:30016/TCP 44s

we test it with curl.We can see that a response came from Nginx.

 curl worker1:30016 # <!DOCTYPE html> # <html> # <head> # <title>Welcome to nginx!</title> # <style> # html { color-scheme: light dark; } # body { width: 35em; margin: 0 auto; # font-family: Tahoma, Verdana, Arial, sans-serif; } # </style> # </head> # <body> # <h1>Welcome to nginx!</h1> # <p>If you see this page, the nginx web server is successfully installed and # working. Further configuration is required.</p> # <p>For online documentation and support please refer to # <a href="http://nginx.org/">nginx.org</a>.<br/> # Commercial support is available at # <a href="http://nginx.com/">nginx.com</a>.</p> # <p><em>Thank you for using nginx.</em></p> # </body> # </html>

We were running our kubectl commands on master1. To run them on our bastion server, we need to copy our kube.config file and install the kubectl cli. While on the bastion server, we run the commands below.

 sudo su curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" chmod +x kubectl cp ./kubectl /usr/local/bin/ apt install bash-completion source /usr/share/bash-completion/bash_completion kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null mkdir -p $HOME/.kube scp root@master1:/etc/rancher/rke2/rke2.yaml $HOME/.kube/config

Kubernetes Upgrade with RKE2

On All Servers

curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=latest sh - systemctl restart rke2-server

When we upgrade the servers, we can see that they've moved to the latest version.

NAME STATUS ROLES AGE VERSION master1 Ready control-plane,etcd,master 12h v1.26.0+rke2r1 master2 Ready control-plane,etcd,master 132m v1.26.0+rke2r1 master3 Ready control-plane,etcd,master 122m v1.26.0+rke2r1 worker1 Ready <none> 9h v1.22.4+rke2r2 worker2 Ready <none> 3h28m v1.22.4+rke2r2 worker3 Ready <none> 3h17m v1.22.4+rke2r2

On All Agents

 curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=latest sh - systemctl restart rke2-agent

When we upgrade the agents too, we can see that they've moved to the latest version.

master1 Ready control-plane,etcd,master 13h v1.26.0+rke2r1 master2 Ready control-plane,etcd,master 156m v1.26.0+rke2r1 master3 Ready control-plane,etcd,master 147m v1.26.0+rke2r1 worker1 Ready <none> 9h v1.26.0+rke2r1 worker2 Ready <none> 3h52m v1.26.0+rke2r1 worker3 NotReady <none> 3h41m v1.26.0+rke2r1

Renewing Certificates

We list the certificates on the Master1 server. By default, certificates are valid for 1 year. If 90 days or less are left before the certificates expire, RKE2 renews them itself.

If we want to do it manually ourselves, we should follow the steps below. To renew the certificates, we first stop the rke2-server service. We renew the certificates and restart it.

Let's start working on our Master1 server.

 systemctl stop rke2-server rke2 certificate rotate systemctl start rke2-server

When we check, we can see that all the certificates have been renewed.

 ls -lsh /var/lib/rancher/rke2/server/tls total 124K 4.0K -rw-r--r-- 1 root root 1.2K Jan 3 18:12 client-admin.crt 4.0K -rw------- 1 root root 227 Jan 3 18:12 client-admin.key 4.0K -rw-r--r-- 1 root root 1.2K Jan 3 18:12 client-auth-proxy.crt 4.0K -rw------- 1 root root 227 Jan 3 18:12 client-auth-proxy.key 4.0K -rw-r--r-- 1 root root 574 Jan 2 10:10 client-ca.crt 4.0K -rw------- 1 root root 227 Jan 2 10:10 client-ca.key 4.0K -rw-r--r-- 1 root root 1.2K Jan 3 18:12 client-controller.crt 4.0K -rw------- 1 root root 227 Jan 3 18:12 client-controller.key 4.0K -rw-r--r-- 1 root root 1.2K Jan 3 18:12 client-kube-apiserver.crt 4.0K -rw------- 1 root root 227 Jan 3 18:12 client-kube-apiserver.key 4.0K -rw------- 1 root root 227 Jan 3 18:12 client-kubelet.key 4.0K -rw-r--r-- 1 root root 1.2K Jan 3 18:12 client-kube-proxy.crt 4.0K -rw------- 1 root root 227 Jan 3 18:12 client-kube-proxy.key 4.0K -rw-r--r-- 1 root root 1.2K Jan 3 18:12 client-rke2-cloud-controller.crt 4.0K -rw------- 1 root root 227 Jan 3 18:12 client-rke2-cloud-controller.key 4.0K -rw-r--r-- 1 root root 1.2K Jan 3 18:12 client-rke2-controller.crt 4.0K -rw------- 1 root root 227 Jan 3 18:12 client-rke2-controller.key 4.0K -rw-r--r-- 1 root root 1.2K Jan 3 18:12 client-scheduler.crt 4.0K -rw------- 1 root root 227 Jan 3 18:12 client-scheduler.key 8.0K -rw-r--r-- 1 root root 4.9K Jan 3 18:13 dynamic-cert.json 4.0K drwxr-xr-x 2 root root 4.0K Jan 3 18:12 etcd 4.0K -rw-r--r-- 1 root root 591 Jan 2 10:10 request-header-ca.crt 4.0K -rw------- 1 root root 227 Jan 2 10:10 request-header-ca.key 4.0K -rw-r--r-- 1 root root 574 Jan 2 10:10 server-ca.crt 4.0K -rw------- 1 root root 227 Jan 2 10:10 server-ca.key 4.0K -rw------- 1 root root 1.7K Jan 2 10:10 service.key 4.0K -rw-r--r-- 1 root root 1.4K Jan 3 18:12 serving-kube-apiserver.crt 4.0K -rw------- 1 root root 227 Jan 3 18:12 serving-kube-apiserver.key 4.0K -rw------- 1 root root 227 Jan 3 18:12 serving-kubelet.key 4.0K drwx------ 2 root root 4.0K Jan 2 10:10 temporary-certs

ETCD Backup Restore Operations with RKE2

By default, rke2 uses the "/var/lib/rancher/rke2" directory as the data folder. But it's possible to change this with the data-dir parameter in the configuration file.

Snapshot files are also put at the "/var/lib/rancher/rke2/server/db/snapshots" address, but it's possible to change that too with the etcd-snapshot-dir parameter. By the way, it's also possible to use the S3 API for backup operations.

To get more information about the configurations, check this address.

For backup restore, we'll first deploy an application and then take a backup. After deleting the application, we'll restore it again and check whether the application comes back.

First, let's get the list of pods under the default namespace. The pods of the nginx deployment I created earlier are visible.

 kubectl get pods # NAME READY STATUS RESTARTS AGE # nginx-deployment-585449566-nxt7t 1/1 Running 1 (3h59m ago) 20h # nginx-deployment-585449566-s5ln4 1/1 Running 3 (4h ago) 22h

Now let's bring up another nginx.

 kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-before-backup spec: selector: matchLabels: app: nginx replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 EOF

We check, and we can see the pods we just added.

kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-585449566-nxt7t 1/1 Running 1 (4h3m ago) 20h nginx-deployment-585449566-s5ln4 1/1 Running 3 (4h3m ago) 22h nginx-deployment-before-backup-6b7f675859-2djqq 1/1 Running 0 8s nginx-deployment-before-backup-6b7f675859-5hl6l 1/1 Running 0 8s nginx-deployment-before-backup-6b7f675859-jxq5n 1/1 Running 0 8s

Before deleting the deployment, we take a backup.

While on Master1, we can take a backup with the command below. By the way, there's no need to stop the system.

rke2 etcd-snapshot save --name before-delete-deployment

let's look at the snapshot list. The one we took most recently is visible at the very top.

 rke2 etcd-snapshot ls Name Location Size Created before-delete-deployment-master1-1672776097 file:///var/lib/rancher/rke2/server/db/snapshots/before-delete-deployment-master1-1672776097 8241184 2023-01-03T20:01:37Z etcd-snapshot-master1-1672660800 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-master1-1672660800 7938080 2023-01-02T12:00:00Z pre-upgrade-snapshot-master1-1672774069 file:///var/lib/rancher/rke2/server/db/snapshots/pre-upgrade-snapshot-master1-1672774069 8241184 2023-01-03T19:27:49Z

Now let's delete the deployment. For this, let's look at our deployment list.

 kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE nginx-deployment 2/2 2 2 22h nginx-deployment-before-backup 3/3 3 3 23m

we can delete both of them.

kubectl delete deployment nginx-deployment nginx-deployment-before-backup

When we check, we can see that it has been deleted.

kubectl get deployment No resources found in default namespace.

To bring the system back, we stop the RKE2 service on all master nodes with the command below.

 systemctl stop rke2-server

Then we reset the cluster with the snapshot we took on master1.

 rke2 server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/rke2/server/db/snapshots/before-delete-deployment-master1-1672776097

When we do this operation, the db from before the restore is backed up to the "/var/lib/rancher/rke2/server/db/etcd-old-%date%/" address.

ls -lhs /var/lib/rancher/rke2/server/db/ total 12K 4.0K drwx------ 3 root root 4.0K Jan 3 20:10 etcd 4.0K drwx------ 3 root root 4.0K Jan 3 18:12 etcd-old-1672776630 0 -rw------- 1 root root 0 Jan 3 20:10 reset-flag 4.0K drwx------ 2 root root 4.0K Jan 3 20:01 snapshots

Now we can bring up the cluster.

We start the RKE2 service on Master1 with the command below.

systemctl start rke2-server

Before starting the other master servers, we delete the dbs on them. We won't delete them, let's move them, just in case.

mv /var/lib/rancher/rke2/server/db /var/lib/rancher/rke2/server/db_old

Now we can start the RKE2 service on these servers too.

systemctl start rke2-server

After all the servers start working, let's check whether the pods have come back.

kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-585449566-nxt7t 1/1 Running 1 (5h4m ago) 21h nginx-deployment-585449566-s5ln4 1/1 Running 3 (5h5m ago) 23h nginx-deployment-before-backup-6b7f675859-2djqq 1/1 Running 0 61m nginx-deployment-before-backup-6b7f675859-5hl6l 1/1 Running 0 61m nginx-deployment-before-backup-6b7f675859-jxq5n 1/1 Running 0 61m

We can see that all the pods have come back.

What did we learn?