K3S Migration

k3s Install

  • install k3s
  curl -sfL https://get.k3s.io -o k3s
  chmod +x k3s
  sudo mv k3s /usr/local/bin/
  • create config file in /etc/rancher/k3s/config.yaml
# /etc/rancher/k3s/config.yaml
token: "your-secret-cluster-token"
cluster-init: true # Only on the first server node
disable:
  - servicelb
flannel-interface: eth0
kubelet-arg:
  - "feature-gates=GracefulNodeShutdown=true"
write-kubeconfig-mode: "0644"
  • for the token generated with openssl rand -hex 64
  • create systemd service unit at /etc/systemd/system/k3s.service
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/k3s
EnvironmentFile=-/etc/rancher/k3s/config.yaml
ExecStart=/usr/local/bin/k3s server
NotifyAccess=all
Restart=always
RestartSec=5s
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
  • enable and start k3s
sudo systemctl daemon-reload
sudo systemctl enable k3s
sudo systemctl start k3s
  • copy kubeconfig to home
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $USER:$USER ~/.kube/config
chmod 600 ~/.kube/config
  • firewalld on VPS
  # Add the main CNI bridge (pod network)
  sudo firewall-cmd --permanent --zone=trusted --add-interface=cni0
  # Add the Flannel overlay interface
  sudo firewall-cmd --permanent --zone=trusted --add-interface=flannel.1
  # Add the Kubernetes service bridge (optional, but good for completeness)
  sudo firewall-cmd --permanent --zone=trusted --add-interface=kube-bridge
  # Reload firewalld to apply changes
  sudo firewall-cmd --reload
  • firewalld on VPS (External Access, did not do this yet)
  # Kubernetes API Server (for kubectl access)
  sudo firewall-cmd --permanent --zone=public --add-port=6443/tcp
  # NodePort Services (if you use them)
  sudo firewall-cmd --permanent --zone=public --add-port=30000-32767/tcp
  # HTTP/HTTPS (if running an Ingress Controller like Traefik or Nginx)
  sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
  sudo firewall-cmd --permanent --zone=public --add-port=443/tcp
  sudo firewall-cmd --reload
  • should script this

Flux Install

  • install flux curl -s https://fluxcd.io/install.sh | sudo bash
  • create ssh key if you dont have one already
  • bootstrap flux
flux bootstrap git \
  --url=ssh://git@forgejo.dwagner.dev:2222/homelab/k3s-homelab.git \
  --branch=main \
  --path=clusters/vps \
  --private-key-file=/home/docker/.ssh/id_ed25519 \
  --components=source-controller,kustomize-controller,helm-controller,notification-controller \
  --password="" \
  --namespace=flux-system

  • clone the repo to the server

  • make change in repo and push

  • flux should sync with repo and deploy

  • installed sops & age for secrets

  • setup traefik

  • SOPS: Encrypts secrets in configs to Git

Certs

Parts and Pieces

  • Cert-Manager: runs on the cluster and watches for cert requests
  • Porkbun Webhook: teaches cert-manager to talk to Porkbun's API to automatically create/delete DNS TXT records
  • Traefik: Reverse Proxy
  • SOPS: Encrypts secrets in configs to Git

Security

  • crowdsec bouncer
  • firewalld
  • capta

Challenges

  • vibe building vs documentation and building
  • running traefik side by side with caddy in docker while building

Keys SOPS

  • export KEY_NAME="flux-sops-key-$(date +%Y%m%d)"
  • export KEY_COMMENT="SOPS key for public repo"
gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF
  • gpg --list-secret-keys "${KEY_NAME}"
  • export KEY_FP="FINGERPRINT_HERE"
gpg --export-secret-keys --armor "${KEY_FP}" | \
kubectl create secret generic sops-gpg \
--namespace=flux-system \
--from-file=sops.asc=/dev/stdin
  • reconcile flux reconcile kustomization vps -n flux-system --with-source

Certificates

  • Installed cert-manager

  • Installed cert-manager-webhook-porkbun

  • Setup secret with porkbun api key and secret

    • Ensure the commited secret file is encrypted

  • Build cluster-issuer.yaml as part of cert-manager

  • Build certificate.yaml as part of deployed application

  • Troubleshooting commands

    • Get All cert-manager logs kubectl logs -n cert-manager -l app.kubernetes.io/instance=cert-manager --prefix --all-containers
    • Get All challenges kubectl get challenge -A

  • docker compose exec

    • kubectl exec -it <pod_name> -- /bin/sh
    • kubectl exec -it <pod_name> -c <container_name> -- /bin/sh

  • flux get helmrelease -n

Files in an App

  • helm-release.yaml
    • this is a flux file, used to deploy helm apps
    • combination of helm values and chart
  • kustomization.yaml
    • tells kubernetes what to deploy
    • this is a flux specific file
  • namespace.yaml
    • creates a namespace
  • certificate.yaml
    • creates a new certificate based on the contents, uses cert-manager
  • cluster-issuer.yaml
    • used to issue new certificates, cert-manager
  • ingress.yaml
    • defines a route into the pods

Troubleshooting Commands

Kubernetes and Kubectl

  • General Cluster and Node Health
    • kubectl get nodes First step to check if nodes are ready or experiencing issues
    • kubectl describe node <node-name> Checks node hardware resource utilization, allocated limits/requests, and system events for errors
  • Pod Diagnostics and Logs
    • kubectl get pods -A --field-selector status.phase!=Running Filter out healthy pods cluster wide to spot unhealthy ones at a glance
    • kubectl describe <pod-name> -n <namespace> Inspect pods lifecycle events, failed volume mounts, or config errors. Events section near the bottom.
    • kubectl logs <pod-name> -n <namespace> --tail=100 -f Steam application logs in real-time to track runtime or startup errors, add --previous if the container just crashed to see why.
  • Networking and Connectivity
    • kubectl get svc,ep -n <namespace> Verify that a service actually has matching endpoints (pods). If endpoints are empty, traffic routing will fail.
    • kubectl run tmp-shell --rm -i --tty --image=curlimages/curl -- sh Spin up an ephemeral debugging pod inside the cluster to test DNS resolution or internal service-to-service connectivity via curl or nslookup

Flux

  • Checking overall GitOps status

    • flux get all -A Get a high-level overview of all Flux custom resources and their sync status across the entire cluster
    • flux check Validate that the cluster has all required Flux components installed, CRDs are up to date, and cluster-side controllers are healthy

  • Tracing Git and Kustomization Sync Issues

    • flux get kustomizations Check weather your application manifests have successfully applied and if they match the latest git version
    • flux reconcile kustomization <name> --with-source Force Flux to immediatly pull the latest commit from your Git repository and apply the manifests, bypassing the standard reconciliation interval.
    • kubectl describe kustomization <name> -n flux-system Deep dive into syncronization errors. If a manifest is invalid or blocked by cluster policy, the exact reason will appear in the status and events.

  • HelmRelease and Controller Diagnostics

    • flux get helmreleases -A Check the installation status of Helm-based deployments managed by Flux
    • flux reconcile helmrelease <name> -n <namespace> Force a retry if a Helm chart upgrade failed due to temporary network issues or chart repository timeouts
    • kubectl logs -n flux-system deployment/getSource-controller --tail=50 Troubleshoot authentication, ssh keys, or webhook connectivity issues between Flux and your Git provider

  • get values for helm chart

    • helm show values <repo_name> --repo <repo_link>

File Structure

  • Repo
    • Apps
      • Traefik
        • crowdsec-secret
        • helm-release
        • middleware
        • namespace
        • kustomization
      • netbird
        • helm-release
        • namespace
        • kustomization
      • etc...
    • Infrastructure
      • cert-manager
        • hem-release
        • namespace
        • porkbun-secret
        • kustomization
      • cert-issuer
        • cluster-issuer
        • kustomization
      • etc...
      • repos
        • cert-manager-repo
        • crowdsec-repo
        • etc...
    • Clusters
      • homelab
        • flux-system
        • infrastrucutre.yaml
        • apps.yaml
        • kustomization.yaml
      • vps
        • flux-system
        • infrastrucutre.yaml
        • apps.yaml
        • kustomization.yaml

Status

  • moved to wireguard
    • working ish
  • fixed reconciliation