A production-inspired Kubernetes platform built on Proxmox with Infrastructure as Code, configuration management, GitOps, observability, and automated validation.
This project is a hands-on Platform Engineering laboratory for designing, automating, operating, and validating a highly available Kubernetes platform on Proxmox.
Infrastructure is provisioned with Terraform, system and Kubernetes configuration is automated with Ansible, while Cilium, Helm, Argo CD, Prometheus, Grafana, and NFS provide networking, application delivery, observability, and storage.
Control Planes
Worker Nodes
Load Balancers
HA Kubernetes Cluster
The platform consists of a Proxmox virtualization layer, highly available load balancing, three Kubernetes control planes, two worker nodes, Cilium networking, and persistent storage through NFS.
The Kubernetes environment starts at the infrastructure layer. Proxmox virtual machines are provisioned declaratively using Terraform and a reusable VM module.
Terraform manages the lifecycle of the Proxmox virtual machines used by the Kubernetes platform.
| Role | Nodes |
|---|---|
| Control Plane | 3 |
| Worker | 2 |
| Load Balancer | 2 |
| Total | 7 VMs |
Ansible is used to transform the provisioned virtual machines into a functional Kubernetes platform.
The automation covers operating-system preparation, Kubernetes prerequisites, control-plane initialization, control-plane joining, load balancing, Helm, and Cilium.
The cluster uses three control-plane nodes behind a highly available load-balancing layer.
Three control-plane nodes provide redundancy for the Kubernetes control-plane layer.
HAProxy provides the API load-balancing layer while Keepalived provides high availability for the endpoint.
Kubernetes was bootstrapped using kubeadm and the control-plane nodes were joined into the HA cluster.
Kubernetes version: v1.34.10
Container runtime: containerd 2.2.1
Operating system: Ubuntu 24.04.4 LTS
Cilium provides the networking layer for the Kubernetes cluster and was validated through actual workload and service connectivity tests.
Persistent storage is provided through NFS and integrated into Kubernetes using the NFS CSI driver.
Argo CD provides the GitOps layer for declarative application delivery into the Kubernetes environment.
Applications were validated through Argo CD synchronization and health status.
The monitoring stack provides visibility into Kubernetes workloads, nodes, and control-plane components.
Readiness check:
curl -s http://localhost:9091/-/ready
Expected result: Prometheus Server is Ready.
Prometheus reported Kubernetes control-plane metrics targets as unavailable.
The monitoring path was traced from Prometheus targets through ServiceMonitors and Kubernetes endpoints to the underlying control-plane component configuration.
The affected metrics endpoint was bound to
127.0.0.1, preventing Prometheus from
reaching it through the expected network path.
The listener configuration was changed to accept connections through the node network interface. The affected component was restarted and the endpoint was tested directly.
sscurlkubectlThe platform was tested under controlled failure scenarios to validate behavior beyond the normal healthy state.
A worker node was made unavailable and the resulting cluster behavior and workload recovery were observed.
Control-plane nodes were tested individually to validate the behavior of the highly available Kubernetes control plane.
Ansible automation was repeatedly executed against an already-configured Kubernetes environment.
The goal was to ensure that automation behaves as repeatable configuration management rather than a one-time installation script.
Proxmox virtual machines are provisioned declaratively using Terraform and reusable infrastructure modules.
Ansible automates operating-system preparation, Kubernetes configuration, Helm, networking, and platform components.
Three control-plane nodes are exposed through a highly available HAProxy and Keepalived layer.
Prometheus, Alertmanager, Grafana, and Kubernetes metrics provide visibility into cluster health.
The platform was also used to investigate real operational issues rather than only validating successful deployments.
One monitoring issue involved Kubernetes control-plane metrics being unreachable by Prometheus. The investigation traced the problem from Prometheus targets through ServiceMonitors and endpoints to the component listener configuration.
The root cause was a metrics endpoint bound to
127.0.0.1. The configuration was corrected,
connectivity was validated, and Prometheus targets were
rechecked.
Ansible automation was repeatedly executed against the existing environment to verify that the declared state converges without unnecessary changes.
Check mode and repeated execution were used as part of the validation process. Unexpected changes were investigated rather than automatically considered acceptable.