K3s Operations Architecture

Status#

Accepted -- Deployed on a single Proxmox VM since Q4 2025. Revised February 2026 to document Authentik forward-auth integration and Cert-Manager wildcard certificate configuration.

Context#

The homelab runs a growing number of containerized services. The initial deployment model was plain Docker Compose on a Proxmox VM. This worked well at small scale but introduced operational friction as the service count grew:

• No declarative desired-state management. Docker Compose defines what to run, but does not reconcile drift. If a container dies and the restart policy does not cover the failure mode, manual intervention is required.
• No native ingress abstraction. Traefik in Docker mode uses container labels for routing configuration. Adding or modifying routes means editing labels on individual compose files, restarting containers, and hoping the label discovery works correctly.
• No rolling updates. Docker Compose recreates containers on docker compose up -d. There is no built-in mechanism for zero-downtime deployments, health-check-gated rollouts, or automatic rollback on failure.
• No resource limits enforcement. Docker resource limits exist but are not enforced by an orchestrator. A runaway container can starve its neighbors without centralized resource management.
• No NetworkPolicy equivalent. Docker networks provide basic isolation, but there is no declarative network policy model to restrict pod-to-pod communication by label or namespace.

The question was not whether to adopt an orchestrator, but which one fits the constraints of a single-node homelab environment.

Requirements#

1. Lightweight footprint -- Must run comfortably on a single Proxmox VM (4 vCPU, 8GB RAM allocated to the orchestrator VM)
2. Declarative desired-state -- Define the intended state of all services; the orchestrator reconciles reality to match
3. Ingress management -- Native ingress controller with TLS termination, host-based routing, and middleware support
4. Rolling updates -- Zero-downtime deployments with health-check-gated rollouts and automatic rollback
5. Network policies -- Ability to restrict inter-service communication at the orchestrator level
6. Helm support -- Package management for third-party deployments (monitoring stacks, cert-manager, etc.)
7. Familiar API -- Kubernetes-compatible API surface to maintain transferable skills
8. Identity integration -- Must integrate with Authentik for forward-auth on ingress routes
9. Certificate automation -- Automated TLS certificate issuance and renewal for all ingress routes
10. Single binary operation -- Minimal operational overhead for a single operator

Constraints#

• Single Proxmox VM (no multi-node cluster initially)
• The operator (me) is the sole administrator
• Services must remain accessible during upgrades where possible
• Must coexist with the existing Cloudflare Tunnel + Authentik architecture
• Resource budget: 4 vCPU, 8GB RAM for the orchestrator node

Decision#

Deploy K3s as the container orchestration layer on a dedicated Proxmox VM. Use the built-in Traefik ingress controller for routing, Cert-Manager for automated TLS, and integrate with the existing Authentik instance for identity-aware access control.

Architecture Overview#

                    Internet
                       |
                       v
              Cloudflare Edge
              (WAF, DDoS, TLS)
                       |
                       v (Tunnel)
              cloudflared (sidecar)
                       |
                       v
    +-----------------------------------------+
    |           K3s Node (Proxmox VM)          |
    |                                          |
    |  +-----------------------------------+   |
    |  |        Traefik Ingress            |   |
    |  |  (IngressRoute, Middleware,       |   |
    |  |   ForwardAuth -> Authentik)       |   |
    |  +----------------+------------------+   |
    |                   |                      |
    |     +-------------+-------------+        |
    |     |             |             |        |
    |     v             v             v        |
    |  Namespace:    Namespace:    Namespace:   |
    |  monitoring   apps          system        |
    |  - Grafana    - App A       - Cert-Mgr   |
    |  - Prometheus - App B       - Authentik   |
    |  - Loki       - App C         Outpost    |
    |                                          |
    +-----------------------------------------+
              Proxmox VE Host

Component Responsibilities#

Component	Responsibility	Namespace
K3s	Cluster runtime, API server, scheduler, kubelet, containerd	System
Traefik	Ingress routing, TLS termination, forward-auth middleware, rate limiting	kube-system
Cert-Manager	Automated certificate issuance via Let's Encrypt, wildcard certs via Cloudflare DNS-01	cert-manager
Authentik Outpost	Forward-auth endpoint for Traefik, session validation	authentik
Cloudflared	Tunnel agent connecting Cloudflare edge to Traefik ingress	cloudflare-system
CoreDNS	Cluster-internal DNS resolution	kube-system
Metrics Server	Resource metrics for HPA and kubectl top	kube-system

Deployment Topology#

Proxmox VM Configuration#

Proxmox VE Host (bare metal)
└── VM: k3s-node-01 (Ubuntu 24.04 LTS)
    ├── vCPU: 4
    ├── RAM: 8GB
    ├── Disk: 100GB (SSD-backed)
    ├── Network: Bridge to VLAN 10 (services)
    └── K3s (single-node, server + agent)

K3s Installation#

K3s is installed with specific flags to align with the existing infrastructure:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=" \
  --disable=servicelb \
  --disable=local-storage \
  --write-kubeconfig-mode=644 \
  --tls-san=k3s.internal.valkyrienexus.com \
  --kube-apiserver-arg=audit-log-path=/var/log/k3s-audit.log \
  --kube-apiserver-arg=audit-log-maxage=30 \
  --kube-apiserver-arg=audit-log-maxbackup=10 \
  --kube-apiserver-arg=audit-log-maxsize=100" sh -

Flag rationale:

• --disable=servicelb -- ServiceLB (formerly Klipper) is disabled because all external traffic arrives through the Cloudflare Tunnel, not through a LoadBalancer service. Traefik handles ingress directly.
• --disable=local-storage -- The default local-path provisioner is replaced with a more predictable storage configuration.
• --write-kubeconfig-mode=644 -- Allows non-root access to the kubeconfig for operational convenience on a single-operator system.
• --tls-san -- Adds the internal hostname to the API server certificate for remote kubectl access via WireGuard.
• --kube-apiserver-arg=audit-log-* -- Enables API server audit logging for security observability.

Namespace Strategy#

kube-system      # K3s system components (Traefik, CoreDNS, metrics-server)
cert-manager     # Certificate automation
authentik        # Authentik outpost (embedded proxy for forward-auth)
cloudflare-system # cloudflared tunnel agent
monitoring       # Grafana, Prometheus, Loki, Alertmanager
apps             # Application workloads

Each namespace has a default ResourceQuota and LimitRange to prevent any single namespace from consuming all cluster resources.

Ingress Architecture#

Traefik Configuration#

K3s ships with Traefik as the default ingress controller. Custom configuration is applied via a HelmChartConfig resource:

# /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    logs:
      access:
        enabled: true
        format: json
    ports:
      web:
        redirectTo:
          port: websecure
      websecure:
        tls:
          enabled: true
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true

Authentik Forward-Auth Integration#

Traefik routes protected services through Authentik using a forward-auth middleware defined as a Kubernetes resource:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: authentik-forward-auth
  namespace: kube-system
spec:
  forwardAuth:
    address: http://authentik-outpost.authentik.svc.cluster.local:9000/outpost.goauthentik.io/auth/traefik
    trustForwardHeader: true
    authResponseHeaders:
      - X-authentik-username
      - X-authentik-groups
      - X-authentik-email
      - X-authentik-name
      - X-authentik-uid

Services reference this middleware in their IngressRoute:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: grafana
  namespace: monitoring
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`grafana.valkyrienexus.com`)
      kind: Rule
      services:
        - name: grafana
          port: 3000
      middlewares:
        - name: authentik-forward-auth
          namespace: kube-system
  tls:
    secretName: wildcard-valkyrienexus-com

Certificate Management#

Cert-Manager handles TLS certificates via Cloudflare DNS-01 challenge for wildcard certificates:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@valkyrienexus.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - dns01:
          cloudflare:
            email: admin@valkyrienexus.com
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-valkyrienexus-com
  namespace: kube-system
spec:
  secretName: wildcard-valkyrienexus-com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - "*.valkyrienexus.com"
    - "valkyrienexus.com"

This approach issues a single wildcard certificate shared across all ingress routes, avoiding per-service certificate overhead.

Security Posture#

What is hardened#

• API server audit logging enabled with 30-day retention
• RBAC enforced (K3s default) -- no anonymous API access
• NetworkPolicies restrict inter-namespace communication (default-deny in apps namespace, explicit allow for required paths)
• ResourceQuotas and LimitRanges per namespace prevent resource exhaustion
• Pod Security Standards enforced at baseline level via namespace labels
• Secrets encryption at rest via K3s --secrets-encryption flag
• Traefik dashboard not exposed externally
• cloudflared credentials stored as Kubernetes secrets, not environment variables
• Cert-Manager Cloudflare API token scoped to DNS zone edit only

What is monitored#

• Traefik access logs (JSON format) ingested by Loki
• Authentik authentication events forwarded to Wazuh via syslog
• Prometheus scrapes cluster metrics (node, pod, container resource usage)
• Alertmanager fires on: pod crash loops, node resource pressure, certificate expiry within 14 days, Authentik outpost health failures
• K3s API server audit logs forwarded to Wazuh

What is not yet hardened#

• Falco or runtime security is not deployed (planned: runtime anomaly detection for container behavior)
• Image signing and verification is not enforced (planned: Cosign + policy controller)
• Backup automation for etcd/SQLite state is manual (planned: CronJob-based automated backups to encrypted S3-compatible storage)

Network Policies#

The apps namespace uses a default-deny ingress policy with explicit allowlists:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: apps
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-traefik-ingress
  namespace: apps
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              app.kubernetes.io/name: traefik

This ensures that pods in the apps namespace only accept traffic from Traefik in kube-system. Direct pod-to-pod communication across namespaces is blocked unless explicitly allowed.

Alternatives Considered#

Full Kubernetes (kubeadm / RKE2)#

A standard Kubernetes deployment via kubeadm or RKE2 was considered.

Rejected because:

• kubeadm requires manual management of etcd, control plane components, and upgrade orchestration. The operational overhead is significant for a single operator.
• RKE2 is a viable alternative (and shares lineage with K3s), but its focus on CIS hardening by default adds complexity that is not required for a homelab environment.
• Both require more resources than K3s. The K3s binary consolidates the API server, controller manager, scheduler, and kubelet into a single process using SQLite instead of etcd (single-node mode), reducing memory footprint by roughly 50%.
• K3s provides a fully conformant Kubernetes API. Skills transfer directly to full Kubernetes environments. The operational simplicity is the differentiator, not the API surface.

Docker Swarm#

Docker Swarm was evaluated as a lightweight orchestration option.

Rejected because:

• Docker Swarm's development has effectively stalled. The last significant feature additions were in 2019.
• No Ingress API equivalent. Swarm's routing mesh is basic compared to Kubernetes Ingress/IngressRoute resources.
• No equivalent to NetworkPolicies. Service isolation is limited to Docker network segmentation.
• No Helm equivalent. Deploying complex third-party stacks (monitoring, cert-manager) requires manual Docker Compose translation.
• Skills invested in Swarm do not transfer to the broader industry. Kubernetes knowledge is universally applicable.
• The only genuine advantage -- simpler setup -- is negated by K3s, which is nearly as simple to install and operate.

HashiCorp Nomad#

Nomad was considered for its multi-workload scheduling (containers, VMs, binaries) and operational simplicity.

Rejected because:

• Nomad is a capable orchestrator, but its ecosystem is smaller. The Kubernetes ecosystem (Helm charts, operators, CRDs) is vastly larger, meaning less custom work for third-party deployments.
• Consul is typically required for service discovery and network segmentation, adding another component to operate.
• The Nomad job specification is different from Kubernetes manifests. Skills do not transfer directly to the Kubernetes-dominated industry.
• Nomad's strength is heterogeneous workload scheduling. My workloads are exclusively containers, which is Kubernetes' core competency.
• If I needed to schedule non-container workloads (raw binaries, VMs), Nomad would be a stronger contender.

Staying on Docker Compose#

Continuing with plain Docker Compose was the "do nothing" option.

Rejected because the operational friction described in the Context section was already impacting my ability to manage the growing service count. The lack of desired-state reconciliation, rolling updates, and network policies meant I was spending more time on container lifecycle management than on the services themselves.

Consequences#

Positive#

• Declarative desired-state -- All services are defined as Kubernetes manifests. kubectl apply reconciles the cluster to match the definition. Drift is automatically corrected.
• Rolling updates with rollback -- Deployments roll out new versions with configurable health-check gates. Failed rollouts automatically revert to the previous revision.
• NetworkPolicies -- Inter-service communication is explicitly controlled. The default-deny model in the apps namespace ensures services only receive traffic from authorized sources.
• Helm ecosystem -- Complex stacks (Prometheus + Grafana + Loki, Cert-Manager, external-dns) deploy with helm install and are upgradeable with helm upgrade.
• Transferable skills -- Everything I learn operating K3s applies directly to any Kubernetes environment. The API surface is identical.
• Resource efficiency -- K3s uses approximately 512MB of RAM for the server process on a single-node cluster. This is roughly half of what a kubeadm control plane consumes.

Negative#

• Learning curve -- Kubernetes has a well-documented learning curve. The initial setup took longer than Docker Compose. The investment pays off at scale, but the first week was slower.
• YAML volume -- Kubernetes manifests are verbose compared to Docker Compose files. A service that was 30 lines in Compose becomes 80-100 lines across Deployment, Service, IngressRoute, and middleware definitions. Helm charts and Kustomize mitigate this, but add their own complexity.
• Single-node limitations -- On a single node, there is no pod rescheduling on node failure. The node itself is the single point of failure. This is accepted because the Proxmox host provides VM-level availability (snapshots, automatic restart on host reboot).
• Debugging complexity -- When something goes wrong, the debugging path involves checking pod logs, events, ingress routes, middleware, and network policies. More layers mean more places to look.

Neutral#

• Resource overhead -- K3s adds approximately 512MB of RAM overhead for the cluster runtime. On an 8GB VM, this is significant but manageable. The monitoring stack (Prometheus, Grafana, Loki) consumes more than K3s itself.
• Upgrade cadence -- K3s follows the Kubernetes release cycle (minor version every ~4 months). This requires regular maintenance windows, documented in the K3s Cluster Maintenance runbook.
• Migration effort -- Moving existing Docker Compose services to Kubernetes manifests is a one-time cost. Each service takes 30-60 minutes to convert, including testing.

Future Direction#

• Multi-node expansion -- If workload density exceeds single-node capacity, add a second Proxmox VM as a K3s agent. The architecture supports this with no changes to the ingress or identity layers.
• GitOps with Flux or ArgoCD -- Move from kubectl apply to a GitOps reconciliation model where the Git repository is the source of truth and the cluster self-heals to match.
• Runtime security with Falco -- Deploy Falco for runtime anomaly detection (unexpected process execution, file access, network connections).
• Image verification -- Enforce container image signing with Cosign and a Kyverno or OPA Gatekeeper policy.

Revision History#

Date	Change
2026-01-15	Initial architecture decision
2026-01-28	Added NetworkPolicy examples and namespace strategy
2026-02-10	Documented Authentik forward-auth integration, Cert-Manager wildcard configuration, updated security posture