Kubernetes Production Guide

Kubernetes (K8s) is an open-source container orchestration platform originally designed by Google and now maintained by the Cloud Native Computing Foundation (CNCF). It automates the deployment, scaling, and management of containerized applications across clusters of machines.

At its core, Kubernetes follows a declarative model: you describe the desired state of your workloads (how many replicas, what image, what resources, what networking), and Kubernetes continuously reconciles the actual state to match. This is fundamentally different from imperative scripting where you tell the system what to do step-by-step.

Architecture

Key concepts

• Control Plane — The brain of the cluster. The API Server is the single entry point for all operations. The Scheduler places pods on nodes. The Controller Manager runs reconciliation loops. etcd stores all cluster state.
• Worker Nodes — Machines that run your workloads. Each node runs a kubelet (agent that talks to the API server), kube-proxy (networking rules), and a container runtime (containerd, CRI-O).
• Pods — The smallest deployable unit. A pod contains one or more containers that share networking and storage. Pods are ephemeral by design.
• Deployments — Declarative way to manage ReplicaSets and pods. You specify the desired number of replicas and the update strategy, and the Deployment controller handles the rest.
• Services — Stable network endpoints that abstract away pod IPs. Services provide load balancing across pods that match a label selector.
• Namespaces — Virtual clusters within a physical cluster. Used for multi-tenancy, environment separation (dev/staging/prod), and resource quota boundaries.

Declarative vs imperative

kubectl apply -f deployment.yaml

kubectl create deployment nginx --image=nginx
kubectl scale deployment nginx --replicas=3

Kubernetes is a set of components, not a single binary you install. Distributions package those components with opinionated defaults for networking, storage, ingress, and container runtime. The three most common lightweight/edge distributions are MicroK8s, K3s, and RKE2.

Comparison table

When to use which

kubectl is the primary CLI for interacting with Kubernetes clusters. It communicates with the API server using configuration stored in a kubeconfig file (default: ~/.kube/config).

Kubeconfig structure

A kubeconfig file has three main sections:

apiVersion: v1
kind: Config
clusters:
  - name: production
    cluster:
      server: https://10.0.1.100:6443
      certificate-authority-data: <base64-ca-cert>
users:
  - name: admin
    user:
      client-certificate-data: <base64-client-cert>
      client-key-data: <base64-client-key>
contexts:
  - name: prod-admin
    context:
      cluster: production
      user: admin
      namespace: default
current-context: prod-admin

• clusters — Define API server endpoints and CA certificates
• users — Define authentication credentials (certs, tokens, OIDC)
• contexts — Bind a cluster + user + optional namespace into a named context
• current-context — The active context that kubectl uses by default

Merging kubeconfigs

When managing multiple clusters, you can merge kubeconfigs using the KUBECONFIG environment variable:

# Merge multiple kubeconfig files
export KUBECONFIG=~/.kube/config:~/.kube/cluster2.yaml:~/.kube/cluster3.yaml

# Flatten into a single file
kubectl config view --flatten > ~/.kube/merged-config
export KUBECONFIG=~/.kube/merged-config

# Switch between contexts
kubectl config get-contexts
kubectl config use-context prod-admin
kubectl config use-context staging-dev

Common kubectl commands

TLS SAN warnings

When connecting to a cluster, you may encounter a certificate error like:

Unable to connect to the server: x509: certificate is valid for 10.0.1.100,
127.0.0.1, not 192.168.1.50

Why this happens: The Kubernetes API server generates a TLS certificate during cluster initialization. That certificate includes a list of Subject Alternative Names (SANs) — the hostnames and IP addresses the certificate is valid for. If you connect to the API server using a hostname or IP that is not in the SAN list, TLS verification fails because the client cannot verify it is talking to the correct server.

This commonly occurs when:

• Accessing a cluster from outside the network (the external IP is not in the cert)
• Using a load balancer IP or DNS name that was not included at install time
• Connecting via a VPN or bastion host with a different IP

Fixing SAN issues per distribution

# /etc/rancher/rke2/config.yaml
tls-san:
  - "k8s.example.com"
  - "192.168.1.50"
  - "10.0.0.100"

# During install
curl -sfL https://get.k3s.io | \
  sh -s - server \
  --tls-san k8s.example.com \
  --tls-san 192.168.1.50

# Or in /etc/rancher/k3s/config.yaml
tls-san:
  - "k8s.example.com"
  - "192.168.1.50"

# Edit the CSR config
sudo nano /var/snap/microk8s/current/certs/csr.conf.template

# Add your SANs under [alt_names]
# IP.3 = 192.168.1.50
# DNS.4 = k8s.example.com

# Refresh the certificates
sudo microk8s refresh-certs --cert server.crt

Workaround: skip TLS verification

# One-off command
kubectl --insecure-skip-tls-verify get nodes

# Set in kubeconfig context permanently
kubectl config set-cluster my-cluster \
  --insecure-skip-tls-verify=true

Ingress is a Kubernetes API object that manages external access to services within a cluster, typically HTTP/HTTPS. It provides URL-based routing, TLS termination, and virtual hosting. An Ingress resource is useless without an Ingress Controller — a pod that reads Ingress objects and configures the underlying proxy (Nginx, Traefik, HAProxy, etc.).

Ingress controllers

Ingress example

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - myapp.example.com
      secretName: myapp-tls
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp-svc
                port:
                  number: 80

Service type: LoadBalancer

In cloud environments, creating a Service of type LoadBalancer automatically provisions a cloud load balancer (AWS ELB, GCP LB, Azure LB). On bare-metal, there is no cloud API to call, so the Service stays in Pending state forever — unless you install MetalLB.

MetalLB for bare-metal

MetalLB provides LoadBalancer service support for bare-metal clusters. It operates in two modes:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.1.200-192.168.1.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system

apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: router
  namespace: metallb-system
spec:
  myASN: 64500
  peerASN: 64501
  peerAddress: 10.0.0.1
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: default
  namespace: metallb-system

cert-manager is the standard way to manage TLS certificates in Kubernetes. It automates the issuance, renewal, and rotation of certificates from various sources including Let's Encrypt, HashiCorp Vault, and self-signed CAs.

Issuer vs ClusterIssuer

Let's Encrypt with cert-manager

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          ingress:
            class: nginx

ACME challenge types

Using cert-manager with Ingress annotations

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
    - hosts:
        - myapp.example.com
      secretName: myapp-tls   # cert-manager creates this Secret
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp
                port:
                  number: 80

When this Ingress is created, cert-manager detects the cert-manager.io/cluster-issuer annotation, requests a certificate from Let's Encrypt, completes the ACME challenge, and stores the resulting certificate in the myapp-tls Secret. The Ingress controller then uses that Secret for TLS termination. Renewal happens automatically before expiry (default: 2/3 through the certificate's duration, which is ~30 days before expiry for standard 90-day Let's Encrypt certificates). You can customize this with spec.renewBefore or spec.renewBeforePercentage.

Self-signed CA

For internal services, air-gapped environments, or development, you can use a self-signed CA:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: internal-ca
  secretName: internal-ca-secret
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca-issuer
spec:
  ca:
    secretName: internal-ca-secret

GitOps is a paradigm where Git is the single source of truth for your infrastructure and application state. A GitOps operator watches a Git repository and automatically synchronizes the cluster state to match what is committed. Changes are made through pull requests, which provides an audit trail, code review, and easy rollback (just revert the commit).

How GitOps works

1. Developer pushes a change to a Git repository (e.g., updates an image tag in a Deployment manifest)
2. The GitOps operator detects the change (via polling or webhook)
3. The operator compares the desired state (Git) with the actual state (cluster)
4. If there is drift, the operator applies the changes to the cluster
5. Health checks verify the deployment succeeded

ArgoCD vs FluxCD

When to use which

ArgoCD Application example

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/k8s-manifests.git
    targetRevision: main
    path: apps/myapp/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp
  syncPolicy:
    automated:
      prune: true        # Delete resources removed from Git
      selfHeal: true      # Revert manual changes in cluster
    syncOptions:
      - CreateNamespace=true

Helm and Kustomize are the two primary tools for managing Kubernetes manifests at scale. They solve overlapping but different problems, and many teams use them together.

Comparison

Helm basics

# Add a chart repository
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# Install a chart
helm install my-postgres bitnami/postgresql \
  --namespace databases --create-namespace \
  --values custom-values.yaml

# Upgrade a release
helm upgrade my-postgres bitnami/postgresql \
  --values custom-values.yaml

# List releases
helm list -A

# Rollback
helm rollback my-postgres 1

Kustomize basics

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namespace: production
patches:
  - target:
      kind: Deployment
      name: myapp
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 5
images:
  - name: myapp
    newTag: v2.1.0

# Build and apply
kubectl apply -k overlays/production/

# Preview rendered output
kubectl kustomize overlays/production/

Using them together

A common pattern is to use Helm for third-party software (databases, monitoring, ingress controllers) and Kustomize for your own applications. You can also render Helm charts into plain YAML and manage them with Kustomize:

# Render a Helm chart to plain YAML
helm template my-release bitnami/postgresql \
  --values values.yaml > base/postgresql.yaml

# Then manage with Kustomize overlays for env-specific tweaks

KubeVirt is a Kubernetes add-on that allows you to run traditional virtual machines alongside containers on the same cluster. It extends Kubernetes with custom resource definitions (CRDs) for managing VM lifecycle using the same kubectl tooling.

Why it matters

• Converged infrastructure — Run VMs and containers side-by-side. No need for separate VMware/Proxmox infrastructure and a separate Kubernetes cluster.
• Migration path — Move legacy workloads that cannot be containerized (Windows apps, kernel-dependent software, legacy databases) into the Kubernetes platform without rewriting them.
• Unified tooling — Use the same CI/CD pipelines, monitoring, networking, and storage for both VMs and containers.
• Harvester — Rancher's Harvester HCI platform is built on KubeVirt, providing a complete hyperconverged infrastructure solution on top of Kubernetes.

Key CRDs

Basic VM example

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: ubuntu-vm
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: ubuntu-vm
    spec:
      domain:
        cpu:
          cores: 2
        memory:
          guest: 4Gi
        devices:
          disks:
            - name: rootdisk
              disk:
                bus: virtio
            - name: cloudinit
              disk:
                bus: virtio
          interfaces:
            - name: default
              masquerade: {}
      networks:
        - name: default
          pod: {}
      volumes:
        - name: rootdisk
          dataVolume:
            name: ubuntu-dv
        - name: cloudinit
          cloudInitNoCloud:
            userData: |
              #cloud-config
              password: changeme
              chpasswd: { expire: false }
---
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: ubuntu-dv
spec:
  source:
    http:
      url: "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img"
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi

KubeVirt vs traditional virtualization

Kubernetes storage is built around three key abstractions: StorageClasses define how storage is provisioned, PersistentVolumes (PVs) represent actual storage resources, and PersistentVolumeClaims (PVCs) are requests for storage by pods. The Container Storage Interface (CSI) is the standard plugin API that connects Kubernetes to storage backends.

Storage flow

Dynamic provisioning

With dynamic provisioning, you do not need to pre-create PVs. When a PVC is created that references a StorageClass, the CSI driver automatically provisions the underlying storage and creates the PV:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume
spec:
  storageClassName: longhorn
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Storage solutions comparison

Access modes

• ReadWriteOnce (RWO) — Mounted as read-write by a single node. Most common for databases and stateful apps.
• ReadOnlyMany (ROX) — Mounted as read-only by many nodes. Good for shared configuration or static content.
• ReadWriteMany (RWX) — Mounted as read-write by many nodes. Required for shared storage across pods. NFS, CephFS, and Longhorn (via built-in NFSv4 share-manager since v1.1) support this.

Kubernetes networking follows a flat model: every pod gets its own IP address, and all pods can communicate with each other without NAT. This is implemented by Container Network Interface (CNI) plugins. The choice of CNI affects performance, security policy support, and operational complexity.

CNI plugins

Network Policies

Network Policies are Kubernetes-native firewall rules that control pod-to-pod traffic. By default, all pods can talk to all other pods. Network Policies restrict this based on labels, namespaces, and ports.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: production
spec:
  podSelector: {}    # Applies to all pods in namespace
  policyTypes:
    - Ingress
  ingress: []        # Empty = deny all ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - port: 8080
          protocol: TCP

Service types

DNS (CoreDNS)

CoreDNS runs as a Deployment in the kube-system namespace and provides DNS-based service discovery for the cluster. Every Service gets a DNS entry:

• my-service.my-namespace.svc.cluster.local — Full qualified domain name
• my-service.my-namespace — Short form (from any namespace)
• my-service — Shortest form (from same namespace only)

Kubernetes security is a broad topic that spans authentication, authorization, workload isolation, secrets management, and supply chain security. The fundamental principle is defense in depth — no single mechanism is sufficient; you need layers.

RBAC (Role-Based Access Control)

RBAC controls who can do what in the cluster. It uses four resource types:

• Role — Namespace-scoped permissions (e.g., "can read pods in namespace X")
• ClusterRole — Cluster-scoped permissions (e.g., "can read nodes", "can create namespaces")
• RoleBinding — Binds a Role to a user/group/ServiceAccount within a namespace
• ClusterRoleBinding — Binds a ClusterRole to a user/group/ServiceAccount cluster-wide

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: production
  name: read-pods
subjects:
  - kind: User
    name: jane
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

ServiceAccounts

Every pod runs as a ServiceAccount. If not specified, it uses the default ServiceAccount in its namespace. Best practice: create dedicated ServiceAccounts for each workload with only the permissions it needs.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: production
automountServiceAccountToken: false  # Don't mount token unless needed

Pod Security Standards

Pod Security Standards (PSS) replaced the deprecated PodSecurityPolicy (PSP). They are enforced via the built-in Pod Security Admission controller using namespace labels:

# Apply restricted security to a namespace
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Secrets management

Kubernetes Secrets are base64-encoded (not encrypted) by default. For production:

• Enable encryption at rest — Configure the API server to encrypt Secrets in etcd using AES-GCM (preferred) or AES-CBC. AES-GCM is faster and provides authenticated encryption.
• External secrets management — Use the External Secrets Operator to sync secrets from HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager
• Sealed Secrets — Bitnami's Sealed Secrets controller allows you to store encrypted secrets in Git. Only the controller in the cluster can decrypt them
• SOPS + age/GPG — Encrypt secret values in YAML files using Mozilla SOPS. Works well with FluxCD's native SOPS decryption

Image scanning and supply chain

• Scan images in CI — Use Trivy, Grype, or Snyk to scan container images during the build pipeline, before they reach the cluster
• Admission control — Use a policy engine to block deployment of unscanned or vulnerable images
• Image signing — Sign images with Cosign and verify signatures at admission time

OPA / Gatekeeper

Open Policy Agent (OPA) Gatekeeper is an admission controller that enforces custom policies on Kubernetes resources. It uses Rego (a policy language) to define constraints:

• Require all images to come from an approved registry
• Block containers running as root
• Require resource limits on all pods
• Enforce label standards across all resources
• Prevent use of latest image tag

Use this checklist when assessing, deploying, or auditing a Kubernetes cluster.