👨‍💻 Kubernetes Advanced: Production-Ready Deployments

Kubernetes Advanced: Production-Ready Deployments

Welcome to the final part of our Kubernetes tutorial series! This advanced guide dives deep into production-grade features and enterprise-level best practices. We’ll explore how to build robust, scalable, and maintainable Kubernetes deployments that can handle real-world workloads.

What We’ll Cover

  1. StatefulSets and Persistent Storage - Managing stateful applications and data persistence
  2. Service Mesh with Istio - Advanced networking and service management
  3. Advanced Monitoring and Logging - Complete observability stack
  4. High Availability Patterns - Ensuring 24/7 uptime
  5. Production Tools and Utilities - Essential tools for Kubernetes management

Prerequisites

  • Completed our Intermediate Kubernetes tutorial
  • Familiarity with Kubernetes resources
  • Understanding of microservices architecture
  • Basic knowledge of storage concepts

StatefulSets and Persistent Storage

Understanding Storage in Kubernetes

Storage management in Kubernetes involves several key concepts:

  1. Volumes: Temporary or persistent storage attached to pods
  2. PersistentVolumes (PV): Cluster-wide storage resources
  3. PersistentVolumeClaims (PVC): Storage requests by applications
  4. StorageClasses: Dynamic provisioning of storage

Storage Classes

StorageClasses define different types of storage with varying performance characteristics:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs  # Cloud provider specific
parameters:
  type: gp3
  iopsPerGB: "10"
  encrypted: "true"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

Persistent Volumes

Define available storage resources:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-storage
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: fast-ssd
  hostPath:  # Example for local testing
    path: /data/storage

Persistent Volume Claims

Request storage for applications:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-storage
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: fast-ssd

StatefulSets with Persistent Storage

StatefulSets are perfect for applications that need:

  • Stable, unique network identifiers
  • Stable, persistent storage
  • Ordered deployment and scaling
  • Ordered automated rolling updates

Here’s a comprehensive example of a StatefulSet with persistent storage:

apiVersion: v1
kind: Service
metadata:
  name: postgresql-headless
spec:
  clusterIP: None
  selector:
    app: postgresql
  ports:
    - port: 5432

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql
spec:
  serviceName: postgresql-headless
  replicas: 3
  selector:
    matchLabels:
      app: postgresql
  template:
    metadata:
      labels:
        app: postgresql
    spec:
      containers:
      - name: postgresql
        image: postgres:14
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secrets
              key: password
        ports:
        - containerPort: 5432
          name: postgresql
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        - name: config
          mountPath: /etc/postgresql/conf.d
        readinessProbe:
          exec:
            command:
            - pg_isready
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast-ssd
      resources:
        requests:
          storage: 10Gi

Best Practices for Storage Management

  1. Capacity Planning:
    • Monitor storage usage trends
    • Set up alerts for capacity thresholds
    • Use volume expansion features when available
  2. Backup and Recovery: 
    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: daily-backup
spec:
  includedNamespaces:
  - "*"
  storageLocation: default
  volumeSnapshotLocations:
  - default
  schedule: "0 1 * * *"
  retention:
 keepDaily: 7
 keepWeekly: 4
  3. Storage Performance:
    • Use the right storage class for your workload
    • Monitor I/O metrics
    • Consider using local volumes for high-performance needs
  4. Security: ```yaml apiVersion: v1 kind: PersistentVolume metadata: name: secure-storage spec: capacity: storage: 5Gi accessModes:
    • ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: encrypted-storage csi: driver: ebs.csi.aws.com volumeHandle: vol-xyz volumeAttributes: encrypted: “true” ```

Service Mesh with Istio

Installing Istio

istioctl install --set profile=demo -y

Enabling Istio Injection

kubectl label namespace default istio-injection=enabled

Virtual Service Configuration

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews-route
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1

Advanced Monitoring

Prometheus Setup

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false

Grafana Dashboard Configuration

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  name: golang-dashboard
spec:
  json: >
    {
      "dashboard": {
        "id": null,
        "title": "Golang Dashboard",
        ...
      }
    }

High Availability Patterns

Pod Disruption Budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: critical-app

Anti-Affinity Rules

apiVersion: apps/v1
kind: Deployment
metadata:
  name: high-availability-app
spec:
  replicas: 3
  template:
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - high-availability-app
            topologyKey: "kubernetes.io/hostname"

Essential Kubernetes Tools and Utilities

The Kubernetes ecosystem is rich with tools that can help you manage, monitor, and optimize your clusters. Here’s a curated list of essential tools for different aspects of Kubernetes management, inspired by the comprehensive collection at KubeTools:

1. Development Tools

IDE Plugins

  • Kubernetes for VS Code: Syntax highlighting and cluster management
  • Lens: The Kubernetes IDE
  • K9s: Terminal-based UI for managing clusters

Local Development

# Using Telepresence for local development
telepresence connect
telepresence intercept my-service --port 8080:80

2. Cluster Management

Cluster Creation and Management

  • kind: Local clusters using Docker
  • k3s: Lightweight production-grade K8s
  • kubeadm: Official cluster bootstrapping
  • kops: Production grade K8s installation on AWS

Policy Management

# Example OPA/Gatekeeper policy
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-require-labels
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["owner", "environment"]

3. Security Tools

Scanner and Analyzers

  • Trivy: Container vulnerability scanner
  • Falco: Runtime security monitoring
  • Snyk: Dependency and container security

Security Management

# Example NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: strict-policy
spec:
  podSelector:
    matchLabels:
      app: secure-app
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 80

4. Monitoring and Debugging

Monitoring Stacks

  • Prometheus + Grafana: Metrics and visualization
  • Elastic Stack: Logging and analysis
  • Jaeger: Distributed tracing

Debugging Tools

# Using kubectl-debug
kubectl debug node/my-node -it --image=ubuntu

# Using stern for log tailing
stern my-app --tail 50

# Using ktunnel for local debugging
ktunnel expose deployment my-deployment 8080:80

5. CI/CD Tools

Pipeline Tools

  • ArgoCD: GitOps continuous delivery
  • Flux: GitOps for cluster management
  • Jenkins X: Cloud native CI/CD

Deployment Tools

  • Helm: Package manager for Kubernetes
  • Kustomize: Template-free configuration
  • Skaffold: Local development workflow

6. Storage and Backup

Storage Management

  • Rook: Cloud native storage orchestrator
  • Velero: Backup and migrate resources
  • OpenEBS: Container attached storage

Backup Solutions

# Example Velero Schedule
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: daily-backup
spec:
  schedule: "@daily"
  template:
    includedNamespaces:
    - "*"
    includedResources:
    - "*"
    storageLocation: default
    ttl: 720h0m0s

7. Network Tools

Service Mesh

  • Istio: Comprehensive service mesh
  • Linkerd: Lightweight service mesh
  • Consul: Service networking platform

Ingress Controllers

  • Nginx Ingress: Popular ingress controller
  • Traefik: Cloud native edge router
  • Contour: High-performance ingress

8. Development Workflows

Local Development

# Using Tilt for local development
tilt up

# Using Skaffold for continuous development
skaffold dev

Testing Tools

  • kube-monkey: Chaos testing
  • k6: Load testing
  • Testkube: Testing framework

Getting Started with Tools

  1. Tool Selection:
    • Start with essential tools
    • Add tools as needs grow
    • Consider team expertise
  2. Installation Methods: ```bash

    Using kubectl krew

    kubectl krew install neat kubectl krew install ctx kubectl krew install ns

Using Helm

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install prometheus prometheus-community/prometheus

Using operators

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml ```

  1. Integration Tips:
    • Use GitOps workflows
    • Automate tool installation
    • Maintain documentation

Visit KubeTools for a complete, up-to-date list of Kubernetes tools and utilities.

Production Checklist

  1. Security:
    • Network Policies
    • RBAC configuration
    • Pod Security Policies
  2. Monitoring:
    • Prometheus metrics
    • Grafana dashboards
    • Alert management
  3. Backup:
    • etcd backup
    • PV snapshots
    • Disaster recovery plan
  4. Scaling:
    • HPA configuration
    • VPA setup
    • Cluster autoscaling

What’s Next?

Consider exploring:

  • GitOps workflows with Flux/ArgoCD
  • Custom Resource Definitions (CRDs)
  • Operator pattern implementation
  • Cloud-native security practices

Additional Resources

Written on July 23, 2025