Skip to main content

Advanced Topics

Operators, Custom Resources, Cluster Management, and advanced Kubernetes patterns for complex deployments.

Custom Resources and CRDs​

Custom Resource Definition (CRD)​

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                engine:
                  type: string
                  enum: ['mysql', 'postgres', 'mongodb']
                version:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 10
                storage:
                  type: object
                  properties:
                    size:
                      type: string
                    storageClass:
                      type: string
              required: ['engine', 'version']
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum: ['Pending', 'Running', 'Failed']
                message:
                  type: string
                endpoints:
                  type: array
                  items:
                    type: string
      additionalPrinterColumns:
        - name: Engine
          type: string
          jsonPath: .spec.engine
        - name: Version
          type: string
          jsonPath: .spec.version
        - name: Status
          type: string
          jsonPath: .status.phase
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
  scope: Namespaced
  names:
    plural: databases
    singular: database
    kind: Database
    shortNames:
      - db

Custom Resource Instance​

apiVersion: example.com/v1
kind: Database
metadata:
  name: my-postgres-db
  namespace: production
spec:
  engine: postgres
  version: '13.4'
  replicas: 3
  storage:
    size: '100Gi'
    storageClass: 'fast-ssd'
  configuration:
    max_connections: 200
    shared_buffers: '256MB'
status:
  phase: Running
  message: 'Database is ready'
  endpoints:
    - 'postgres-primary.production.svc.cluster.local:5432'
    - 'postgres-read-1.production.svc.cluster.local:5432'
    - 'postgres-read-2.production.svc.cluster.local:5432'

CRD Operations​

# Create CRD
kubectl apply -f database-crd.yaml

# List CRDs
kubectl get crds
kubectl get customresourcedefinitions

# Describe CRD
kubectl describe crd databases.example.com

# Create custom resource
kubectl apply -f database-instance.yaml

# Get custom resources
kubectl get databases
kubectl get db
kubectl get databases.example.com

# Describe custom resource
kubectl describe database my-postgres-db

# Delete custom resource
kubectl delete database my-postgres-db

# Delete CRD (removes all instances)
kubectl delete crd databases.example.com

Operators​

Basic Operator Structure​

apiVersion: apps/v1
kind: Deployment
metadata:
  name: database-operator
  namespace: operator-system
spec:
  replicas: 1
  selector:
    matchLabels:
      name: database-operator
  template:
    metadata:
      labels:
        name: database-operator
    spec:
      serviceAccountName: database-operator
      containers:
        - name: manager
          image: database-operator:latest
          command:
            - /manager
          args:
            - --leader-elect
          env:
            - name: WATCH_NAMESPACE
              value: ''
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: 'database-operator'
          resources:
            limits:
              cpu: 200m
              memory: 256Mi
            requests:
              cpu: 100m
              memory: 128Mi

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: database-operator
  namespace: operator-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: database-operator
rules:
  - apiGroups: ['']
    resources: ['pods', 'services', 'configmaps', 'secrets']
    verbs: ['*']
  - apiGroups: ['apps']
    resources: ['deployments', 'statefulsets']
    verbs: ['*']
  - apiGroups: ['example.com']
    resources: ['databases']
    verbs: ['*']

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: database-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: database-operator
subjects:
  - kind: ServiceAccount
    name: database-operator
    namespace: operator-system

Operator Development with Kubebuilder​

# Initialize new operator project
kubebuilder init --domain example.com --repo github.com/example/database-operator

# Create API and controller
kubebuilder create api --group apps --version v1 --kind Database

# Generate manifests
make manifests

# Build and push image
make docker-build docker-push IMG=myregistry/database-operator:latest

# Deploy operator
make deploy IMG=myregistry/database-operator:latest

# Undeploy operator
make undeploy

Operator Lifecycle Manager (OLM)​

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  name: database-operator.v1.0.0
  namespace: operators
spec:
  displayName: Database Operator
  description: Manages database instances
  version: 1.0.0
  keywords:
    - database
    - postgres
    - mysql
  maintainers:
    - name: Example Team
      email: team@example.com
  provider:
    name: Example Inc
  maturity: stable
  install:
    strategy: deployment
    spec:
      deployments:
        - name: database-operator
          spec:
            replicas: 1
            selector:
              matchLabels:
                name: database-operator
            template:
              metadata:
                labels:
                  name: database-operator
              spec:
                serviceAccountName: database-operator
                containers:
                  - name: manager
                    image: database-operator:latest
  customresourcedefinitions:
    owned:
      - name: databases.example.com
        version: v1
        kind: Database
        displayName: Database
        description: Represents a database instance

Helm Charts​

Basic Chart Structure​

mychart/
├── Chart.yaml
├── values.yaml
├── templates/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   └── _helpers.tpl
└── charts/

Chart.yaml​

apiVersion: v2
name: myapp
description: A Helm chart for my application
type: application
version: 0.1.0
appVersion: '1.0.0'
keywords:
  - web
  - application
home: https://example.com
sources:
  - https://github.com/example/myapp
maintainers:
  - name: Example Team
    email: team@example.com
dependencies:
  - name: postgresql
    version: 11.6.12
    repository: https://charts.bitnami.com/bitnami
    condition: postgresql.enabled

values.yaml​

replicaCount: 3

image:
  repository: myapp
  pullPolicy: IfNotPresent
  tag: 'latest'

service:
  type: ClusterIP
  port: 80
  targetPort: 8080

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: myapp.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: myapp-tls
      hosts:
        - myapp.example.com

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

postgresql:
  enabled: true
  auth:
    database: myapp
    username: myapp
  primary:
    persistence:
      enabled: true
      size: 8Gi

Template Example​

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "myapp.fullname" . }}
  labels:
    {{- include "myapp.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "myapp.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "myapp.selectorLabels" . | nindent 8 }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        ports:
        - name: http
          containerPort: {{ .Values.service.targetPort }}
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: http
        readinessProbe:
          httpGet:
            path: /ready
            port: http
        resources:
          {{- toYaml .Values.resources | nindent 12 }}
        env:
        - name: DATABASE_URL
          value: "postgresql://{{ .Values.postgresql.auth.username }}:$(POSTGRES_PASSWORD)@{{ include "myapp.fullname" . }}-postgresql:5432/{{ .Values.postgresql.auth.database }}"
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: {{ include "myapp.fullname" . }}-postgresql
              key: password

Helm Commands​

# Create new chart
helm create mychart

# Lint chart
helm lint mychart

# Dry run install
helm install myapp mychart --dry-run --debug

# Install chart
helm install myapp mychart
helm install myapp mychart --values custom-values.yaml

# Upgrade release
helm upgrade myapp mychart
helm upgrade myapp mychart --set image.tag=v2.0.0

# List releases
helm list
helm list --all-namespaces

# Get release info
helm get values myapp
helm get manifest myapp
helm get notes myapp

# Rollback release
helm rollback myapp 1

# Uninstall release
helm uninstall myapp

# Package chart
helm package mychart

# Add repository
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# Search charts
helm search repo nginx
helm search hub postgres

Cluster Management​

Cluster Autoscaler​

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
        - image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.21.0
          name: cluster-autoscaler
          resources:
            limits:
              cpu: 100m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 300Mi
          command:
            - ./cluster-autoscaler
            - --v=4
            - --stderrthreshold=info
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --expander=least-waste
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/kubernetes
          env:
            - name: AWS_REGION
              value: us-west-2

Node Affinity and Anti-Affinity​

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values:
                      - linux
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
                matchExpressions:
                  - key: node-type
                    operator: In
                    values:
                      - compute-optimized
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - frontend
              topologyKey: kubernetes.io/hostname
      containers:
        - name: frontend
          image: nginx

Taints and Tolerations​

# Add taint to node
kubectl taint nodes node1 key1=value1:NoSchedule
kubectl taint nodes node1 key1=value1:NoExecute
kubectl taint nodes node1 key1=value1:PreferNoSchedule

# Remove taint
kubectl taint nodes node1 key1=value1:NoSchedule-

# Get node taints
kubectl describe node node1 | grep Taints
# Pod with tolerations
apiVersion: v1
kind: Pod
metadata:
  name: toleration-example
spec:
  tolerations:
    - key: 'key1'
      operator: 'Equal'
      value: 'value1'
      effect: 'NoSchedule'
    - key: 'key2'
      operator: 'Exists'
      effect: 'NoExecute'
      tolerationSeconds: 3600
  containers:
    - name: app
      image: nginx

Advanced Networking​

Service Mesh with Istio​

# Istio Gateway
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - bookinfo.example.com

---
# Virtual Service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
    - bookinfo.example.com
  gateways:
    - bookinfo-gateway
  http:
    - match:
        - uri:
            exact: /productpage
      route:
        - destination:
            host: productpage
            port:
              number: 9080
    - match:
        - uri:
            prefix: /api/v1
      route:
        - destination:
            host: reviews
            port:
              number: 9080
            subset: v2
          weight: 80
        - destination:
            host: reviews
            port:
              number: 9080
            subset: v3
          weight: 20

---
# Destination Rule
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2
    - name: v3
      labels:
        version: v3

Ingress Controllers​

# NGINX Ingress Controller
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    nginx.ingress.kubernetes.io/rate-limit: '100'
    nginx.ingress.kubernetes.io/rate-limit-window: '1m'
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.example.com
      secretName: api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /api/v1(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 80

---
# Traefik Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: default-auth@kubernetescrd
spec:
  ingressClassName: traefik
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-service
                port:
                  number: 80

GitOps and CI/CD​

ArgoCD Application​

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/myapp-config
    targetRevision: HEAD
    path: manifests
    helm:
      valueFiles:
        - values-production.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Flux CD Kustomization​

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 10m
  path: './clusters/production'
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp-repo
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: myapp
      namespace: production
  postBuild:
    substitute:
      env: 'production'
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars

Multi-Cluster Management​

Cluster API​

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: my-cluster
  namespace: default
spec:
  clusterNetwork:
    services:
      cidrBlocks: ['10.96.0.0/12']
    pods:
      cidrBlocks: ['192.168.0.0/16']
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AWSCluster
    name: my-cluster
  controlPlaneRef:
    kind: KubeadmControlPlane
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    name: my-cluster-control-plane

---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSCluster
metadata:
  name: my-cluster
  namespace: default
spec:
  region: us-west-2
  sshKeyName: my-key

Rancher Multi-Cluster​

apiVersion: management.cattle.io/v3
kind: Cluster
metadata:
  name: production-cluster
spec:
  displayName: 'Production Cluster'
  description: 'Production Kubernetes cluster'
  rancherKubernetesEngineConfig:
    kubernetesVersion: 'v1.21.3-rancher1-1'
    nodes:
      - address: 10.0.1.10
        user: ubuntu
        role: [controlplane, etcd]
      - address: 10.0.1.11
        user: ubuntu
        role: [worker]
      - address: 10.0.1.12
        user: ubuntu
        role: [worker]
    services:
      etcd:
        backup:
          enabled: true
          intervalHours: 12
          retention: 6
      kubelet:
        extraArgs:
          max-pods: '110'

Quick Reference​

Custom Resources​

  • kubectl get crds - List Custom Resource Definitions
  • kubectl describe crd name - CRD details
  • kubectl get customresource - List custom resource instances
  • kubectl api-resources - List all available resources

Helm Commands​

  • helm create chart-name - Create new chart
  • helm install release chart - Install chart
  • helm upgrade release chart - Upgrade release
  • helm list - List releases
  • helm rollback release revision - Rollback release

Advanced Scheduling​

  • Node Affinity - Schedule pods to specific nodes
  • Pod Affinity/Anti-Affinity - Co-locate or separate pods
  • Taints and Tolerations - Repel pods from nodes
  • Priority Classes - Pod scheduling priority

Cluster Management Tools​

  • Cluster Autoscaler - Automatic node scaling
  • Vertical Pod Autoscaler - Automatic resource adjustment
  • Prometheus Operator - Monitoring stack management
  • Cert-Manager - Automatic certificate management
  • External-DNS - Automatic DNS record management