Upgrading a Kubernetes cluster without downtime | Antyxsoft Cloud

Kubernetes ships a minor release roughly every four months and supports each one for about a year. That arithmetic is unforgiving: skip upgrades for eighteen months and you are not doing a maintenance task any more, you are doing a migration.

The good news is that a routine upgrade on a well-configured cluster is uneventful. The work is in the configuration, not the upgrade.

1. Read the release notes properly

Every minor version removes APIs that were deprecated several versions earlier. The failure mode is specific and avoidable: a manifest applies fine today, and after the upgrade the resource simply does not exist.

Scan the cluster rather than trusting your memory of it:

# removed and deprecated APIs still in use
kubent

# and against the manifests in your repositories
pluto detect-files -d ./deploy

Check your add-ons at the same time — ingress controller, CSI driver, cert-manager, metrics-server, service mesh. Add-on incompatibility causes more upgrade incidents than the Kubernetes upgrade itself. Each project publishes a compatibility matrix; read all of them before you touch anything.

2. Understand version skew

Control plane first, nodes after. The kubelet may be up to three minor versions behind the API server but never ahead of it, so upgrading a node pool past the control plane is not something to attempt. One minor version at a time, in order: 1.30 to 1.32 means going through 1.31.

Your own tooling matters too. A kubectl more than one minor version away from the server produces confusing errors, and so does a CI image nobody has updated in two years.

3. PodDisruptionBudgets are the actual mechanism

An upgrade drains nodes. A drain evicts pods. Without a PodDisruptionBudget, nothing stops every replica of a service being evicted at once, and your zero-downtime upgrade takes the service down for as long as the new pods need to become ready.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

Use minAvailable as a number for small deployments and a percentage for large ones. Two rules keep this honest: a PDB on a single-replica deployment blocks the drain forever rather than protecting anything, and minAvailable equal to the replica count does the same. Both are common, and both turn an upgrade into a stuck node and a confused afternoon.

While you are there, make sure replicas are actually spread across nodes. Three replicas on one node satisfy a PDB right up to the moment that node is drained.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: api

4. Rehearse on a copy, not on staging

Staging usually differs from production in the ways that matter: fewer replicas, no PDBs, different add-on versions. On managed Kubernetes the cheap move is to provision a short-lived cluster at the current version, apply your manifests, upgrade it, and throw it away. An afternoon of cluster time buys you the list of things that will break.

5. The sequence

1. Snapshot or back up anything stateful, and verify the backup is readable.
2. Upgrade the control plane. On a managed cluster this is one action and the API server is briefly unavailable while running workloads carry on serving.
3. Confirm the control plane is healthy and add-ons are still reconciling.
4. Upgrade one node pool — the least important one — and watch a full pod cycle.
5. Upgrade the remaining pools, one at a time, with surge capacity so a new node joins before an old one is drained.
6. Re-run the API scan, check certificates renewed, check your dashboards are still receiving metrics.

Turn the cluster autoscaler off for the duration, or it will fight the drain by removing the capacity you just added.

kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data --timeout=600s
# ... node replaced or upgraded ...
kubectl uncordon node-3

6. What goes wrong

Pods that ignore SIGTERM. A drain sends SIGTERM and waits for terminationGracePeriodSeconds. An application that does not shut down gracefully drops in-flight requests on every node in the cluster, once per upgrade.

Long-running jobs. A four-hour batch job and a node drain are incompatible. Either drain with a timeout long enough to let it finish, or make the job resumable before upgrade day.

Zone-bound volumes. A pod with a PVC can only reschedule where the volume can attach. If that zone has no spare capacity, the pod pends and the drain stalls.

Single-replica admission webhooks. When the webhook pod is the thing being evicted, every subsequent pod creation fails validation and the cluster stops being able to heal itself. Run webhooks with two replicas and a sane failurePolicy.

DaemonSets you forgot about. Log shippers, CNI agents and monitoring agents are pinned to node versions more often than people expect. Upgrade them before the nodes, not after.

7. There is no rollback

Worth saying plainly: you cannot downgrade a control plane. Once the API server is on the new minor version, forward is the only direction. Restoring an etcd snapshot to an earlier version is a disaster-recovery procedure, not a rollback plan, and it loses everything since the snapshot.

What you can roll back is a node pool — create a new pool at the new version, move workloads onto it, and keep the old pool until you are satisfied. That is the closest thing to an undo button, and it is why the node-pool-at-a-time approach is worth the extra hour.

The short version

Scan for removed APIs and check every add-on's compatibility matrix. Give every multi-replica service a PDB and a spread constraint, and never put a PDB on a single replica. Rehearse on a throwaway cluster. Control plane first, then one node pool at a time with surge capacity and the autoscaler paused. Assume forward-only, and upgrade often enough that each one is boring.