Use kubectl patch safely

kubectl patch updates selected fields on an existing Kubernetes object without replacing the complete manifest. It is useful for a small controlled change or an incident response, but patch semantics—especially for lists—can produce a different result from what the short command appears to say.

Make the target, patch type, preview, owner, and durable follow-up explicit.

Choose the patch type first

The official kubectl patch reference supports three patch types:

Strategic merge is the default for supported built-in types, but it is not supported for custom resources. For a CRD, use JSON merge or JSON Patch according to the schema and desired list behavior.

Never choose a patch type by habit. Retrieve the object and understand the field being changed first.

Preview against the server

Use an explicit context and namespace, and ask the API server to validate the proposed result without persisting it:

kubectl --context production --namespace inference patch deployment model-server \
  --type=merge \
  --patch '{"spec":{"replicas":4}}' \
  --dry-run=server \
  -o yaml

Review more than the field you intended to change. Admission controllers can default or mutate the object, and server-side validation can reject fields that a client-only preview would not catch.

Then run the same command without --dry-run=server only after the output and target are approved.

Be careful with lists

Lists cause many surprising patches. A JSON merge patch that supplies one container can replace the whole containers list. A strategic merge patch may merge a built-in list by a schema-defined key such as container name. JSON Patch targets a specific path or position, which can become fragile if list order changes.

For example, add an annotation with JSON Patch using an escaped JSON Pointer path:

kubectl --context production --namespace inference patch deployment model-server \
  --type=json \
  --patch='[{"op":"add","path":"/metadata/annotations/operations.polyaxon.com~1ticket","value":"INC-1234"}]' \
  --dry-run=server \
  -o yaml

If the annotations map itself does not exist, that path needs a different operation sequence. Previewing catches the mismatch before a write.

Put non-trivial patches in a file

Shell quoting quickly becomes harder to review than the change. Store a reusable patch in version control and use --patch-file:

spec:
  template:
    metadata:
      annotations:
        operations.polyaxon.com/change: INC-1234

kubectl --context production --namespace inference patch deployment model-server \
  --type=merge \
  --patch-file=patch.yaml \
  --dry-run=server \
  -o yaml

The patch file is still not the source of truth unless your deployment system owns and reapplies it.

Do not remove safety metadata casually

Finalizers, owner references, selectors, service accounts, and admission annotations often represent another controller's contract. Removing a finalizer to force deletion can orphan infrastructure or bypass required cleanup. Changing a selector can disconnect a controller from its Pods. A temporary service-account patch can expand privileges.

Before modifying controller-managed fields, identify the owning controller and why reconciliation has stopped. Prefer repairing that control path. Reserve direct metadata surgery for a documented recovery procedure with explicit approval and cleanup.

Expect reconciliation

GitOps agents, operators, and deployment controllers may revert a live patch or treat it as drift. That is correct if another system owns the field.

For an emergency patch:

1. identify the source of truth and field owner;
2. preview and apply the smallest change;
3. verify the workload outcome;
4. make the equivalent reviewed change in declarative configuration;
5. confirm reconciliation preserves the intended state;
6. remove temporary annotations or overrides when the incident ends.

Use Manage Kubernetes contexts safely to reduce wrong-cluster risk, and use kubectl diff or the platform's deployment preview for the durable configuration change.

Patch Polyaxon-managed workloads at the owner

Polyaxon operations create Kubernetes resources as part of a managed workload lifecycle. A direct patch to a generated Pod can disappear on retry or replacement and may make the live object diverge from the recorded operation.

Put lasting resource, scheduling, security, and lifecycle changes in the operation specification or an approved scheduling preset. Use a live patch only when incident scope requires it, record the exact command and resulting object, and follow through at the owning configuration layer.

The safe mental model is simple: kubectl patch changes live state; it does not automatically change intent. A complete patch workflow brings those two back together.