Running Redis on Kubernetes for ML platforms

Redis can support caches, queues, rate limits, coordination, and short-lived state in an ML platform. Running it on Kubernetes can standardize deployment and identity, but it does not make Redis durable, highly available, or correctly scaled by itself.

Start with the data contract and failure requirements. The deployment topology, persistence mode, storage, client behavior, and recovery procedure should follow from what the application can safely lose or replay.

Decide whether Redis belongs in the cluster

An in-cluster Redis deployment gives the platform team control over placement, configuration, networking, and upgrade timing. It also gives that team responsibility for memory behavior, replication, failover, storage, backup, security, and incident response.

A managed Redis service may provide stronger operational separation and an established recovery model. It adds network dependency, provider limits, and possibly data-residency or cost constraints.

Compare the complete operating model. Co-location is not automatically lower latency if Pods, volumes, or clients cross zones.

Choose the Redis topology deliberately

The word “cluster” is ambiguous. Separate the Kubernetes cluster from the Redis deployment mode:

• a standalone Redis instance is simple but has one process-level failure domain;
• primary-replica replication adds copies but needs a failover mechanism;
• Sentinel coordinates discovery and failover for replicated Redis;
• Redis Cluster shards keys across primary nodes and provides limited availability during some failures.

The official Redis Cluster documentation explains its sharding, client, port, and partition behavior.

Choose based on dataset size, write semantics, client compatibility, recovery point, recovery time, and operational skill. Three Redis Pods do not become a Redis Cluster merely because Kubernetes created them.

Use StatefulSets for stable identity

Kubernetes StatefulSets provide stable Pod identities, ordered lifecycle, and persistent-volume association. A headless Service can expose the stable DNS records Redis members need.

Those primitives do not configure Redis membership or safely promote replicas. Use a maintained operator, chart, or owned automation that understands the selected Redis mode and version.

Pin images by digest and review configuration generated by charts or operators. Define how membership changes, upgrades, and failed reconciliation are observed and reversed.

Design storage and recovery together

First decide whether Redis data is disposable, reconstructable, or authoritative. Then select snapshotting, append-only persistence, replication, backups, and storage accordingly.

Persistent volumes keep data across Pod replacement, but they are not backups. A corrupted dataset, accidental command, or failed application write can persist perfectly.

Test:

• Pod replacement with the same volume;
• node and zone loss;
• replica promotion;
• restore into an isolated deployment;
• version upgrade and rollback constraints;
• recovery while clients continue retrying.

Use storage classes whose topology matches the failure design. Node-local volumes can bind recovery to a failed machine; network volumes can introduce latency and throughput limits.

Treat memory as a hard capacity boundary

Redis keeps active data in memory, and persistence or replication can require additional memory during forks, rewrites, synchronization, and failover.

Set Kubernetes requests from measured steady-state use and limits with enough headroom for the selected persistence behavior. Configure Redis eviction or no-eviction policy according to the application contract.

An OOM-killed primary and an application-level eviction are different failures. Monitor process memory, allocator fragmentation, Kubernetes working set, evicted keys, rejected writes, fork duration, and persistence status.

Keep system and Redis memory reservations aligned. A Pod limit that is lower than Redis's own maximum-memory plan makes Kubernetes the unexpected eviction policy.

Plan networking and client behavior

Redis Cluster requires clients to understand redirection and reach the advertised addresses of its members. Validate the selected client library, DNS behavior, Services, network policies, and both client and cluster-bus traffic.

Use topology-aware placement to avoid putting every primary and replica in one node or zone. At the same time, account for cross-zone latency and transfer cost.

Clients need bounded timeouts, connection pools, backoff, and retry rules. Retrying a non-idempotent queue or coordination operation can duplicate work even if the Redis connection eventually recovers.

Secure access and administration

Restrict network access to approved workloads and administrative paths. Use Redis authentication and transport encryption where supported by the deployment model, store credentials in controlled secrets, and rotate them through an owned process.

Run containers with a narrow security context and dedicated service account. Do not expose Redis publicly or assume a ClusterIP Service is a complete security boundary.

Separate application credentials from administrative capability. Audit configuration changes, failovers, flush operations, and backup access.

Observe service and workload outcomes

Track command latency, operations per second, connected and blocked clients, hit ratio where meaningful, memory, evictions, rejected connections, replication offset, link state, persistence health, cluster state, and Kubernetes restarts or volume events.

Relate infrastructure signals to user outcomes: queue delay, cache miss cost, agent-state recovery, model-service latency, or failed operation submission. A healthy Redis process can still be serving stale or semantically invalid data.

Run these read-only checks with an explicit target:

kubectl get statefulsets,pods,services \
  --context acme-staging \
  --namespace platform-cache \
  --output wide

kubectl get events \
  --context acme-staging \
  --namespace platform-cache \
  --sort-by=.metadata.creationTimestamp

Connect Redis dependencies to Polyaxon

If an ML workload depends on Redis, expose that dependency through an approved Polyaxon connection rather than embedding endpoints or credentials in code and images.

Use Polyaxon run metadata, logs, and metrics to identify which operation experienced a Redis failure and whether it retried safely. Keep durable checkpoints and artifacts in storage designed for that lifecycle; a cache or coordination store should not become the only copy of valuable ML output.

Redis on Kubernetes is successful when the deployment has an explicit data contract, tested recovery, bounded memory, secure access, compatible clients, and an owner for day-two operations.