Should an ML platform run databases on Kubernetes?

Kubernetes can run stateful applications, but the ability to start a database container is not the same as the ability to operate a reliable database service. The real decision is who owns storage, replication, upgrades, backup, recovery, security, and performance—and whether Kubernetes improves that operating model.

For an ML platform, distinguish the metadata database from datasets, model artifacts, feature data, caches, and training checkpoints. They have different consistency, capacity, and recovery requirements.

Compare the actual operating models

The common choices are:

• a managed database service outside the cluster;
• a database operated on virtual or physical machines;
• a database managed on Kubernetes by an experienced team or operator;
• an embedded or temporary database for development and testing.

A managed service often removes control-plane maintenance, automated backups, patching, and parts of failover. It does not remove schema design, connection management, query performance, data governance, or recovery testing.

Kubernetes can standardize deployment and policy when a team already operates stateful services well. It can also combine the database's failure domain with the cluster, storage driver, network, and admission stack.

Know what StatefulSet provides

A StatefulSet maintains stable Pod identities, ordered lifecycle behavior, and a predictable relationship between each Pod and its PersistentVolumeClaim. That is useful infrastructure for a database.

It does not understand transaction logs, replication health, leader election, quorum, consistent backup, schema migration, or application-level failover. The StatefulSet documentation describes the identity and rollout guarantees; database-aware automation must provide the rest.

Choose an operator only if its supported database versions, upgrade paths, backup integration, failure handling, and ownership model match production requirements. An operator transfers repetitive knowledge into software; it does not eliminate the need for database expertise.

Evaluate storage beyond capacity

A PVC request such as 500Gi says little about latency, throughput, durability, replication, topology, snapshot consistency, or recovery time. Measure the storage behavior the database needs under steady load, compaction, checkpointing, backup, and failure.

Review:

• access mode and volume topology;
• filesystem and block-device support;
• IOPS, throughput, and latency guarantees;
• attachment and node-volume limits;
• expansion, snapshot, clone, and restore behavior;
• reclaim policy and deletion protection;
• encryption keys and access controls;
• zone or node failure recovery.

Local disks can provide performance but couple availability to a node. Network volumes improve mobility but add their own latency and attachment dependencies.

Design for database-level availability

Kubernetes restarts containers and replaces Pods. A restarted database process still needs to join the correct cluster, recover its log, verify storage, and avoid split brain. Readiness should represent whether the instance can safely serve its role, not merely whether a TCP port is open.

Spread replicas across failure domains while respecting storage topology and database quorum. Use Pod disruption budgets to reduce voluntary simultaneous disruption, but remember that they do not prevent node failure, node-pressure eviction, direct deletion, or every upgrade mistake.

Test loss of a Pod, node, zone, network path, storage attachment, and control-plane access separately. “Three replicas” is not evidence that recovery works.

Make backup independent of the cluster

A backup stored only on the same cluster or storage system does not protect against cluster-wide mistakes or provider failures. Define recovery point and recovery time objectives, then test restores into an isolated environment.

Database-native backups may need coordination with writes and transaction logs. Volume snapshots are useful building blocks but are not automatically application-consistent. Record the database version, schema state, encryption keys, and restoration procedure with every protected backup set.

Recovery testing should include the application reconnecting and validating data, not just a restored Pod entering Running.

Plan upgrades and maintenance

A Kubernetes rollout can replace Pods in order, but it cannot decide whether two database versions can form a cluster or whether a schema change is reversible. Separate application migrations, database engine upgrades, operator upgrades, Kubernetes upgrades, and storage-driver upgrades.

Define compatibility gates, maintenance windows, rollback limitations, and backup checkpoints. Verify that autoscaling, rescheduling, and disruption automation cannot conflict with a controlled database procedure.

Protect the data path

Use distinct workload identities, least-privilege network access, encrypted connections, encryption at rest, audited administrative access, and managed credential rotation. Avoid placing database superuser credentials in broad namespace Secrets or application environment dumps.

Database telemetry can expose query text or customer identifiers. Apply the same classification and retention controls to logs, traces, and slow-query samples as to the data itself.

Decide with an ownership checklist

Running the database on Kubernetes is more defensible when:

• the team has demonstrated database and Kubernetes operational experience;
• the database has supported automation for failover, backup, and upgrades;
• storage and topology meet measured requirements;
• recovery is tested regularly outside the primary failure domain;
• a clear team owns incidents at every layer;
• the benefits of a consistent platform exceed the additional dependencies.

Prefer a managed or separately operated database when those conditions are not met, particularly for critical metadata with modest scale. Operational simplicity is a reliability feature.

Keep Polyaxon data roles explicit

Polyaxon uses different systems for different data roles. Keep service metadata in a transactional database, artifacts and large outputs in appropriate object or shared storage, and disposable working data in explicit scratch or cache locations. Configure access through Polyaxon connections rather than embedding infrastructure credentials in components.

Database placement should be invisible to most experiment authors but fully visible to platform owners. Choose the model that produces the clearest ownership, safest recovery, and most predictable ML platform—not the one that places the most components inside the cluster.