Polyaxon v3 is coming →

How to deploy Postgres on Kubernetes

Deploy a single PostgreSQL instance with a Secret, persistent storage, and private access, then verify that its data survives a Pod replacement.

March 31, 2024by Polyaxon
How to deploy Postgres on Kubernetes

A small PostgreSQL database is useful for a development application, an internal ML tool, or learning how Kubernetes manages persistent state. This walkthrough deploys one database instance, connects to it, writes a record, and checks that the record survives a replacement of the database Pod.

The example has one writer and planned downtime during restarts. PostgreSQL replication, automatic failover, and backups require additional configuration. For the broader placement decision, read Should an ML platform run databases on Kubernetes?.

Why deploy Postgres on Kubernetes?

PostgreSQL and Kubernetes can fit well together when a team already uses Kubernetes to operate its applications. The same deployment, configuration, monitoring, and access-management tools can support the database, while PostgreSQL handles transactions and data consistency.

Performance and application deployment

Modern cloud-native applications are often built using microservices, which are small, self-contained services that can be deployed and scaled individually. PostgreSQL can provide persistent application data while Kubernetes manages the database container and the services that connect to it.

Running these components on a shared platform makes their configuration and resource requirements easier to manage together. Database performance still depends on query design, indexes, memory, storage latency, and the network path between clients and the database.

Easier disaster recovery

User error or infrastructure failure can interrupt an application or damage its data. PostgreSQL's Write-Ahead Log (WAL) records changes before the corresponding data pages are written, allowing the database to recover logged changes after a crash. Persistent storage lets a replacement Pod reopen that database.

Recovery from a lost disk or an accidental deletion requires a separate backup strategy. PostgreSQL backups and continuous WAL archiving can support recovery to a chosen point in time when configured together. Kubernetes provides a way to recreate the workload and attach storage; the database's backup and restore procedures determine which data can be recovered.

Better utilization of resources

Kubernetes lets you declare CPU and memory requests, set limits, and choose where a database workload runs. These controls help teams plan capacity alongside application services and other workloads instead of managing each server independently.

For a database, adjust capacity using measured query latency, connection counts, memory use, and storage I/O. Scaling PostgreSQL requires a database-aware design: additional Kubernetes replicas alone do not share transactions or configure read replicas.

Check the prerequisites

You need:

  • A development Kubernetes cluster with Linux nodes and permission to create a namespace, Secrets, Services, StatefulSets, and PersistentVolumeClaims.
  • kubectl configured for that cluster and openssl on your machine. You can use a local PostgreSQL psql client or the container's client shown below.
  • A default StorageClass that dynamically provisions filesystem volumes with ReadWriteOnce access and supports the volume permissions configured below. Storage provisioned for the example may incur charges.

Check the selected cluster and available storage before creating anything:

kubectl config current-context
kubectl get storageclass

The manifest uses the default StorageClass. If your cluster has no default, add your chosen storageClassName under volumeClaimTemplates[].spec before applying it. Some classes wait for a consuming Pod before binding a claim; the StatefulSet created below provides that consumer.

Local development provisioners may keep data on one node. A PVC can preserve files across Pod replacement, but the storage backend determines whether data remains available after node or zone failure. See persistent volumes for ML workloads for access modes, topology, and reclaim policies.

Create the namespace and password Secret

Use a fresh namespace for this exercise:

kubectl create namespace postgres-demo

Generate a password in a local scratch directory outside version control, then create a Secret from the file. The generated password stays out of the manifest and command-line arguments:

umask 077
openssl rand -base64 32 > postgres-password.txt
kubectl --namespace postgres-demo create secret generic postgres-auth \
  --from-file=password=postgres-password.txt

Keep the password available for the psql login later in the tutorial. A Kubernetes Secret is the appropriate object for credentials; a ConfigMap is for non-confidential configuration. Secrets still require restricted access and encryption at rest in the cluster. Kubernetes Secrets documentation.

Deploy PostgreSQL on Kubernetes

The steps below create configuration, Services, and a database workload in separate files so you can inspect each resource as it is added. The example uses postgres:17.11-bookworm. Check the PostgreSQL version policy when maintaining the image, and use an approved image digest when immutable image identity is required.

The resources have distinct responsibilities:

ResourcePurpose
ConfigMap postgres-configNames the database and configures initialization and the data directory.
Secret postgres-auth, created aboveSupplies the initial administrator password through a mounted file.
Headless Service postgres-headlessProvides the StatefulSet's network identity.
ClusterIP Service postgresProvides an internal client endpoint on port 5432.
StatefulSet postgresMaintains the single database Pod, postgres-0.
Claim template dataRequests a volume through the PVC data-postgres-0.

Create a ConfigMap

ConfigMaps separate non-confidential configuration from the container image. Here, the ConfigMap sets the database name, the administrator role used for initialization, host authentication, and the directory where PostgreSQL writes its data. The password remains in the Secret created earlier.

Save this as postgres-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  namespace: postgres-demo
data:
  POSTGRES_DB: appdb
  POSTGRES_USER: postgres
  POSTGRES_INITDB_ARGS: "--auth-host=scram-sha-256"
  PGDATA: /var/lib/postgresql/data/pgdata

Apply the configuration:

kubectl apply -f postgres-config.yaml

For a quick CLI-driven setup, the following command is an alternative to applying that file. Choose one creation method:

kubectl --namespace postgres-demo create configmap postgres-config \
  --from-literal=POSTGRES_DB=appdb \
  --from-literal=POSTGRES_USER=postgres \
  --from-literal=POSTGRES_INITDB_ARGS='--auth-host=scram-sha-256' \
  --from-literal=PGDATA=/var/lib/postgresql/data/pgdata

Keep postgres-config.yaml if you want to manage the configuration declaratively. The image initializes the database and password only when the data directory is empty. Updating the Secret later does not change an existing database password: rotate the database role's password and its client credentials together.

Understand the persistent volume and claim

A PersistentVolume (PV) represents storage available to the cluster. A PersistentVolumeClaim (PVC) requests that storage by capacity, access mode, and StorageClass. The Pod mounts the claim, and the database files remain on the underlying volume when that Pod is replaced.

With dynamic provisioning, the StorageClass's provisioner creates the volume in response to the claim. You do not need to create a hostPath PV manually. The StatefulSet below includes a volumeClaimTemplates entry that requests 5Gi and creates the claim data-postgres-0 for its first Pod. The data volume mount then attaches that claim to the database container.

The claim uses ReadWriteOnce, which restricts read-write access by node, not by Pod. This access mode does not make node-local storage portable or provide database replication. Choose the StorageClass for its durability, performance, and failure-domain requirements as well as its capacity. The volume documentation explains the access modes and claim lifecycle.

Create the PostgreSQL Services

Services give clients a stable way to reach selected Pods even when Pod addresses change. A ClusterIP Service exposes an internal endpoint; NodePort also exposes a port on cluster nodes. For this database, use internal access and a local port-forward when administering it from your workstation.

The headless Service supplies the StatefulSet's network identity, while the ClusterIP Service supplies the client endpoint. Save both resources below in postgres-services.yaml, including the --- separator:

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
  namespace: postgres-demo
spec:
  clusterIP: None
  selector:
    app: postgres-demo
  ports:
    - name: postgres
      port: 5432
      targetPort: postgres
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: postgres-demo
spec:
  type: ClusterIP
  selector:
    app: postgres-demo
  ports:
    - name: postgres
      port: 5432
      targetPort: postgres

Apply the Services:

kubectl apply -f postgres-services.yaml

They will select the database Pod once it is created and ready.

Create the PostgreSQL workload

Kubernetes workload controllers manage the desired number of Pods and their replacement during changes. A Deployment is commonly used for interchangeable application replicas. This example uses a StatefulSet so the database Pod has a stable name and a predictable relationship with its storage claim.

Save the following as postgres-statefulset.yaml. The container reads configuration from the ConfigMap, the password from a mounted Secret, and database files from the persistent volume:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: postgres-demo
spec:
  serviceName: postgres-headless
  replicas: 1
  selector:
    matchLabels:
      app: postgres-demo
  template:
    metadata:
      labels:
        app: postgres-demo
    spec:
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        runAsGroup: 999
        fsGroup: 999
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: postgres
          image: postgres:17.11-bookworm
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          ports:
            - name: postgres
              containerPort: 5432
          envFrom:
            - configMapRef:
                name: postgres-config
          env:
            - name: POSTGRES_PASSWORD_FILE
              value: /etc/postgres-auth/password
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 1Gi
          startupProbe:
            exec:
              command: ["pg_isready", "-h", "127.0.0.1", "-U", "postgres", "-d", "appdb"]
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 30
          readinessProbe:
            exec:
              command: ["pg_isready", "-h", "127.0.0.1", "-U", "postgres", "-d", "appdb"]
            periodSeconds: 5
            timeoutSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
            - name: postgres-auth
              mountPath: /etc/postgres-auth
              readOnly: true
      volumes:
        - name: postgres-auth
          secret:
            secretName: postgres-auth
            defaultMode: 0440
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi

The PostgreSQL 17 data volume is mounted at /var/lib/postgresql/data; PGDATA uses a subdirectory so database files are separate from filesystem-created entries at the volume root. The official image documentation describes its initialization settings and the different volume layout used by PostgreSQL 18 and later.

UID and GID 999 match the postgres account in this Debian image. fsGroup makes compatible mounted volumes accessible to that group. If your platform assigns different user IDs or storage permissions, adapt the security context with your platform administrator.

Keep replicas: 1. More replicas would create independent databases and claims; a StatefulSet does not configure PostgreSQL replication. Avoid force-deleting a database Pod when its old process might still be running. The StatefulSet documentation explains its identity and lifecycle guarantees.

Start PostgreSQL and inspect readiness

With the Secret, ConfigMap, and Services in place, apply the workload and wait for the StatefulSet:

kubectl apply -f postgres-statefulset.yaml
kubectl --namespace postgres-demo rollout status statefulset/postgres --timeout=600s
kubectl --namespace postgres-demo get pods,pvc,services

Look for postgres-0 with a ready container and data-postgres-0 bound to a volume. The startup probe allows roughly five minutes for the server to begin accepting TCP connections. It avoids treating the image's temporary, socket-only initialization server as ready.

pg_isready reports whether the server accepts connections. It does not validate the password or prove that an application query succeeds; the next step checks both. The resource values are a starting point for this small exercise and should be adjusted for a real workload.

If startup stalls, inspect the relevant objects:

kubectl --namespace postgres-demo describe pod postgres-0
kubectl --namespace postgres-demo describe pvc data-postgres-0
kubectl --namespace postgres-demo logs postgres-0 --container postgres
SymptomWhat to inspect
PVC remains PendingDefault StorageClass, provisioner health, capacity, and storage topology; delayed binding may be normal until a Pod can schedule.
Pod remains PendingResource availability, admission requirements, and volume attachment events.
Permission denied during initializationStorage support for fsGroup, the mounted directory's ownership, and the image's runtime user.
Password authentication failsThe original database password; a changed Secret does not reinitialize an existing PVC.

Connect privately and write a record

Clients in the namespace can use postgres:5432; clients in other namespaces can use postgres.postgres-demo:5432 when network policy permits. A ClusterIP Service provides internal discovery and routing. It does not restrict which other Pods may connect or enable database TLS.

For this local exercise, start a port-forward bound to loopback and leave it running:

kubectl --namespace postgres-demo port-forward \
  --address=127.0.0.1 service/postgres 15432:5432

In another terminal, connect with psql and enter the generated password from postgres-password.txt when prompted:

psql -h 127.0.0.1 -p 15432 -U postgres -d appdb -W

Inside psql, create one record:

CREATE TABLE IF NOT EXISTS tutorial_messages (
    id integer PRIMARY KEY,
    message text NOT NULL
);

INSERT INTO tutorial_messages (id, message)
VALUES (1, 'stored-on-the-pvc')
ON CONFLICT (id) DO UPDATE SET message = EXCLUDED.message;

SELECT id, message FROM tutorial_messages WHERE id = 1;

The query should return the row with message stored-on-the-pvc. Exit with \q, then stop the port-forward with Ctrl+C. This exercise uses the database administrator account; applications should have their own roles with only the permissions they require.

Connect from inside the Pod with kubectl exec

You can also use the psql client included in the PostgreSQL image. This is useful when you do not have psql installed locally or need to investigate from the database container:

kubectl --namespace postgres-demo exec -it postgres-0 -- \
  psql -h 127.0.0.1 -U postgres -d appdb -W

-i keeps standard input open, and -t allocates a terminal for the interactive session. The -- separates kubectl's arguments from the command executed inside the container. Enter the same database password at the prompt, run the SQL above, and exit with \q.

The explicit host uses a TCP connection and the host authentication configured in this example. Omitting the host can use local Unix-socket authentication instead, which would not check the same password-authenticated path.

Verify persistence after Pod replacement

Restart the StatefulSet and wait for its replacement Pod:

kubectl --namespace postgres-demo rollout restart statefulset/postgres
kubectl --namespace postgres-demo rollout status statefulset/postgres --timeout=600s
kubectl --namespace postgres-demo get pod postgres-0
kubectl --namespace postgres-demo get pvc data-postgres-0

The new Pod keeps the name postgres-0 and mounts the existing claim. Reconnect using either method above: restart the port-forward and local psql client, or run kubectl exec again. Then run:

SELECT id, message FROM tutorial_messages WHERE id = 1;

The same row should still be present. This demonstrates persistence through a controlled Pod replacement. Recovery from storage loss or a node failure depends on the storage backend and your backup strategy.

Best practices for deploying PostgreSQL on Kubernetes

The example provides a starting point for understanding how these Kubernetes objects work together. Before putting valuable data in the database, apply the same care to its runtime, access controls, and recovery procedures.

Run the container as an unprivileged user

The database process should run as a nonroot user with access to its data directory. The manifest sets runAsNonRoot, uses the image's PostgreSQL account, drops Linux capabilities, and disables privilege escalation. It also avoids mounting a Kubernetes service-account token that the database does not need.

Check the image's user and the volume's ownership together. Setting a nonroot user without giving it access to the mounted data directory can prevent initialization or restart. Database roles are separate from Linux users: give applications dedicated PostgreSQL roles instead of sharing the administrator login used for this exercise.

Encrypt your data

Configure PostgreSQL TLS for application connections, encryption for the storage backend, and protection for backups and their encryption keys. An internal Service and a namespace do not enable TLS automatically. The local port-forward is an administration path; applications need their own connection and certificate configuration.

Encryption protects data confidentiality. Preserve availability separately with backups, recovery procedures, and appropriate storage durability.

Use a separate namespace and control access

The postgres-demo namespace keeps this example's objects together and makes cleanup explicit. For a real database, a dedicated namespace can also scope RBAC permissions, resource quotas, and operational ownership.

Restrict who can read Secrets, modify the database workload, or execute commands inside its Pod. Use NetworkPolicies to allow the required application and administration paths when your networking implementation supports them. A namespace alone does not block traffic from other workloads.

Plan backups and availability

Keep backups outside the live volume and exercise restores. PostgreSQL offers logical dumps, filesystem-level backups, and continuous archiving. Choose an approach for the amount of data you can afford to lose and how quickly service must recover.

For replication and failover, choose a database-aware operator or managed service with recovery procedures that meet your availability requirements. Include application reconnection and data verification in the recovery process.

Maintain the image and size resources

Follow supported minor releases and plan major-version migrations explicitly. Changing the image tag across major versions does not migrate an existing data directory.

Monitor disk space, connections, query latency, memory, and I/O. Size requests and limits from observed behavior, and account for the container's shared-memory requirements. Keep enough capacity for maintenance operations as well as normal application traffic.

For an ML platform, distinguish PostgreSQL records from datasets, checkpoints, and model artifacts. Polyaxon's own metadata database has separate operational requirements; its PostgreSQL setup guide describes external database configuration. When a Polyaxon workload accesses an application database, supply its credentials through configured connections or mounted Secrets.

Clean up the development database

Exit psql and stop any port-forward. Scale the database down and wait for the Pod to terminate:

kubectl --namespace postgres-demo scale statefulset/postgres --replicas=0
kubectl --namespace postgres-demo wait --for=delete pod/postgres-0 --timeout=120s

If the wait times out, inspect the Pod and resolve the shutdown before continuing. Once the Pod has terminated, remove the workload, Services, and configuration:

kubectl delete -f postgres-statefulset.yaml -f postgres-services.yaml
kubectl --namespace postgres-demo delete configmap postgres-config

With the default StatefulSet retention behavior used here, the PVC remains. Check it before deciding whether to discard the data:

kubectl --namespace postgres-demo get pvc data-postgres-0

Only when the tutorial data is disposable, delete the claim and namespace. A PV with reclaim policy Delete can delete the underlying storage when its claim is removed:

kubectl --namespace postgres-demo delete pvc data-postgres-0
kubectl delete namespace postgres-demo
rm postgres-password.txt

If the reclaim policy is Retain, the volume requires separate administrator cleanup. Keep the connection, query, and persistence exercise as a small repeatable check when you change the image, storage class, or database configuration.

Final thoughts

The ConfigMap, Secret, persistent volume, StatefulSet, and Services each solve a different part of running PostgreSQL on Kubernetes. Following a record from insertion through Pod replacement makes their relationship concrete. Use that exercise as a foundation, then choose storage, access, and recovery arrangements that fit the application and the team operating it.