Since keeping database records of your users, projects, experiments, and jobs is very important. Polyaxon offers a couple of ways to set a high available database.
The easiest way to keep your database intact in case of node failures or restarts, is by enabling postgresql persistence, enabling persistence will tell the database to use a persistent volume to store data instead of using the host node.
Here's an example of using dynamic provisioning:
postgresql: persistence: enabled: true size: 5Gi
Using this configuration, the volume will be provisioned dynamically, based on default storage class defined in your cluster.
In case you want to have more control where and how the persistence claims is created,
and should not be managed by the chart,
you need to define a PVC and assign it to the
- Create the PersistentVolume
- Create the PersistentVolumeClaim
Use the name of this claim in the chart
postgresql: persistence: enabled: true existingClaim: myClaimName
If you prefer to have the postgresql database managed and hosted outside of Kubernetes, instead of the in-cluster one provided by Polyaxon, e.g. Amazon RDS for PostgreSQL, Azure Database for PostgreSQL, GCP Cloud SQL for PostgreSQL, or DigitalOcean managed PostgreSQL. You need to disable the in-cluster database, and provide the information needed to establish a connection to the external one, e.g.:
postgresql: enabled: false externalServices: postgresql: user: polyaxon password: polyaxon database: postgres host: 35.262.163.88
Please check this integration guide for cloud specific instructions on how to setup a postgreSQL server instance.
If you decided to deploy Polyaxon in-cluster make sure to set proper node scheduling to avoid running high load runs on the same node hosting the database.
connMaxAge allows you to set the lifetime of a database connection, in seconds.
0 to close database connections at the end of each request and
None for unlimited persistent connections.
If you are running a high load Polyaxon deployment, we suggest using a connection pooler, e.g pgbouncer.