Polyaxon & NVIDIA NIM

How to use Polyaxon and NVIDIA NIM together

+

NIM packages an inference runtime and supported model profile. Polyaxon schedules that container on Kubernetes, resolves the NGC credentials and cache, exposes the service, and keeps the release context available for audits and evaluation.

See the NVIDIA NIM documentation for the upstream configuration and requirements.

Prerequisites

• NVIDIA AI Enterprise or NGC entitlement for the selected NIM and model.
• An image-pull connection and NGC API key stored as scoped secrets.
• A supported GPU profile with sufficient memory, local disk or persistent cache, and compatible drivers.

Configuration

Define the NIM service

Resolve registry and runtime credentials through Polyaxon connections and request the exact GPU count required by the selected NIM profile.

version: 1.1
kind: component
name: nvidia-nim
run:
  kind: service
  ports: [8000]
  rewritePath: true
  connections: [ngc-registry, ngc-api-key, nim-cache]
  container:
    image: nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.12
    env:
    - name: NIM_CACHE_PATH
      value: /opt/nim/.cache
    resources:
      requests:
        cpu: "8"
        memory: 32Gi
      limits:
        nvidia.com/gpu: "1"

This is NVIDIA's documented model-specific NIM image. Confirm the selected tag and hardware profile in NVIDIA's support matrix before deployment.

Submit the operation

Run the component through the target Polyaxon project, queue, preset, and approval path.

polyaxon run -f nim.yaml

Check NIM readiness and model metadata

Resolve the protected service URL, wait for readiness, and inspect the model endpoint before sending production-shaped requests.

SERVICE_URL=$(polyaxon ops service --external --url)
curl "$SERVICE_URL/v1/models"

Deployment checks

• Pin the runtime image and model revision; warm the model cache before measuring startup or latency.
• Keep the service private by default and add authentication, TLS, rate limits, and network policy deliberately.
• Define readiness, liveness, timeout, graceful shutdown, and rollback behavior before production traffic.
• Measure latency distributions, throughput, quality, errors, and accelerator memory under representative load.
• Retain NIM license and entitlement evidence with the deployment process and keep the NGC key out of component text and logs.

Troubleshooting

The service never becomes ready

Inspect model download, credentials, disk space, GPU memory, runtime flags, port binding, and readiness behavior.

Requests fail through the URL

Verify the service port, rewrite-path setting, authentication header, ingress, network policy, and API path.

Latency degrades under load

Measure queueing, batch settings, context length, KV cache pressure, replica saturation, and storage or network contention.

References

• NVIDIA NIM documentation — Supported models, profiles, containers, configuration, and deployment guidance.
• Install NIM for LLMs — Published model-specific and model-free NIM image names, tags, and registry requirements.
• Polyaxon service runtime — Service ports, replicas, connections, volumes, and external access.
• Polyaxon model serving — Patterns for deploying APIs, loading models, and operating inference workloads.