Polyaxon v3 is coming →
NVIDIA NIM

Deploy NVIDIA NIM on Polyaxon

NIM packages an inference runtime and supported model profile. Polyaxon schedules that container on Kubernetes, resolves the NGC credentials and cache, exposes the service, and keeps the release context available for audits and evaluation.

Outcome

A private NIM-backed API with its container version, resource request, secret, cache, and smoke test recorded.

Start the guideServing documentation

Service responsibilities

Polyaxon schedules

The service component fixes the image, model, port, accelerator request, connections, and health boundary.

The runtime serves

The inference engine loads the model and exposes its native or OpenAI-compatible API.

Clients validate

Smoke tests and evaluation operations exercise the endpoint while Polyaxon retains the deployment context.

Prerequisites

  • NVIDIA AI Enterprise or NGC entitlement for the selected NIM and model.
  • An image-pull connection and NGC API key stored as scoped secrets.
  • A supported GPU profile with sufficient memory, local disk or persistent cache, and compatible drivers.

Deploy and validate

  1. 1

    Define the NIM service

    Resolve registry and runtime credentials through Polyaxon connections and request the exact GPU count required by the selected NIM profile.

    version: 1.1kind: componentname: nvidia-nimrun:  kind: service  ports: [8000]  rewritePath: true  connections: [ngc-registry, ngc-api-key, nim-cache]  container:    image: nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.12    env:    - name: NIM_CACHE_PATH      value: /opt/nim/.cache    resources:      requests:        cpu: "8"        memory: 32Gi      limits:        nvidia.com/gpu: "1"

    This is NVIDIA's documented model-specific NIM image. Confirm the selected tag and hardware profile in NVIDIA's support matrix before deployment.

  2. 2

    Submit the operation

    Run the component through the target Polyaxon project, queue, preset, and approval path.

    polyaxon run -f nim.yaml
  3. 3

    Check NIM readiness and model metadata

    Resolve the protected service URL, wait for readiness, and inspect the model endpoint before sending production-shaped requests.

    SERVICE_URL=$(polyaxon ops service --external --url)curl "$SERVICE_URL/v1/models"

Production checklist

Pin the runtime image and model revision; warm the model cache before measuring startup or latency.
Keep the service private by default and add authentication, TLS, rate limits, and network policy deliberately.
Define readiness, liveness, timeout, graceful shutdown, and rollback behavior before production traffic.
Measure latency distributions, throughput, quality, errors, and accelerator memory under representative load.
Retain NIM license and entitlement evidence with the deployment process and keep the NGC key out of component text and logs.

Troubleshooting

The service never becomes ready

Inspect model download, credentials, disk space, GPU memory, runtime flags, port binding, and readiness behavior.

Requests fail through the URL

Verify the service port, rewrite-path setting, authentication header, ingress, network policy, and API path.

Latency degrades under load

Measure queueing, batch settings, context length, KV cache pressure, replica saturation, and storage or network contention.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare AI platformsExplore LLMOps