Measure MIG and GPU time-slicing under co-load

Seven advertised GPU allocation units do not necessarily mean seven isolated GPUs. MIG can divide a physical device into instances with partitioned resources. Time-slicing lets several allocations share a device without equivalent memory and fault isolation.

Our MIG and time-slicing overview explains those choices. This article adds an experiment: run multiple Polyaxon workers together, establish a common measurement interval, and compare per-worker performance. The question is whether a sharing policy suits your workload mix, not whether an idle GPU can be made to look busy.

Start from the allocation contract

Use administrator-prepared nodes with exactly one A100 80 GB physical GPU per node. Keep host configuration, GPU SKU, drivers, CPU allocation, and software aligned. The example assumes these three modes:

Under these particular plugin configurations, each Pod requests nvidia.com/gpu: 1. That identical resource key hides a different unit. A mixed MIG strategy or renamed shared resource uses different keys and requires changing the configuration. NVIDIA MIG guide, GPU Operator sharing documentation.

The same NVIDIA documentation explains that time-slicing distributes execution among GPU processes. Keep one CUDA process per Pod in this experiment. Multiple processes in one workload would change its share. Requesting more advertised replicas also does not establish a proportional compute entitlement.

The benchmark does not enable MIG or reconfigure a production node. Prepare isolated nodes first. Verify the actual allocation and physical device identity: a hostname alone cannot prove co-location on one GPU if the node contains several devices.

Choose workloads and a meaningful baseline

The companion uses ResNet18 and synthetic resident inputs, with batch size 32 and image shape 3×224×224. A worker either runs forward/backward/SGD or runs inference under inference_mode. Each batch ends with CUDA synchronization so its elapsed time includes completed device work.

This isolates a GPU execution workload. It excludes data loading, an HTTP server, application queueing, and model-quality evaluation. The measured p99 is batch execution latency, not a production request-latency estimate. A serving decision needs a subsequent experiment with its real arrival pattern.

Measure a solo training and solo inference worker on the exclusive node. Then collect homogeneous groups of one, four, and seven workers for MIG and time-slicing. The solo MIG run still occupies one slice; it does not inherit the idle capacity of the other six. That is part of the policy being compared.

A useful final group has one inference worker and six training workers. Preserve the two workload types separately. Adding training samples per second to inference samples per second produces a number with no clear operational meaning.

Make workers overlap before timing

Independent jobs can start seconds or minutes apart. If one worker times itself before its peers become active, its result describes a different load. The companion uses a shared POSIX directory for a barrier:

1. Every worker initializes and warms up for twenty batches.
2. It writes a ready record with its cohort configuration.
3. Worker zero waits for every expected record and chooses a start five seconds ahead.
4. Workers begin their forty-five-second windows at that time.
5. Analysis keeps only complete batches inside their common active interval.

Barrier and analysis design, not observed timing data.

Late starts, missing peers, and configuration disagreement fail the cohort. The summary requires all replica IDs and at least 80% window overlap. A run that fails before emitting an artifact makes the cohort incomplete; do not quietly compare the surviving smaller group.

The shared connection is only used before timing. Input tensors stay resident, and batch records are written after measurement. A shared artifact-store URL is not a shared filesystem; configure a writable volume-claim connection named gpu-benchmark-sync at /mnt/benchmark-sync. Polyaxon PVC connections.

Schedule separate Pods on the chosen node

Use a configured project and an agent/queue able to admit all seven jobs concurrently. Each worker requests two CPUs, 4 GiB host memory, and one GPU allocation unit. Queue concurrency and node CPU capacity must allow the whole cohort to reach the barrier.

The supplied Polyaxon component selects the node and mounts the barrier connection:

run:
  kind: job
  connections: [gpu-benchmark-sync]
  environment:
    nodeSelector:
      kubernetes.io/hostname: "{{ node_name }}"

Fragment from the full component, whose container includes the GPU request, source upload, runtime installation, and worker arguments.

Prepare the source and launch a seven-worker MIG training cohort:

git clone https://github.com/polyaxon/polyaxon-examples.git
cd polyaxon-examples/blog/gpu-sharing-experiment
EXPERIMENT_ID=$(uuidgen)
for REPLICA in 0 1 2 3 4 5 6; do
  polyaxon run -f polyaxonfile.yaml -u . -p YOUR_PROJECT --queue YOUR_AGENT/YOUR_GPU_QUEUE -P experiment="$EXPERIMENT_ID" -P node_name=YOUR_MIG_NODE -P mode=mig -P replica="$REPLICA" -P count=7 -P workload=train
done

Approve the runs if required. Use a fresh experiment ID for every cohort. The worker times out if peers never arrive; a queue admitting one job at a time cannot run this experiment. For four-worker groups, submit replica IDs zero through three and set count=4. For time-slicing, use its prepared node and mode=timeslice. The exclusive baseline uses one worker with count=1.

Download each run's worker.json and batches.jsonl into a separate directory. The summary helper checks cohort identity and computes rates over the same overlap:

python summarize.py downloaded/worker-0 downloaded/worker-1 downloaded/worker-2 downloaded/worker-3 downloaded/worker-4 downloaded/worker-5 downloaded/worker-6 --out mig-training-seven.json

Compare useful work and interference

For homogeneous groups, inspect aggregate throughput alongside the per-worker rates and batch latency distribution. A better total can conceal a worker that falls well below its solo performance. For the mixed group, inspect inference tail latency and training progress independently.

Repeat complete cohorts with fresh IDs and retain their variation. Keep power/thermal behavior and host CPU pressure with the device configuration and Polyaxon run UUIDs. A sharing mode that changes effective memory, compute, or interference is not equivalent to simply changing Pod count; make those differences visible in the comparison.

MIG's isolation properties are established by its architecture. This workload measures neither fault containment nor security. Time-slicing's shared memory behavior also means several models must collectively fit on the device even when Kubernetes advertises enough allocations. A GPU allocation count is not a memory quota.

Use utilization as supporting evidence. NVIDIA documents a container-attribution limitation for DCGM Exporter under time-slicing, so do not infer each Pod's utilization from a device-wide sample. The worker's completed batches, timing, and allocation identity provide the direct evidence for this experiment. GPU Operator time-slicing limitations.

The companion is source-reviewed and unexecuted. It supplies no throughput result or recommendation that one mode always wins. Once measured, the useful conclusion is specific: which allocation policy meets the memory, throughput, and latency needs of this workload mix at this occupancy.

References

• NVIDIA MIG user guide: partitioned resources and isolation.
• NVIDIA GPU Operator sharing guide: replica semantics, per-process sharing, and attribution limits.
• CUDA best practices: completed-device timing and synchronization.
• Polyaxon PVC integration: the shared barrier connection.
• MIG versus time-slicing overview: the preceding explanation of the allocation choices.