Run batch LLM evaluations on Kubernetes
Design reliable batch LLM evaluations with stable shards, bounded concurrency, GPU-aware placement, resumable results, and complete aggregation.
EvaluationKubernetesLLMOps
All blog articles
Practical guides to building, running, and improving ML and AI in production.
All articles
Page 4 of 20
Build a shared library of agent evaluators
Package application-owned agent evaluators as versioned Polyaxon components with typed contracts, reusable execution profiles, and consistent result evidence.
AI AgentsEvaluationAutomation
ML infrastructure explained for business teams
Understand what ML infrastructure pays for, how it affects delivery and reliability, and how to evaluate an investment using measurable workflow outcomes.
MLOpsInfrastructureGuides
Operating long-running AI agents
Choose jobs, services, queues, and capacity controls for agent workloads using task duration, dependency limits, resume latency, and execution cost.
AI AgentsOrchestrationKubernetes
An AI agent deployment checklist for the first production release
Build an agent release checklist around Polyaxon qualification jobs, versioned components, artifact reports, termination settings, and manual approval.
AI AgentsLLMOpsGuides
Roll out a new model when every GPU is occupied
Plan GPU inference rollouts around surge capacity, temporary reduced availability, model warmup, and request draining with a worked Deployment scenario.
KubernetesInferenceDeployment
Secure enterprise AI from data to deployment
Apply security controls across data, training, evaluation, artifacts, deployment, and operation without slowing every AI workload equally.
SecurityGovernanceMLOps
Share CPU and memory across containers in a Pod
Use Pod-level resource budgets for cooperating containers, understand the remaining isolation boundaries, and evaluate the fit for ML services.
KubernetesResourcesInference
Place ML workloads close to their data
Design region-aware ML execution around dataset, registry, model, cache, artifact, and service locality without confusing proximity with data residency.
InfrastructureKubernetesGuides
Build a reliable foundation for autonomous agents
Build an operational foundation for autonomous agents using Polyaxon workload profiles, durable state, bounded authority, evaluation, and recovery ownership.
AI AgentsPlatform EngineeringReliability
Continuous AI red teaming in CI/CD
Turn AI security findings into repeatable CI checks with versioned cases, complete result manifests, explicit release gates, and retained evidence.
Red TeamingPipelinesEvaluation
Production LLM systems: Where to invest after the prototype
Use Polyaxon run tracking, comparison dashboards, resource monitoring, and repeatable evaluation to decide what to improve after an LLM prototype.
LLMOpsMLOpsGuides
Trace a failed training run to an unhealthy GPU
Connect Kubernetes DRA allocations, device health, and application failures without confusing missing telemetry with healthy hardware.
KubernetesGPUTroubleshooting
Docker build caching for ML workloads on Kubernetes
Reduce container build and startup time for ML jobs with reusable dependency layers, persistent build caches, and deliberate image and model caching.
DockerKubernetesInfrastructure
Durable execution for AI agents
Build agent recovery around committed checkpoints, versioned inputs, safe side effects, and approval state so interrupted tasks can continue reliably.
AI AgentsOrchestrationGuides
Move faster with risk-tiered AI delivery
Use consequence-based AI risk tiers to apply proportionate data, evaluation, security, approval, deployment, monitoring, and incident controls.
GovernanceSecurityMLOps
Understand Kubernetes workload-aware preemption
Understand how Kubernetes preempts capacity for PodGroups, choose disruption behavior for training and evaluation, and keep recovery separate from priority.
KubernetesSchedulingDistributed Training
Secure AIOps automation for ML platforms
Use AIOps to investigate ML incidents and automate low-risk remediation without giving an AI system unrestricted production access.
AIOpsSecurityObservability
Build guardrails for AI-generated code
Control what coding agents can read, change, execute, merge, and deploy with layered checks from context selection through production.
GuardrailsSecurityEvaluation
Designing a control plane for AI agents
Separate live agent requests from versioning, evaluation, policy, rollout, identity, evidence, and recovery across the application lifecycle.
AI AgentsOrchestrationObservability
Gang scheduling for distributed training
Understand how gang scheduling prevents partial distributed jobs from holding GPUs, how minimum membership works, and what Polyaxon supports.
SchedulingKubernetes
MCP security testing: tools, permissions, and untrusted content
Build MCP security tests for tool discovery, authorization, injected tool results, session identity, and approval boundaries in AI agents.
MCPRed TeamingAI Agents
Design open infrastructure for portable AI workloads
Keep AI workloads portable with explicit execution contracts, open packaging and telemetry, hardware abstraction, data boundaries, and tested migration paths.
InfrastructureKubernetesMLOps
Designing the runtime layer for AI agents
Design agent execution around durable state, safe tool retries, approval waits, and recovery, with clear responsibilities for Polyaxon and your agent framework.
AI AgentsOrchestrationGuides