Kubernetes audit logs for ML platforms

Kubernetes audit logs record requests to the API server: who or what made a request, which resource it targeted, when it happened, and how the API server responded. In a shared ML cluster, that evidence can connect a changed workload, Secret access, deleted Job, or new role binding to the identity and control path that initiated it.

Audit logs do not record every action inside a container or every change in an external artifact store. They are one layer in an investigation, alongside Polyaxon audit events, run metadata, application logs, and infrastructure telemetry.

Understand the audit event

The Kubernetes auditing documentation describes events generated as requests pass through stages such as request receipt and response completion. Useful fields include:

• authenticated user and groups;
• source addresses and user agent;
• verb, API group, resource, namespace, and object name;
• request URI and response status;
• request and response timestamps;
• audit identifier and annotations added by authorization or admission components.

An identity may be a human, a service account, or another authenticated client. A user agent is useful context but not a trustworthy identity by itself.

Choose audit levels deliberately

An audit policy selects events and the detail recorded at one of four levels:

Higher detail can expose Secrets, tokens, environment values, model configuration, or user data and can materially increase storage and API-server overhead. Start from metadata and add request bodies only for narrow, reviewed cases.

A simplified policy might look like this:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: None
    nonResourceURLs:
      - /healthz*
      - /readyz*
      - /livez*
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]
  - level: Metadata
    verbs: ["create", "update", "patch", "delete"]
  - level: Metadata

This is an illustration, not a complete production policy. Validate API groups, exempted traffic, volume, and managed-provider capabilities for your environment.

Keep sensitive payloads out

Metadata that a Secret was read is usually valuable. Recording the Secret response is dangerous. The same concern applies to ConfigMaps, Pod specifications with environment values, custom resources containing credentials, and admission reviews that include complete objects.

Apply data classification to audit policy and destination access. Encrypt logs in transit and at rest, restrict readers, record access to the log system, and define retention and deletion requirements. Do not forward unrestricted audit bodies to a broadly accessible observability workspace.

Use a reliable backend

Self-managed control planes can write audit files or send events to a webhook backend. Managed Kubernetes services expose provider-specific audit categories and destinations. In either case, design for:

• buffering and behavior when the destination is slow;
• rotation and local disk limits;
• durable centralized retention;
• integrity and tamper resistance;
• time synchronization;
• ingestion delay and dropped-event monitoring;
• access and tenant separation.

Audit configuration sits on the control-plane request path. Test performance and failure behavior before increasing detail or volume.

Correlate an investigation

Begin with a concrete object, identity, or time window. For an unexpectedly deleted training Job:

1. retrieve the Polyaxon run and expected lifecycle event;
2. identify the Kubernetes namespace, resource name, and deletion window;
3. query audit events for the delete verb and object;
4. inspect authenticated identity, groups, source, and response code;
5. determine whether a person, controller, cleanup policy, or retry path initiated it;
6. connect the event to configuration changes and downstream artifact state.

Preserve the original event and record investigation queries separately. Avoid exporting a broad slice of unrelated tenant activity into an incident ticket.

Alert on behavior, not every request

High-value detections include unexpected privilege changes, new cluster-admin bindings, access to protected Secrets, denied requests from unusual identities, changes to admission or audit configuration, and destructive actions outside approved workflows.

Baseline controllers and automation first. A deployment controller legitimately performs many updates; an alert per update creates noise. Detect deviations in identity, scope, resource, or time rather than treating all writes as suspicious.

Join Kubernetes and Polyaxon governance

Polyaxon RBAC governs access to platform resources, and audit logs record supported platform actions. Kubernetes audit logs cover the underlying API server. Retain the mapping among organization, project, user or service identity, Polyaxon run, namespace, and generated Kubernetes resources.

That layered view answers both sides of an operational question: what the ML platform intended to do and what the Kubernetes API accepted. An effective audit program makes that evidence available without turning sensitive request bodies into a new liability.