Investigate runtime threats in Polyaxon agent workloads
Correlate Polyaxon run records with external runtime-security events to investigate agent workloads, preserve evidence, and coordinate containment.
An agent workload can finish with exit code zero and still perform an unexpected action. Conversely, a burst of CPU usage or a failed command may be ordinary debugging rather than an intrusion.
Runtime investigation needs both execution context and security evidence. Polyaxon provides run status, logs, metrics, and artifacts. Host, network, and process-security detection must come from the controls deployed in your infrastructure.
Establish a joinable identity
For each tool execution, preserve the application request ID, tenant or project reference, Polyaxon run UUID, command receipt, and workload image revision. Keep sensitive request content out of identifiers and tags.
Configure external security collection so an event can be associated with the relevant cluster, namespace, Pod, and container. Correlate those records with the Polyaxon run rather than relying on a human-readable name alone.
Use run logs and the execution timeline to understand what the workload was doing around the event. A detector alert without this context can be difficult to reproduce or triage.
Distinguish operational symptoms from security signals
| Observation | Initial investigation |
|---|---|
| Memory limit reached | Input size, resource request, or unexpected allocation |
| Repeated command timeouts | Infinite loop, slow dependency, or abusive request |
| Denied outbound connection | Intended destination, egress policy, or unauthorized access |
| Unexpected sensitive-file access | Granted mounts, process identity, and tool scope |
| New process outside the expected command path | Execution harness and external runtime event details |
These are investigation categories, not proof of malicious intent. Define an expected baseline for each component and revision. A package installation may be normal in one development workspace and forbidden in a production evaluator.
Polyaxon resource monitoring should not be described as an automatic syscall or intrusion detector. Integrate the external sensor that matches your threat model.
Preserve a bounded incident record
Save the exact image reference, resolved configuration, relevant input identifiers, and a sanitized execution receipt. Reference the external alert and its timestamp so responders can inspect the original evidence under the security system's access policy.
A trusted collector can persist approved files through the artifact workflow. Do not indiscriminately upload a compromised filesystem: it may contain secrets, misleading content, or oversized files intended to disrupt collection.
Retain enough context to distinguish a policy failure from a workload defect. For example, an unexpected cloud credential in the environment points to credential placement, even if the process never used it.
Coordinate containment outside the suspect process
Define who can stop the run, revoke credentials, restrict network access, and quarantine affected infrastructure. The generated program should not control the mechanism responsible for containing it.
Stopping a service does not revoke an external token or undo a write. Follow the relevant identity, storage, and incident-response procedures as well. Preserve evidence before cleanup when policy and risk permit.
For future executions, review sandbox connections, resource limits, and network access. A narrower workload contract often reduces the number of ambiguous events.
Feed findings into repeatable checks
Turn a confirmed incident pattern into a controlled regression case using synthetic data and an approved evaluation environment. Record which component and policy revisions address it.
Polyaxon makes these checks repeatable and comparable. The surrounding security infrastructure supplies detection and enforcement. Keeping those responsibilities explicit gives responders a more reliable picture than treating every run metric as a security verdict.