Trigger Polyaxon workflows from Kafka events

A Kafka event can signal that a dataset is ready, an evaluation is due, or a model needs retraining. Connect these events to Polyaxon with a consumer that validates each message and submits a predefined operation through the Python SDK or CLI.

You implement and operate the consumer; this workflow does not install Kafka brokers or a managed event bridge. Keep Kafka responsible for event delivery and Polyaxon responsible for the submitted workloads. For reading records inside a job, see Kafka data pipelines.

Define the event contract

Prefer a small event containing a stable event ID, a schema version, and a reference to an immutable dataset. Do not put training data, credentials, arbitrary shell commands, or unrestricted component definitions in the message.

Map allowed event types to components controlled by your team. Validate the input schema, dataset location, target project, and permitted parameters before submitting anything. For example, a dataset.ready event might select a known training operation and supply its dataset URI.

Connect the consumer to Polyaxon

1. Give the consumer network access to its Kafka brokers and the Polyaxon API.
2. Configure Kafka credentials separately from Polyaxon authentication. Use a Polyaxon identity limited to the intended projects and operations.
3. Install a compatible Polyaxon client in the consumer's environment. Configure POLYAXON_HOST and supply authentication through your secret-management system; see client authentication.
4. After validation and duplicate checks, submit the trusted operation with RunClient.create_from_polyaxonfile, or a controlled Hub component with RunClient.create_from_hub.
5. Persist the returned run UUID alongside the event ID. Submission means the run was accepted, not that training succeeded; observe its status separately.

Use queues and bounded submission concurrency so an event burst does not create an uncontrolled workload backlog. For multi-stage processing, submit a Polyaxon DAG instead of coordinating every training step in the consumer.

Design for replay and uncertain submissions

With manual offset management, persist the event-to-run mapping before committing the corresponding offset. Keep pending, submitted, and needs-reconciliation states in a ledger that survives restarts and supports concurrent consumers. In a batch, do not commit past an earlier message whose handoff is still unresolved.

There is still a failure window: Polyaxon might accept a run immediately before the consumer loses the response or crashes. A local duplicate check followed by a create request is not an atomic transaction. Keep a durable submission ledger, associate a stable event identifier with the run, and reconcile uncertain outcomes before retrying. A run name or tag helps reconciliation but is not a uniqueness constraint.

Do not describe this bridge as exactly-once delivery. Kafka's transaction guarantees do not automatically include a separate Polyaxon API call. Make downstream work safe to repeat as well.

Operational checks

• Send one authorized event and confirm that its run UUID is recorded.
• Replay that event and verify your duplicate-handling policy.
• Simulate an API timeout after submission and confirm that reconciliation does not blindly create another run.
• Route invalid messages to a controlled failure path; alert on consumer lag, rejected submissions, and unresolved events.

These checks belong to your integration's deployment process; they are not provided by Kafka or Polyaxon configuration alone.