RunClient

polyaxon.client.run.RunClient(owner=None, project=None, run_uuid=None, client=None, is_offline=None)

RunClient is a client to communicate with Polyaxon runs endpoints.

If no values are passed to this class, Polyaxon will try to resolve the owner, project, and run uuid from the environment:

  • If you have a configured CLI, Polyaxon will use the configuration of the cli.
  • If you have a cached run using the CLI, the client will default to that cached run unless you override the values.
  • If you use this client in the context of a job or a service managed by Polyaxon, a configuration will be available to resolve the values based on that run.

If you intend to create a new run instance or to list runs, only the owner and project parameters are required.

You can always access the self.client to execute more APIs.

  • Properties:

    • project: str.
    • owner: str.
    • run_uuid: str.
    • run_data: V1Run.
    • status: str.
    • namespace: str.
    • client: PolyaxonClient
  • Args:

    • owner: str, optional, the owner is the username or the organization name owning this project.
    • project: str, optional, project name owning the run(s).
    • run_uuid: str, optional, run uuid.
    • client: PolyaxonClient, optional, an instance of a configured client, if not passed, a new instance will be created based on the available environment.
  • Raises:

    • PolyaxonClientException: If no owner and/or project are passed and Polyaxon cannot resolve the values from the environment.

get_inputs

get_inputs(self)

Gets the run's inputs.

  • Returns: dict, all the run inputs/params.

get_outputs

get_outputs(self)

Gets the run's outputs.

  • Returns: dict, all the run outputs/metrics.

refresh_data

refresh_data(self)

Fetches the run data from the api.


update

update(self, data, async_req=False)

Updates a run based on the data passed.

Run API

  • Args:

    • data: Dict or V1Run, required.
    • async_req: bool, optional, default: False, execute request asynchronously.
  • Returns: V1Run, run instance from the response.


create

create(self, name=None, description=None, tags=None, content=None, is_managed=True, pending=None, meta_info=None)

Creates a new run based on the data passed.

N.B. Create methods are only useful if you want to create a run programmatically, if you run a component/operation from the CLI/UI an instance will be created automatically.

This is a generic create function, you can check other methods for creating runs:

  • from yaml: create_from_polyaxonfile
  • from url: create_from_url
  • from hub: create_from_hub

Note: If the content param is not passed, the run will be marked as non-managed.

Run API

  • Args:

    • name: str, optional, it will override the name in the operation if provided.
    • description: str, optional, it will override the description in the operation if provided.
    • tags: str or List[str], optional, list of tags, it will override the tags in the operation if provided.
    • content: str or Dict or V1Operation, optional.
    • is_managed: bool, flag to create a managed run.
    • pending: str, to specify if the run is pending approval (requires human validation) or pending upload.
    • meta_info: dict, meta info to create the run with.
  • Returns: V1Run, run instance from the response.


create_from_polyaxonfile

create_from_polyaxonfile(self, polyaxonfile, name=None, description=None, tags=None, params=None, presets=None, queue=None, nocache=None, cache=None)

Creates a new run based on a polyaxonfile.

N.B. Create methods are only useful if you want to create a run programmatically, if you run a component/operation from the CLI/UI an instance will be created automatically.

Run API

  • Args:

    • polyaxonfile: str, path to the polyaxonfile containing a YAML/Json specification. The polyaxonfile should contain a V1Component or an V1Operation.
    • name: str, optional, it will override the name in the operation if provided.
    • description: str, optional, it will override the description in the operation if provided.
    • tags: str or List[str], optional, list of tags, it will override the tags in the operation if provided.
    • params: dict, optional, a dictionary of parameters that will be used to resolve the component's inputs/outputs.
    • presets: List[str], optional, the name of the presets.
    • queue: str, optional, the name of the queue to assign the run to.
    • nocache: bool, optional, simple flag to disable cache check. If passed and the Polyaxonfile has cache section, it will be patched with disabled: true.
    • cache: bool, optional, simple flag to enable cache check. If passed and the Polyaxonfile has cache section, it will be patched with disabled: false.
  • Returns: V1Run, run instance from the response.


create_from_url

create_from_url(self, url, name=None, description=None, tags=None, params=None, presets=None, queue=None, nocache=None, cache=None)

Creates a new run from a url containing a Polyaxonfile specification.

N.B. Create methods are only useful if you want to create a run programmatically, if you run a component/operation from the CLI/UI an instance will be created automatically.

Run API

  • Args:

    • url: str, url containing a YAML/Json specification. The url's polyaxonfile should contain a V1Component or an V1Operation.
    • name: str, optional, it will override the name in the operation if provided.
    • description: str, optional, it will override the description in the operation if provided.
    • tags: str or List[str], optional, list of tags, it will override the tags in the operation if provided.
    • params: dict, optional, a dictionary of parameters that will be used to resolve the component's inputs/outputs.
    • presets: List[str], optional, the name of the presets.
    • queue: str, optional, the name of the queue to assign the run to.
    • nocache: bool, optional, simple flag to disable cache check. If passed and the Polyaxonfile has cache section, it will be patched with disabled: true.
    • cache: bool, optional, simple flag to enable cache check. If passed and the Polyaxonfile has cache section, it will be patched with disabled: false.
  • Returns: V1Run, run instance from the response.


create_from_hub

create_from_hub(self, component, name=None, description=None, tags=None, params=None, presets=None, queue=None, nocache=None, cache=None)

Creates a new run from the hub based on the component name.

N.B. Create methods are only useful if you want to create a run programmatically, if you run a component/operation from the CLI/UI an instance will be created automatically.

If the component has required inputs, you should pass the params.

Run API

  • Args:

    • component: str, name of the hub component.
    • name: str, optional, it will override the name in the component if provided.
    • description: str, optional, it will override the description in the component if provided.
    • tags: str or List[str], optional, list of tags, it will override the tags in the component if provided.
    • params: dict, optional, a dictionary of parameters that will be used to resolve the component's inputs/outputs.
    • presets: List[str], optional, the name of the presets.
    • queue: str, optional, the name of the queue to assign the run to.
    • nocache: bool, optional, simple flag to disable cache check. If passed and the Polyaxonfile has cache section, it will be patched with disabled: true.
    • cache: bool, optional, simple flag to enable cache check. If passed and the Polyaxonfile has cache section, it will be patched with disabled: false.
  • Returns: V1Run, run instance from the response.


log_status

log_status(self, status, reason=None, message=None, last_transition_time=None, last_update_time=None)

Logs a new run status.

N.B. If you are executing a managed run, you don't need to call this method manually. This method is only useful for manual runs outside of Polyaxon.

N.B you will probably use one of the simpler methods:

  • log_succeeded
  • log_stopped
  • log_failed
  • start
  • end

Run API

  • Args:
    • status: str, a valid Statuses value.
    • reason: str, optional, reason or service issuing the status change.
    • message: str, optional, message to log with this status.
    • last_transition_time: datetime, default now.
    • last_update_time: datetime, default now.

get_statuses

get_statuses(self, last_status=None)

Gets the run's statuses.

Run API

  • Args:

    • last_status: str, a valid Statuses value.
  • Returns: Tuple[str, List[Conditions]], last status and ordered status conditions.


wait_for_condition

wait_for_condition(self, statuses=None, print_status=False)

Waits for the run's last status to meet a condition.

If a list of statuses is passed, it will wait for the condition:

  • last status is one of the statuses passed.

Otherwise, it will wait until the user interrupts the function or when the run reaches a final status.

N.B. if you want to watch the statuses and receive the status/conditions, please use watch_statuses instead which yields the results.


watch_statuses

watch_statuses(self, statuses=None)

Watches run statuses.

If statuses is passed the watch will wait for a condition:

  • last status is one of the statuses passed.

Otherwise, it will watch until the user interrupts it or when the run reaches a final status.

N.B. if you just want to wait for a status condition without expecting a yield, please use wait_for_condition instead

  • Yields: Tuple[status, List[conditions]]: This function will yield the last status and condition for every check.

get_logs

get_logs(self, last_file=None, last_time=None)

Gets the run's logs.

This method return up-to 2000 line logs per request.

  • Returns: V1Logs

watch_logs

watch_logs(self, hide_time=False, all_info=False)

Watches run logs.

  • Args:
    • hide_time: bool, optional, default: False, remove time information from log lines.
    • all_info: bool, optional, default: False, show all information about log lines.

get_events

get_events(self, kind, names, orient=None, force=False)

Gets the run's events

  • Args:
    • kind: str, a valid V1ArtifactKind.
    • names: List[str], list of events to return.
    • orient: str, csv or dict.
    • force: bool, force reload the events.

get_multi_run_events

get_multi_run_events(self, kind, runs, names, orient=None, force=False)

Gets multi-run events.

  • Args:
    • kind: str, a valid V1ArtifactKind.
    • runs: List[str], list of run uuids to return events for.
    • names: List[str], list of events to return.
    • orient: str, csv or dict.
    • force: bool, force reload the events.

get_artifact

get_artifact(self, path, stream=True, force=False)

Gets the run's artifact.

  • Args:

    • path: str, the relative path of the artifact to return.
    • stream: bool, optional, default: True, whether to stream the artifact content.
    • force: bool, force reload the artifact.
  • Returns: str.


download_artifact

download_artifact(self, path, force=False, path_to=None)

Downloads a single run artifact.

  • Args:

    • path: str, the relative path of the artifact to return.
    • path_to: str, optional, path to download to.
    • force: bool, force reload the artifact.
  • Returns: str


download_artifacts

download_artifacts(self, path='', path_to=None, untar=True, delete_tar=True, extract_path=None)

Downloads a subpath containing multiple run artifacts.

  • Args:

    • path: str, the relative path of the artifact to return.
    • path_to: str, optional, path to download to.
    • untar: bool, optional, default: true.
    • delete_tar: bool, optional, default: true.
    • extract_path: str, optional.
  • Returns: str.


upload_artifact

upload_artifact(self, filepath, path=None, untar=False, overwrite=True)

Uploads a single artifact to the run's artifacts store path.

  • Args:

    • filepath: str, the filepath to upload.
    • path: str, optional, path to upload to, otherwise it will be on the run's root path.
    • untar: bool, optional, if the file uploaded is tar.gz and it should be decompressed on the artifacts store.
    • overwrite: bool, optional, if the file uploaded should overwrite any previous content.
  • Returns: str


upload_artifacts

upload_artifacts(self, files, path='', overwrite=True, relative_to=None)

Uploads a multiple artifacts to the run's artifacts store path.

  • Args:

    • files: List[str], list of files to upload.
    • path: str, the relative path of the artifact to return.
    • overwrite: bool, optional, if the file uploaded should overwrite any previous content.
    • relative_to: str, optional, if the path uploaded is not the current dir, and you want to cancel the relative path.
  • Returns: str.


upload_artifacts_dir

upload_artifacts_dir(self, dirpath, path='', overwrite=True, relative_to=None)

Uploads a full directory to the run's artifacts store path.

This function crawls all files to upload and uses upload_artifacts.

  • Args:

    • dirpath: str, the dirpath to upload.
    • path: str, the relative path of the artifact to return.
    • overwrite: bool, optional, if the file uploaded should overwrite any previous content.
    • relative_to: str, optional, if the path uploaded is not the current dir, and you want to cancel the relative path.
  • Returns: str.


delete_artifact

delete_artifact(self, path)

Deletes a single run artifact.

  • Args:
    • path: str, the relative path of the artifact to return.

delete_artifacts

delete_artifacts(self, path)

Deletes a subpath containing multiple run artifacts.

  • Args:
    • path: str, the relative path of the artifact to return.

get_artifacts_tree

get_artifacts_tree(self, path='')

Return the artifacts tree based on the path.

  • Args:

    • path: str, the relative path of the artifact tree to return.
  • Returns: V1ArtifactTree.


stop

stop(self)

Stops the current run.


invalidate

invalidate(self)

Invalidates the current run.


restart

restart(self, override_config=None, copy=False, copy_dirs=None, copy_files=None, name=None, description=None, tags=None)

Restarts the current run

  • Args:

    • override_config: Dict or str, optional, config to use for overriding the original run's config.
    • copy: bool, optional, default: False, to restart with copy mechanism.
    • copy_dirs: List[str], optional, default: None or all in copy mode, list of dirs to copy.
    • copy_files: List[str], optional, default: None or all in copy mode, list of files to copy.
    • name: str, optional, default: None, name to use for the restarted run.
    • description: str, optional, default: None, description to use for the restarted run.
    • tags: list[str], optional, default: None, tags to use for the restarted run.
  • Returns: V1Run instance.


resume

resume(self, override_config=None)

Resumes the current run

  • Args:

    • override_config: Dict or str, optional, config to use for overriding the original run's config.
  • Returns: V1Run instance.


set_description

set_description(self, description, async_req=True)

Sets a new description for the current run.

  • Args:
    • description: str, the description to set.
    • async_req: bool, optional, default: False, execute request asynchronously.

set_name

set_name(self, name, async_req=True)

Sets a new name for the current run.

  • Args:
    • name: str, the name to set.
    • async_req: bool, optional, default: False, execute request asynchronously.

log_inputs

log_inputs(self, reset=False, async_req=True)

Logs or resets new inputs/params for the current run.

Note: If you are starting a run from the CLI/UI polyaxon will track all inputs from the Polyaxonfile, so you generally don't need to set them manually. But you can always add or reset these params/inputs once your code starts running.

  • Args:
    • reset: bool, optional, if True, it will reset the whole inputs state. Note that Polyaxon will automatically populate the inputs based on the Polyaxonfile inputs definition and params passed.
    • async_req: bool, optional, default: False, execute request asynchronously.
    • inputs: **kwargs, e.g. param1=value1, param2=value2, ...

log_outputs

log_outputs(self, reset=False, async_req=True)

Logs a new outputs/results for the current run.

  • Args:
    • reset: bool, optional, if True, it will reset the whole outputs state. Note that Polyaxon will automatically populate some outputs based on the Polyaxonfile outputs definition and params passed.
    • async_req: bool, optional, default: False, execute request asynchronously.
    • outputs: **kwargs, e.g. output1=value1, metric2=value2, ...

log_tags

log_tags(self, tags, reset=False, async_req=True)

Logs new tags for the current run.

  • Args:
    • tags: str or List[str], tag or tags to log.
    • reset: bool, optional, if True, it will reset the whole tags state. Note that Polyaxon will automatically populate the tags based on the Polyaxonfile.
    • async_req: bool, optional, default: False, execute request asynchronously.

log_meta

log_meta(self, reset=False, async_req=True)

Logs meta_info for the current run.

Note: Use carefully! The meta information is used by Polyaxon internally to perform several information.

Polyaxon Client already uses this method to log information about several events and artifacts, Polyaxon API/Scheduler uses this information to set meta information about the run.

An example use case for this method is to update the concurrency of a pipeline to increase/decrease the initial value:

from polyaxon.client import RunClient
client = RunClient()
client.log_meta(concurrency=5)
  • Args:
    • reset: bool, optional, if True, it will reset the whole meta info state.
    • async_req: bool, optional, default: False, execute request asynchronously.
    • meta: **kwargs, e.g. concurrency=10, has_flag=True, ...

start

start(self)

Sets the current run to running status.

N.B. If you are executing a managed run, you don't need to call this method manually. This method is only useful for manual runs outside of Polyaxon.

log_succeeded

log_succeeded(self, message='Operation has succeeded')

Sets the current run to succeeded status.

N.B. If you are executing a managed run, you don't need to call this method manually. This method is only useful for manual runs outside of Polyaxon.

log_stopped

log_stopped(self, message='Operation is stopped')

Sets the current run to stopped status.

N.B. If you are executing a managed run, you don't need to call this method manually. This method is only useful for manual runs outside of Polyaxon.

log_failed

log_failed(self, reason=None, message=None)

Sets the current run to failed status.

N.B. If you are executing a managed run, you don't need to call this method manually. This method is only useful for manual runs outside of Polyaxon.
  • Args:
    • reason: str, optional, reason or service issuing the status change.
    • message: str, optional, message to log with this status.

log_artifact_ref

log_artifact_ref(self, path, kind, name=None, hash=None, content=None, summary=None, is_input=False, rel_path=None)

Logs an artifact reference with custom kind.

Logging a generic file reference to the lineage table:

# Get outputs artifact path
asset_path = tracking.get_outputs_path("test.txt")
with open(asset_path, "w") as f:
    f.write("Artifact content.")
# Log reference to the lineage table
# Name of the artifact will default to test
tracking.log_artifact_ref(path=asset_path, kind=V1ArtifactKind.FILE)
  • Note: This is a generic method that is used by log_file_ref and log_model_ref.

  • Args:

    • path: str, filepath, the name is extracted from the filepath.
    • kind: V1ArtifactKind, the artifact kind.
    • name: str, if the name is passed it will be used instead of the filename from the path.
    • hash: str, optional, default = None, the hash version of the file, if not provided it will be calculated based on the file content.
    • content: the file content.
    • summary: Dict, optional, additional summary information to log about data in the lineage table.
    • is_input: bool, if the file reference is an input or outputs.
    • rel_path: str, optional relative path to the run artifacts path.

log_model_ref

log_model_ref(self, path, name=None, framework=None, summary=None, is_input=False, rel_path=None)

Logs model reference.

Note: The difference between this method and the log_model is that this one does not copy or move the asset, it only registers a lineage reference. If you need the model asset to be on the artifacts_path or the outputs_path you have to copy it manually using a relative path to self.get_artifacts_path or self.get_outputs_path.

# Get outputs artifact path
asset_path = tracking.get_outputs_path("model/model_data.h5")
with open(asset_path, "w") as f:
   f.write("Artifact content.")
# Log reference to the lineage table
# Name of the artifact will default to model_data
tracking.log_model_ref(path=asset_path)
  • Args:
    • path: str, filepath, the name is extracted from the filepath.
    • name: str, if the name is passed it will be used instead of the filename from the path.
    • framework: str, optional ,name of the framework
    • summary: Dict, optional, additional summary information to log about data in the lineage table.
    • is_input: bool, if the file reference is an input or outputs.
    • rel_path: str, optional relative path to the run artifacts path.

log_code_ref

log_code_ref(self, code_ref=None, is_input=True)

Logs code reference as a lineage information with the code_ref dictionary in the summary field.

In offline

  • Args:
    • code_ref: dict, optional, if not provided, Polyaxon will detect the code reference from the git repo in the current path.
    • is_input: bool, if the code reference is an input or outputs.

log_data_ref

log_data_ref(self, name, hash=None, path=None, content=None, summary=None, is_input=True)

Logs data reference.

  • Args:
    • name: str, name of the data.
    • hash: str, optional, default = None, the hash version of the data, if not provided it will be calculated based on the data in the content.
    • path: str, optional, path of where the data is coming from.
    • summary: Dict, optional, additional summary information to log about data in the lineage table.
    • is_input: bool, if the data reference is an input or outputs.
    • content: the data content.

log_file_ref

log_file_ref(self, path, name=None, hash=None, content=None, summary=None, is_input=False, rel_path=None)

Logs file reference.

  • Args:
    • path: str, filepath, the name is extracted from the filepath.
    • name: str, if the name is passed it will be used instead of the filename from the path.
    • hash: str, optional, default = None, the hash version of the file, if not provided it will be calculated based on the file content.
    • content: the file content.
    • summary: Dict, optional, additional summary information to log about data in the lineage table.
    • is_input: bool, if the file reference is an input or outputs.
    • rel_path: str, optional relative path to the run artifacts path.

log_dir_ref

log_dir_ref(self, path, name=None, summary=None, is_input=False, rel_path=None)

Logs dir reference.

  • Args:
    • path: str, dir path, the name is extracted from the path.
    • name: str, if the name is passed it will be used instead of the dirname from the path.
    • summary: Dict, optional, additional summary information to log about data in the lineage table.
    • is_input: bool, if the dir reference is an input or outputs.
    • rel_path: str, optional relative path to the run artifacts path.

log_tensorboard_ref

log_tensorboard_ref(self, path, name='tensorboard', is_input=False, rel_path=None)

Logs dir reference.

  • Args:
    • path: str, path to the tensorboard logdir.
    • name: str, if the name is passed it will be used instead of the dirname from the path.
    • is_input: bool, if the tensorboard reference is an input or outputs
    • rel_path: str, optional relative path to run the artifacts path.

log_artifact_lineage

log_artifact_lineage(self, body, async_req=True)

Logs an artifact lineage.

Note: This method can be used to log manual lineage objects, it is used internally to log model/file/artifact/code refs

  • Args:
    • body: dict or List[dict] or V1RunArtifact or List[V1RunArtifact], body of the lineage.
    • async_req: bool, optional, default: False, execute request asynchronously.

get_namespace

get_namespace(self)

Fetches the run namespace.


delete

delete(self)

Deletes the current run.


list

list(self, query=None, sort=None, limit=None, offset=None)

Lists runs under the current owner - project.

Run API

  • Args:

    • query: str, optional, query filters, please refer to Project PQL
    • sort: str, optional, fields to order by, please refer to Project PQL
    • limit: int, optional, limit of runs to return.
    • offset: int, optional, offset pages to paginate runs.
  • Returns: List[V1Run], list of run instances.


list_children

list_children(self, query=None, sort=None, limit=None, offset=None)

Lists run's children if the current run has a pipeline.

Run API

  • Args:

    • query: str, optional, query filters, please refer to Project PQL
    • sort: str, optional, fields to order by, please refer to Project PQL
    • limit: int, optional, limit of runs to return.
    • offset: int, optional, offset pages to paginate runs.
  • Returns: List[V1Run], list of run instances.


sync_events_summaries

sync_events_summaries(self, last_check, events_path)

Syncs all tracked events and auto-generates summaries and lineage data.

Note: Both in-cluster and offline modes will manage syncing events summaries automatically, so you should not call this method manually.


persist_offline_run

persist_offline_run(self, artifacts_path)

Persists an offline run to a local path.

Note: You generally do not need to call this method manually, When the offline mode is enabled, this method is triggered automatically at the end.