Polyaxon v3 is coming →

Common Git commands for data-scientists

Common Git commands for data-scientists and machine learning engineers

September 13, 2019by Polyaxon

This guide introduces common Git commands through one small workflow: prepare a branch, change a training configuration, inspect the change, and preserve its exact revision. The examples assume Git is installed, your author identity is configured, and the repository is one you can access.

Use the sections as a command reference. In a normal change, initialize or clone once, create a branch, edit, inspect, stage, commit, then push for review. Paths such as configs/baseline.yaml represent files in your team's project.

Git init

Create an empty Git repository or reinitialize an existing one on the local machine.

git init ticket-classifier
cd ticket-classifier

Use this for a new local project. If your team already has a repository, clone it instead.

Reference: Please refer to git-init

Git clone

Make a local copy of an existing remote repository.

git clone https://github.com/polyaxon/polyaxon-quick-start.git
cd polyaxon-quick-start

For your team's work, substitute its repository URL. Cloning the public example gives you a local copy, not permission to push to the upstream project.

Reference: Please refer to git-clone

Git add

Add file(s) to the index to be saved.

After editing your team's training configuration, stage that specific file:

git add configs/baseline.yaml
git diff --cached

The path is an example file in your own project. Reviewing the staged diff helps avoid committing generated data, credentials, or unrelated notebook changes.

Reference: Please refer to git-add

Git status

Show the working-tree and staging state.

git status --short
git diff

git diff shows unstaged changes; git diff --cached shows what the next commit will include.

Reference: Please refer to git-status

Git commit

Create a new commit based on the current index of the saved file(s).

git commit -m "Adjust baseline classifier configuration"
git rev-parse HEAD

The commit identifies the saved source state. Changes made after that commit are not part of it.

Reference: Please refer to git-commit

Git push

Update the remote repository with the saved changes (commits).

git push -u origin experiment/class-weighting

This assumes you created the branch below and origin points to a repository you can write to. For a fork-based workflow, push to your fork and open a pull request against the upstream project.

Reference: Please refer to git-push

Git pull

Fetch changes (commits) from remote repository and integrate with the local machine.

git pull --ff-only

Run this on a branch with a configured upstream. It stops when the histories have diverged instead of choosing a merge or rebase policy for you; inspect the divergence and follow your team's policy.

Reference: Please refer to git-pull

Git branch

Branch management: list, create, rename, and delete branches.

git branch
git switch -c experiment/class-weighting

Create the experiment branch before editing the configuration. Git switch makes branch changes explicit.

Reference: Please refer to git-branch

Git checkout

Switch branches or restore working tree files.

checkout supports both tasks. For a clearer distinction, use git switch to change branches and git restore for file restoration. To unstage a file while keeping your edits:

git restore --staged configs/baseline.yaml

This changes the staging area, not the working file. Restoring a working file is a different operation and can discard uncommitted edits.

Reference: Please refer to git-checkout

Git merge

Join two or more development histories together.

After reviewing the experiment branch, a local integration workflow could be:

git switch main
git pull --ff-only
git merge experiment/class-weighting

Use your repository's actual default branch name. Teams that require pull requests should perform the merge through that review process instead.

Reference: Please refer to git-merge

Connect the revision to an experiment

Git identifies source files; it does not automatically identify a changing dataset or model checkpoint. Keep large artifacts in their appropriate store, record retained dataset versions or manifests, and ignore local caches and secrets before staging.

In Polyaxon, Git initialization can fetch a chosen source revision for a workload. Record or select the exact commit used, and preserve the data identity and environment with the run. The GitHub Actions workflow shows how to bind an automated submission to the triggering revision. A SHA in a run description is useful context only if the workload actually executes that source state.