Using the API to interact with Git for project versioning

In this tutorial, you will learn how to use the API to interact with a project’s Git repository.

Prerequisites

  • Dataiku >= 12.4.2

  • Access to a Dataiku instance with a personal API Key

  • Access to an existing project with the following permissions:

    • “Read project content”

    • “Write project content”

  • Access to an external Git repository with necessary authentication

    • Both you and the Dataiku instance need to have access to this repository.

Note

If the Dataiku instance does not have access to your repository, follow the Configuring per-user SSH in Dataiku tutorial to configure SSH access.

Introduction

The Git integration allows you to perform versioning in Dataiku directly in your IDE using the API client. This tutorial presents a day-to-day use case using Git and project variables.

Note

All steps in this tutorial that require code and API usage can be performed within Dataiku using one of the hosted notebooks. For clarity, most of the steps and code will assume you are interacting with the platform from an external IDE.

Connecting to the instance

A tutorial on connecting to the instance is already available here but to quickly start, here is the code:

Refresher - connecting to the instance
import dataiku

DATAIKU_HOST = ""  # Fill in your Dataiku instance's URL
API_KEY = "" # Fill in your personal API key to access that instance

# connect to your instance
dataiku.set_remote_dss(DATAIKU_HOST, API_KEY)
client = dataiku.api_client()

# list all the projects of the instance 
project_keys = client.list_project_keys()
print(f"N-projects on instance: {len(project_keys)}")

Managing the repository

The first step is to interact with the project via a handle.

# work with a specific sample project
PROJECT_KEY = "DKU_TUT_VARCOD"
project = client.get_project(PROJECT_KEY)

Next, configure the project’s Git remote to point to the external repository. Please note that you will use the SSH address instead of the URL to connect.

REPOSITORY_SSH = "" # e.g. "[email protected]:username/repo.git"

# get the project git
project_git = project.get_project_git()

# set the remote
project_git.set_remote(REPOSITORY_SSH)

If the project is already associated with a remote repository, you can show its address using the get_remote() method.

# read remote
remote = project_git.get_remote()

You can now create a new working branch and switch to it, so that your changes are tracked separately from the branch you started on.

# create a new branch
project_git.create_branch('definitely-not-master')
project_git.switch('definitely-not-master')

You have now created your working branch, on which changes will be tracked. If they exist, list_branches() allows you to check the other branches of the repository.

Add project variables

You can also change variables in a project, for this you can use the update_variables() method.

# add these variables
project.update_variables({
   "country_name": "Germany",
   "merchant_url": "lidl"})

The code adds two variables: country_name and merchant_url. You can retrieve these stored variables using the get_variables() method.

Publishing changes

Once the changes are made, you can get the branch’s status, make commits and push these changes.

project_git.get_status()
project_git.commit(message="add project vars")
project_git.push()

Note

Dataiku enables auto-commit as the default commit-tracking behavior. Hence, if you’ve made changes via the UI, you only have to push() at the end. If you wish to make your commits manually, you can switch to Explicit mode under Settings > Change Management > Commit mode.

The commit() function already includes git add, so files marked as untracked in get_status() are always included in the commit. get_status() also reports how many commits your branch is ahead of or behind its tracked remote branch, via the trackingCount key.

Fig. 1: Explicit mode.

Adapting to organizational processes

Git in workflows

Various organizations have unique ways of working with projects in Dataiku and with Git. Some might require version control systems to track changes, while others might need approval processes before changes are implemented. These Git-related APIs can help accommodate your ways of working within the platform to meet these needs.

When changes are made to a project and pushed to a remote repository, you can retrieve the new branches and the content of your working branch using:

project_git.fetch()
project_git.pull()

Implement a review process

Once branch development finishes, you might need to use standard review processes to validate the changes. Often, this involves having changes reviewed and approved by others. Using these APIs, you can use your IDE to connect to your instance and make unit changes in projects to code and code-like assets. Minor changes can often be easier to review and approve.

Wrapping up

This is an example of a version control workflow using the project’s Git APIs. It demonstrates the first step towards programmatically interacting with and modifying Dataiku projects, enabling further automation and scaling.

Full Tutorial Code
import dataiku

DATAIKU_HOST = ""  # Fill in your Dataiku instance's URL
API_KEY = "" # Fill in your personal API key to access that instance

# connect to your instance
dataiku.set_remote_dss(DATAIKU_HOST, API_KEY)
client = dataiku.api_client()

# list all the projects of the instance 
project_keys = client.list_project_keys()
print(f"N-projects on instance: {len(project_keys)}")

# work with a specific sample project
PROJECT_KEY = "DKU_TUT_VARCOD"
project = client.get_project(PROJECT_KEY)

REPOSITORY_SSH = "" # e.g. "[email protected]:username/repo.git"

# get the project git
project_git = project.get_project_git()

# set the remote
project_git.set_remote(REPOSITORY_SSH)

# read remote
remote = project_git.get_remote()

# create a new branch
project_git.create_branch('definitely-not-master')
project_git.switch('definitely-not-master')

# add these variables
project.update_variables({
   "country_name": "Germany",
   "merchant_url": "lidl"})

# commit & push
project_git.get_status()
project_git.commit(message="add project vars")
project_git.push()

# fetch & pull
project_git.fetch()
project_git.pull()

Reference documentation

Classes

dataikuapi.dss.project.DSSProject(client, ...)

A handle to interact with a project on the DSS instance.

dataikuapi.dss.project.DSSProjectGit(client, ...)

Handle to manage the git repository of a DSS project (fetch, push, pull, ...)

Functions

get_project_git()

Gets an handle to perform operations on the project's git repository.

update_variables(variables[, type])

Updates a set of variables for this project

get_variables()

Gets the variables of this project.

set_remote(url[, name])

Set the URL of the remote repository.

get_remote([name])

Get the URL of the remote repository.

create_branch(branch_name[, commit, ...])

Create a new local branch on the project's git repository and switches to it.

switch(branch_name)

Switch the current repository to the specified branch.

list_branches([remote])

List all branches (local only or local & remote) of the project's git repository.

get_status()

Get the current state of the project's git repository.

commit(message)

Commit pending changes in the project's git repository with the given message.

push([branch_name])

Update the remote repository with the project's local commits.

fetch()

Fetch branches and/or tags (collectively, "refs") from the remote repository to the project's git repository.

pull([branch_name])

Incorporate changes from a remote repository into the current branch on the project's git repository.