Using Git shared code in your projects

Developers benefit from collective knowledge when they code with others developing projects on the same Dataiku instance. One of the most common ways to access and share code in Dataiku is through project libraries. When the code you need is available in a Git repository, you can import it into your project library and share that library with other projects for maximum reusability. To learn more about Git and Dataiku, visit Working with Git and Importing code from Git in project libraries.

Prerequisites

• Dataiku >= 14

• Access to an existing project with the following permissions:

  • “Read project content”

  • “Write project content”

Introduction

In this tutorial, you will create a single project library to share among projects. To do so, we will use two projects, based on the Shared Code projects from Learning Projects.

To create the first project, click the New project button, then select Learning projects. In the Developer section, select Project A.

Navigate back to the home screen of your Dataiku instance and follow the same steps for Project B.

Alternatively, you can download the project from Project A and Project B and manually installing them.

Note

All steps in this tutorial that require code and API usage can be performed within Dataiku using a notebook or directly with the UI. For the latest, please see Concept | Shared code.

Creating a code library in Project A

In this section, you will add the monthly_total_transactions Python function to our shared code library in Project A by cloning a library from the Dataiku Academy samples repository. This function applies a group-by to a dataset and returns a new one.

First, open your Project A and create a notebook to write and run your code.

import dataiku

client = dataiku.api_client()
# Work with the provided Project A
PROJECT_KEY_A = "" # Replace with the correct project key
project_a = client.get_project(PROJECT_KEY_A)

project_git = project_a.get_project_git()

# Add an external library to those already present
REPO_URL = 'https://github.com/dataiku/academy-samples.git'
GIT_PATH = 'shared-code'

project_git.add_library(REPO_URL, 'python', 'main', path_in_git_repository=GIT_PATH, as_type='object')

Note

This repository is public, hence the fetch is made with HTTPS.

Important

This code imports files into your library. This is not a link, and you will have to import it again if the git repository is updated.

You will need to force a reload to get the new library on your notebook.

You can then handle the project dataset and apply the new function.

import pandas as pd, numpy as np
# import new library
from total_transactions import monthly_total_transactions

# Read recipe inputs
ecommerce_transactions = dataiku.Dataset("ecommerce_transactions")
ecommerce_transactions_df = ecommerce_transactions.get_dataframe()

# Apply new functions to our dataset
monthly_transactions_df = monthly_total_transactions(ecommerce_transactions_df)

Sharing the library with other projects

Now, open your Project B so you can access its library.

import dataiku
import json


client = dataiku.api_client()
# Get the second project
PROJECT_KEY_B = "" # Replace with the correct project key
project_b = client.get_project(PROJECT_KEY_B)

lib_b = project_b.get_library()

# Read the external library to get the JSON format
external_libraries_json = json.loads(lib_b.get_file("/external-libraries.json").read())
# Add the project key of the first project to access its library
PROJECT_KEY_A = "" # Replace with the correct project key
external_libraries_json['importLibrariesFromProjects'].append(PROJECT_KEY_A)
# Write the new JSON into the library
lib_b.get_file("/external-libraries.json").write(json.dumps(external_libraries_json, indent=2))

Then, you can read the library’s JSON and write the project key for Project A. This way, you’ll have access, in Project B, to all the code available in the Project A library.

import dataiku
# import new library
from total_transactions import monthly_total_transactions

# read recipe inputs
online_retail_dataset = dataiku.Dataset("online_retail_dataset")
online_retail_dataset_df = online_retail_dataset.get_dataframe()

df_gby = monthly_total_transactions(online_retail_dataset_df, date_col= 'InvoiceDate', marketplace_col = 'CustomerID')

To finish, you can apply the shared function to the new dataset.

Important

If you are working on a notebook, you may need to completely reload it to access the new external libraries once imported.

import dataiku
import json


client = dataiku.api_client()
# Get the second project
PROJECT_KEY_B = "" # Replace with the correct project key
project_b = client.get_project(PROJECT_KEY_B)

lib_b = project_b.get_library()

# Read the external library to get the JSON format
external_libraries_json = json.loads(lib_b.get_file("/external-libraries.json").read())
# Add the project key of the first project to access its library
PROJECT_KEY_A = "" # Replace with the correct project key
external_libraries_json['importLibrariesFromProjects'].append(PROJECT_KEY_A)
# Write the new JSON into the library
lib_b.get_file("/external-libraries.json").write(json.dumps(external_libraries_json, indent=2))

Wrapping up

This example demonstrates how to use code shared in a Git repository across multiple projects using the Dataiku API. It underscores Dataiku’s capability to facilitate programmatic interactions and collaboration among various projects, significantly enhancing the reusability and scalability of code within the platform.

Reference documentation

Classes

dataikuapi.DSSClient(host[, api_key, ...])	Entry point for the DSS API client
dataikuapi.dss.project.DSSProjectGit(client, ...)	Handle to manage the git repository of a DSS project (fetch, push, pull, ...)
dataikuapi.dss.project.DSSProject(client, ...)	A handle to interact with a project on the DSS instance.
dataikuapi.dss.projectlibrary.DSSLibrary(...)	A handle to manage the library of a project It saves locally a copy of taxonomy to help navigate in the library All modifications done through this object and related library items are done locally and on remote.

Functions

add_library(repository, local_target_path, ...)	Add a new external library to the project and pull it.
get_dataframe([columns, sampling, ...])	Read the dataset (or its selected partitions, if applicable) as a Pandas dataframe.
get_file(path)	Retrieves a file in the library
get_library()	Get a handle to manage the project library
get_project(project_key)	Get a handle to interact with a specific project.
get_project_git()	Gets an handle to perform operations on the project's git repository.