Contents Menu Expand Light mode Dark mode Auto light/dark mode
Dataiku Developer Guide
Dataiku
  • Academy
    • Join the Academy
      Benefit from guided learning opportunities →
      • Quick Starts
      • Learning Paths
      • New Features
      • Certifications
      • Academy Discussions
  • Community
      • Explore the Community
        Discover, share, and contribute →
      • Learn About Us
      • Ask A Question
      • What's New?
      • Discuss Dataiku
      • Using Dataiku
      • Setup And Configuration
      • General Discussion
      • Plugins & Extending Dataiku
      • Product Ideas
      • Programs
      • Frontrunner Awards
      • Dataiku Neurons
      • Community Resources
      • Community Feedback
      • User Research
  • Documentation
    • Reference Documentation
      Comprehensive specifications of Dataiku →
      • User's Guide
      • Specific Data Processing
      • Automation & Deployment
      • APIs
      • Installation & Administration
      • Other Topics
  • Knowledge
    • Knowledge Base
      Articles and tutorials on Dataiku features →
      • User Guide
      • Admin Guide
      • Dataiku Solutions
      • Dataiku Cloud
  • Developer
    • Developer Guide
      Tutorials and articles for developers and coder users →
      • Getting Started
      • Concepts and Examples
      • Tutorials
      • API Reference
  • Getting started
    • Introduction
    • Development environment
    • Basic workflow
    • MLOps lifecycle
    • The Dataiku Python packages
    • Quickstart Tutorial
      • Step 1: Prepare the input dataset for ML modeling
      • Step 2: Test different Machine Learning models for heart failures prediction
      • Step 3: Create a Dataiku Saved Model using the best-performing model
  • Concepts and examples
    • Datasets
      • Datasets (reading and writing data)
      • Datasets (other operations)
    • Managed folders
    • The main API client
    • Pyspark recipes
    • Projects
    • Recipes
    • Scenarios
    • Flow creation and management
    • LLM Mesh
    • Visual Machine learning
    • Experiment Tracking
    • Statistics worksheets
    • API Designer & Deployer
    • Project Deployer
    • Static insights
    • Jobs
    • Authentication information and impersonation
    • Importing tables as datasets
    • Wikis
    • Discussions
    • Performing SQL, Hive and Impala queries
    • SQL Query
    • Project libraries
    • Meanings
    • Users and groups
    • Connections
    • Scenarios (in a scenario)
    • Code envs
    • Plugins
    • Project folders
    • Dataiku Applications
    • Feature Store
    • Streaming Endpoints
    • Snowpark
    • Databricks Connect
    • Metrics and checks
    • Model Evaluation Stores
    • Administration
    • Utilities
    • Clusters
    • Code Studios
    • Fleet Manager
      • The main FMClient class
      • Fleet Manager Instances
      • Fleet Manager Virtual Networks
      • Fleet Manager Instance Templates
      • Fleet Manager Tenant
      • Fleet Manager Future
    • Dataiku Govern
      • The main GovernClient class
      • Artifacts
        • Govern Artifacts
        • Govern Artifact Search
        • Retrieve files from Govern artifacts
        • Retrieve time series data from Govern artifacts
      • Administration
        • Authentication information and impersonation
        • Users and groups
        • Other administration tasks
        • Govern Roles and Permissions
          • Add a rule to assign a role to a new user
          • Bootstrap default groups matching provided roles
        • Utilities
      • Advanced Govern
        • Govern Custom Pages
        • Govern Custom Pages Handler
        • Automate actions with Govern Hooks
        • Govern Blueprint Designer
          • View blueprints and blueprint versions
          • Fork a Govern project blueprint version
          • Script a blueprint migration
          • Govern Blueprint HTML Documentation
      • Govern Helpers
        • Dataiku and Govern items helpers functions
      • Govern Deployments
        • Governance checks with multi envs
    • Workspaces
    • Data Collections
    • Data Quality
    • Webapps
  • Tutorials
    • Administration
      • Retrieve AWS Credentials from Dataiku API
    • Data Engineering
      • Leveraging SQL in Python & R
      • Using Snowpark Python in Dataiku: basics
      • Using Databricks Connect Python in Dataiku: basics
      • Data quality assessments (SQL Datasets)
    • Developer tools
      • Dataiku API
        • Dataiku’s Public API: A Comprehensive Guide
        • Using Dataiku’s Python packages
        • Dataiku REST API
      • IDE extensions
        • VSCode extension for Dataiku DSS
        • PyCharm plugin for Dataiku DSS
      • Code Studio
        • Using VSCode for Code Studios
        • Using JupyterLab in Code Studios
        • Using RStudio in Code Studios
      • Miscellaneous
        • Running unit tests on project libraries
        • Using the API to interact with git for project versioning
    • Generative AI
      • LLM-based agent
      • Image Generation
      • Natural language
        • Programmatic RAG with Dataiku’s LLM Mesh and Langchain
        • Using LLM Mesh to benchmark zero-shot classification models
        • Zero-shot text classification with the LLM Mesh
        • Few-shot classification with the LLM Mesh
        • Using standard OpenAI API calls via the LLM Mesh
      • Webapps
        • API Endpoint on LLM
        • Dahs: API LLM based agent
        • Dash: GPT-powered web app assistant
        • Dash: LLM based agent
        • Gradio: First application
        • Gradio: LLM based agent
        • Voilà: First application
        • Voilà: LLM based agent
    • Machine Learning
      • Experiment Tracking
        • Experiment Tracking with the PythonModel module
        • Experiment Tracking for NLP with Keras/Tensorflow
        • Experiment tracking with Catboost
        • Experiment tracking with LightGBM
        • Experiment tracking with scikit-learn
      • Pre-trained Models
        • Load and re-use a TensorFlow Hub model
        • Load and re-use a PyTorch model
        • Load and re-use a Hugging Face model
        • Load and re-use a SentenceTransformers word embedding model
        • Load and re-use a spaCy named-entity recognition model
        • Load and re-use an NLTK tokenizer
      • Model Import
        • Importing serialized scikit-learn pipelines as Saved Models for MLOps
      • Model Export
        • Wrapping an exported model in a CLI tool
    • Plugins development
      • Foreword
      • Creating and configuring a plugin
      • Setting up a dedicated instance for developing plugins
      • Versioning for plugin
        • Git integration for plugins
        • Plugin version management with Git
      • Datasets
        • Creating a plugin Dataset component
      • File Format
        • Writing a plugin File Format component to allow ICal import in Dataiku
      • Macros
        • Creating a plugin Macro component
        • Writing a macro for project creation
        • Writing a macro for managing regression tests
      • Processor plugin component
        • Creating a plugin Processor component
      • Prediction algorithm
        • Creating a plugin Prediction Algorithm component
      • Recipes
        • Creating a plugin Recipe component
        • Writing a custom recipe to remove outliers from a dataset
      • Webapps
        • Turning an existing webapp into a webapp plugin component
    • Webapps
      • Common subjects on webapps
        • Accessible resources from webapps
        • Creating an API endpoint from webapps
        • Querying the LLM from an headless API
        • Impersonation with webapps
      • Bokeh
        • Bokeh: Your first webapp
      • Dash
        • Create an empty dash webapp
        • Dash: your first webapp
        • Display a Bar chart using Dash
        • Create a simple admin project dashboard using Dash.
        • Create a simple multi-page webapp
        • Uploading or downloading files with Managed Folders in Dash
        • How to create a form for data input?
        • Simple scoring application
        • Creating an API endpoint for using an LLM-Based agent
        • Using Dash and LLM Mesh to build a GPT-powered web app assistant
        • Creating a Dash application using an LLM-based agent
      • Gradio
        • Gradio: your first web application
        • Creating a Gradio application using an LLM-based agent
      • Standard (HTML, CSS, JS)
        • HTML/CSS/JS: your first webapp
        • Adapt a D3.js Template in a Webapp
        • Use Custom Static Files (Javascript, CSS) in a Webapp
        • Upload to Dataiku DSS in a Webapp
        • Download from a Dataiku DSS Webapp
        • How to create a form for data input?
        • Simple scoring application
      • Streamlit
        • Streamlit: your first webapp
      • Use your own framework.
        • Basic setup: Code studio template
        • Basic setup: Quickstart with Angular & Vue Templates
        • Basic setup: Deploy your web application
        • Advanced setup: Code Studio template creation
        • Advanced setup: Web application creation and publication
      • Voilà
        • Voilà: your first web application
        • Creating a Voilà application using an LLM-based agent
  • API Reference
    • Python
      • Index
      • Datasets
      • Managed folders
      • The main API client
      • Interaction with Pyspark
      • Projects
      • Recipes
      • Scenarios
      • Flow creation and management
      • LLM Mesh
      • Machine learning
      • Experiment Tracking
      • Statistics worksheets
      • API Designer & Deployer
      • Project Deployer
      • Static insights
      • Jobs
      • Authentication information and impersonation
      • Importing tables as datasets
      • Wiki
      • Discussions
      • Performing SQL, Hive and Impala queries
      • SQL Query
      • Project libraries
      • Meanings
      • Users and groups
      • Connections
      • Scenarios (in a scenario)
      • Code envs
      • Plugins
      • Project folders
      • Macros
      • Dataiku applications
      • Feature Store
      • Streaming endpoints
      • Snowpark
      • Databricks Connect
      • Unified Monitoring
      • Metrics and checks
      • Model Evaluation Stores
      • Other administration tasks
      • Utilities
      • Index of the dataiku package
      • Index of the dataikuapi package
      • API for plugin components
        • API for plugin recipes
        • API for plugin datasets
        • API for plugin formats
        • API for plugin FS providers
      • Clusters
      • Code Studios
      • Fleet Manager
      • Govern
      • Workspaces
      • Data Collections
      • Data Quality
      • Webapps
      • Dashboards
      • Insights
      • Messaging channels
You are viewing the developer guide for version 13 of DSS.
  • Go back to the homepage
  • Tutorials
  • Data Engineering
Back to top

Data Engineering#

The tutorials of this section aim at teaching the user on advanced use-cases for data preparation and feature engineering.

  • Leveraging SQL in Python & R
  • Using Snowpark Python in Dataiku: basics
  • Using Databricks Connect Python in Dataiku: basics
  • Data quality assessments (SQL Datasets)
Next
Leveraging SQL in Python & R
Previous
Retrieve AWS Credentials from Dataiku API
Copyright © 2024, Dataiku
Made with Sphinx and @pradyunsg's Furo