Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode
Dataiku Developer Guide
Dataiku
  • Academy
    • Join the Academy
      Benefit from guided learning opportunities →
      • Quick Starts
      • Learning Paths
      • Certifications
      • Academy Discussions
  • Community
      • Explore the Community
        Discover, share, and contribute →
      • Learn About Us
      • Ask A Question
      • What's New?
      • Discuss Dataiku
      • Using Dataiku
      • Setup And Configuration
      • General Discussion
      • Plugins & Extending Dataiku
      • Product Ideas
      • Programs
      • Frontrunner Awards
      • Dataiku Neurons
      • Community Resources
      • Community Feedback
      • User Research
  • Documentation
    • Reference Documentation
      Comprehensive specifications of Dataiku →
      • User's Guide
      • Specific Data Processing
      • Automation & Deployment
      • APIs
      • Installation & Administration
      • Other Topics
  • Knowledge
    • Knowledge Base
      Articles and tutorials on Dataiku features →
      • Get Started
      • User Guide
      • Dataiku Cloud
      • Additional Offerings
      • Admin Guide
  • Developer
    • Developer Guide
      Tutorials and articles for developers and coder users →
      • Getting Started
      • Concepts and Examples
      • Tutorials
      • API Reference
  • Getting started
    • Accelerate AI success with Dataiku
    • Coding with Dataiku
      • My first Code Studio
      • PyCharm plugin for Dataiku
      • VSCode extension for Dataiku
    • Basic workflow
    • MLOps lifecycle
    • Dataiku Python APIs
      • Dataiku’s Public API: A Comprehensive Guide
      • Using Dataiku’s Python packages
      • Dataiku REST API
  • Concepts and examples
    • Datasets
      • Datasets (reading and writing data)
      • Datasets (other operations)
    • Managed folders
    • The main API client
    • Pyspark recipes
    • Projects
    • Recipes
    • Scenarios
    • Flow creation and management
    • LLM Mesh
    • Agents
    • Visual Machine learning
    • Experiment Tracking
    • Statistics worksheets
    • API Designer
      • Calling another endpoint
      • Access HTTP Request Metadata
      • Return custom HTTP responses
    • API Deployer
    • Project Deployer
    • Static insights
    • Jobs
    • Authentication information and impersonation
    • Importing tables as datasets
    • Wikis
    • Discussions
    • Performing SQL, Hive and Impala queries
    • SQL Query
    • Project libraries
    • Meanings
    • Users and groups
    • Connections
    • Scenarios (in a scenario)
    • Code envs
    • Plugins
    • Project folders
    • Dataiku Applications
    • Feature Store
    • Streaming Endpoints
    • Snowpark
    • Databricks Connect
    • Bigframes
    • Metrics and checks
    • Model Evaluation Stores
    • Administration
    • Utilities
    • Clusters
    • Code Studios
    • Fleet Manager
      • The main FMClient class
      • Fleet Manager Instances
      • Fleet Manager Virtual Networks
      • Fleet Manager Instance Templates
      • Load Balancers
      • Fleet Manager Accounts
      • Fleet Manager Tenant
      • Fleet Manager Future
    • Cloud (Launchpad)
    • Dataiku Govern
      • The main GovernClient class
      • Artifacts
        • Govern Artifacts
        • Govern Artifact Workflow
        • Govern Artifact Search
        • Retrieve files from Govern artifacts
        • Retrieve time series data from Govern artifacts
      • Administration
        • Authentication information and impersonation
        • Users and groups
        • Other administration tasks
        • Govern Roles and Permissions
          • Add a rule to assign a role to a new user
          • Bootstrap default groups matching provided roles
        • Utilities
      • Advanced Govern
        • Scripts for Governance Policies
        • Govern Custom Pages
        • Govern Custom Pages Handler
        • Automate actions with Govern Hooks
        • Govern Actions
        • Govern Blueprint Designer
          • View blueprints and blueprint versions
          • Fork a Govern project blueprint version
          • Script a blueprint migration
          • Govern Blueprint HTML Documentation
      • Govern Helpers
        • Dataiku and Govern items getter helpers functions
        • Dataiku and Govern items modifier helpers functions
      • Govern Deployments
        • Governance checks with multi envs
    • Workspaces
    • Catalog Collections
    • Enterprise Asset Library
    • Data Quality
    • Webapps
  • Tutorials
    • Administration
      • Monitoring Disk Usage
      • Retrieve AWS Credentials from the Dataiku API
    • AI Governance
      • Hooks and Scenarios
      • Manipulating Artifact Fields with Govern Custom Actions
    • Data Engineering
      • Data quality assessments (SQL Datasets)
      • Leveraging SQL in Python & R
      • Using Bigframes Python in Dataiku: basics
      • Using Dask DataFrames in Dataiku
      • Using Snowpark Python in Dataiku: basics
      • Using Spark RAPIDS on Dataiku
      • Sessionization in SQL, Hive, and Python
    • Developer tools
      • Code Studio
        • My first Code Studio
        • Code assistants in Code Studios
          • Best practices with code assistants
          • Using Claude Code
          • Using Codex
          • Using GitHub Copilot
          • Using Gemini CLI
          • Using OpenCode
        • Editing Code
        • Editing Code Agent
        • Editing Webapp
        • Using JupyterLab
        • Using RSudio
      • Git usage in Dataiku
        • Git basics in Dataiku
        • Configuring per-user SSH in Dataiku
        • Git collaboration
        • Using the API to interact with Git for project versioning
      • Miscellaneous
        • Using external libraries for projects
        • Running unit tests on project libraries
    • Generative AI
      • Agents & tools
        • Creating a Custom Python Tool
        • Creating a custom tool
        • Leveraging a custom tool in a Visual Agent
        • Creating and using a Code Agent
        • Integrating an agent framework
        • Creating a custom agent
        • Creating a custom LLM Connection
        • Multi agents: sequential workflow
        • LLM Mesh agentic applications
          • Defining and using tools with the LLM Mesh
          • Creating an LLM-based agent that uses multiple tools
          • Building a Web Application with the agent
        • Connecting to an external Vector Store
        • Model Context Protocol (MCP)
          • Building your MCP Server in Dataiku
        • Building and using an agent with Dataiku’s LLM Mesh and Langchain
        • Using a Dataiku Agent in an external application
        • Using the LLM Mesh to parse and output JSON objects
        • Building Auto Prompt Strategies with DSPy in Dataiku
        • Agent logging
        • Adding traces to your Agent
      • Pre-training an LLM
      • Aligning an LLM
      • Multimodal capabilities
        • Images Generation
        • Images Captioning
        • Generative Adversarial Networks
      • Natural language
        • Programmatic RAG with Dataiku’s LLM Mesh and Langchain
        • RAG: Improving your Knowledge Bank retrieval
        • Creating and using a Knowledge Bank
        • Using LLM Mesh to benchmark zero-shot classification models
        • Zero-shot text classification with the LLM Mesh
        • Few-shot classification with the LLM Mesh
        • Using OpenAI-compatible API calls via the LLM Mesh
        • Customizing a Text Embedding Model for RAG Applications
      • Webapps
        • API Endpoint on LLM
        • Dash: API LLM based agent
        • Dash: GPT-powered web app assistant
        • Dash: LLM based agent
        • Gradio: First application
        • Gradio: LLM based agent
        • Voilà: First application
        • Voilà: LLM based agent
    • Machine Learning
      • Quickstart Tutorial
        • Step 1: Prepare the input dataset for ML modeling
        • Step 2: Test different Machine Learning models for heart failures prediction
        • Step 3: Create a Dataiku Saved Model using the best-performing model
      • Using LIME for Model Explainability
      • Predictive Model Maintenance for Binary Classification
      • Using Reinforcement Learning for Hyperparameter Tuning
      • Transfer Learning Techniques
        • Transductive Transfer Learning
        • Unsupervised Transfer Learning
      • Experiment Tracking
        • Experiment Tracking with the PythonModel module
        • Experiment Tracking for NLP with Keras/Tensorflow
        • Experiment tracking with Catboost
        • Experiment tracking with LightGBM
        • Experiment tracking with scikit-learn
      • Pre-trained Models
        • Load and re-use a TensorFlow Hub model
        • Load and re-use a PyTorch model
        • Load and re-use a Hugging Face model
        • Load and re-use a SentenceTransformers word embedding model
        • Load and re-use a spaCy named-entity recognition model
        • Load and re-use an NLTK tokenizer
      • Model Import
        • Importing serialized scikit-learn pipelines as Saved Models for MLOps
      • Model Export
        • Wrapping an exported model in a CLI tool
        • Deploying a model to an edge device with AWS IoT Greengrass
      • Distributed ML Training using Ray
      • Scanning Models with Protect AI Guardian Using Python
    • Plugins development
      • Foreword
      • Creating and configuring a plugin
      • Setting up a dedicated instance for developing plugins
      • Versioning for plugin
        • Git integration for plugins
        • Plugin version management with Git
      • Datasets
        • Creating a plugin Dataset component
      • File Format
        • Writing a plugin File Format component to allow ICal import in Dataiku
      • LLM Mesh
        • Creating a custom agent
        • Creating a custom tool
        • Creating a custom guardrail
        • Creating a custom LLM Connection
        • Creating a structured agent block component
      • Macros
        • Creating a plugin Macro component
        • Writing a macro for project creation
        • Writing a macro for managing regression tests
      • Parameter sets
        • Creating a parameter set component
      • Processor plugin component
        • Creating a plugin Processor component
      • Project Standards Check component
        • Creating a Project Standards Check component
      • Prediction algorithm
        • Creating a plugin Prediction Algorithm component
      • Recipes
        • Creating a plugin Recipe component
        • Writing a custom recipe to remove outliers from a dataset
      • Sample dataset
        • Creating a sample dataset
      • Webapps
        • Turning an existing webapp into a webapp plugin component
    • Webapps
      • Common subjects on webapps
        • Accessible resources from webapps
        • Creating an API endpoint from webapps
        • Querying the LLM from an headless API
        • Accessing and understanding logs for WebApps
        • Impersonation with webapps
      • Bokeh
        • Bokeh: Your first webapp
      • Dash
        • Create an empty dash webapp
        • Dash: your first webapp
        • Display a Bar chart using Dash
        • Create a simple admin project dashboard using Dash.
        • Create a simple multi-page webapp
        • Uploading or downloading files with Managed Folders in Dash
        • How to create a form for data input?
        • Simple scoring application
        • Listing and deleting a selection of projects
        • Creating an API endpoint for using an LLM-Based agent
        • Using Dash and LLM Mesh to build a GPT-powered web app assistant
        • Creating a Dash application using an LLM-based agent
      • Gradio
        • Gradio: your first web application
        • Creating a Gradio application using an LLM-based agent
      • Standard (HTML, CSS, JS)
        • HTML/CSS/JS: your first webapp
        • Adapt a D3.js Template in a Webapp
        • Use Custom Static Files (Javascript, CSS) in a Webapp
        • Serve and share your static webapp
        • Uploading to Dataiku in a Webapp
        • Downloading from a Dataiku Webapp
        • How to create a form for data input?
        • Simple scoring application
      • Streamlit
        • Streamlit: your first webapp
        • Streamlit: your first webapp using code studio
      • Use your own framework.
        • Basic setup: Code studio template
        • Basic setup: Quickstart with Angular & Vue Templates
        • Basic setup: Deploy your web application
        • Advanced setup: Code Studio template creation
        • Advanced setup: Web application creation and publication
      • Voilà
        • Voilà: your first web application
        • Creating a Voilà application using an LLM-based agent
    • xOps
      • Project deployment
  • What’s new
    • 2026
      • January 2026
      • February 2026
      • March, 2026
      • April 2026
      • May 2026
  • API Reference
    • Python
      • Index
      • Datasets
      • Managed folders
      • The main API client
      • Interaction with Pyspark
      • Projects
      • Recipes
      • Scenarios
      • Flow creation and management
      • LLM Mesh
        • LLM Mesh Core
        • LLM Mesh Retrieval And Knowledge Banks
        • LLM Mesh Integrations
        • Document Extraction For LLM Workflows
        • Local Models For LLM Mesh
      • Agents
      • Machine learning
      • Experiment Tracking
      • Statistics worksheets
      • Semantic models
      • API Designer
      • API Deployer
      • Project Deployer
      • Static insights
      • Jobs
      • Authentication information and impersonation
      • Importing tables as datasets
      • Wiki
      • Discussions
      • Performing SQL, Hive and Impala queries
      • SQL Query
      • Project libraries
      • Meanings
      • Users and groups
      • Connections
      • Scenarios (in a scenario)
      • Code envs
      • Plugins
      • Project folders
      • Macros
      • Dataiku applications
      • Feature Store
      • Streaming endpoints
      • Snowpark
      • Databricks Connect
      • Bigframes
      • Unified Monitoring
      • Metrics and checks
      • Model Evaluation Stores
      • Other administration tasks
      • Utilities
      • Index of the dataiku package
      • Index of the dataikuapi package
      • API for plugin components
        • API for plugin recipes
        • API for plugin datasets
        • API for plugin formats
        • API for plugin FS providers
      • Clusters
      • Datadir footprint
      • Code Studios
      • Fleet Manager
      • Govern
      • Workspaces
      • Catalog Collections
      • Data Quality
      • Webapps
      • Dashboards
      • Insights
      • Messaging channels
      • Project Standards
      • Enterprise Asset Library
      • Agents Review
      • Cloud
You are viewing the developer guide for version 15 of DSS.
  • Go back to the homepage
  • Concepts And Examples
  • Pyspark Recipes
Back to top

Pyspark recipes¶

For an introduction about pyspark recipes in Dataiku, please see code recipes

Additional documentation¶

dataiku.spark

Next
Projects
Previous
The main API client
Copyright © 2026, Dataiku
Made with Sphinx and @pradyunsg's Furo
Table of contents
  • Pyspark recipes
    • Additional documentation