Basic workflow#

This section gives you a coder’s view of how workflows are built and orchestrated in Dataiku. It is inspired by this documentation page and reframes it for developers who build data, analytics, machine learning, and AI workflows with code.

Connecting to your data#

When working on a project, one of the first steps is often to connect to the platforms that host your data, such as data warehouses or object storage buckets. To do so, you should provide the location of those sources and a set of relevant credentials. Handling this manually in code can be cumbersome, repetitive, and error-prone, so Dataiku provides a framework that simplifies access to enterprise data.

In practice, each data source is materialized in Dataiku as a “connection” object that contains information for authenticating and following a path to read and write your data. By doing so, Dataiku enforces a clear separation of responsibilities: while platform administrators create and maintain connections, you can focus on implementing your workflow logic.

Once you have access to the data storage, the next step is to retrieve the specific data you need for your project. For that, the concept of “dataset” provided by Dataiku comes in handy. In short, a dataset acts as a pointer to tabular data sources living on a data storage platform for which a connection has been defined. In practice, when you write code in Dataiku, you manipulate dataset objects that rely on the connection properties to authenticate against the data source and read/write data from/to it. Dataset definitions include the underlying data schema, column names, and types.

To further facilitate data manipulation, dataset objects have been designed to give end users access to data through well-known programmatic interfaces, such as SQL tables and pandas/PySpark DataFrames. This way, the data storage type is abstracted away from the user, who only has to worry about what to do with the data rather than where and how to find it.

In that same spirit, pointers to more generic (i.e., not necessarily tabular) data are also available in Dataiku as “managed folders.” They often handle unstructured data such as images or text documents. This is also relevant for AI workflows. LLM-powered applications and agents often rely on external knowledge sources, documents, or unstructured assets in addition to tabular datasets. In Dataiku, these inputs can be exposed through managed folders, datasets, or other governed project objects, so your code can access them through the same platform abstractions.

See also

Running code#

In your project, your code will likely read data, process it, and then write the results. In other words, your code works with input items and produces output items. In Dataiku, however, you usually do not run tasks directly. Instead, you build output items. This data-driven mindset is similar to standard build tools in the software engineering ecosystem (e.g., make, Maven, or Gradle), where developers focus on the target they want to build rather than manually invoking each step.

In practice, to run code in a Dataiku project, you will write its logic into a recipe, and to execute this code, you will build the recipe’s output items. Recipes can be written in Python, R, or SQL and edited in various environments (see the Tools for coding page).

While datasets are the most common type of buildable item, Dataiku supports many others, including managed folders, ML models, and other workflow objects used in analytics and AI projects. All in all, the entire logic of your project can be articulated by chaining buildable items and linking them with recipes, forming a direct acyclic graph (DAG) called the Flow.

See also

Building a Flow#

Dataiku’s Flow reveals the core value of a data-driven workflow: dependency management. In a Dataiku workflow, the most critical parts are often materialized by the final elements of the workflow’s DAG. This is especially useful when AI workflows depend on multiple upstream steps, such as data preparation, document processing, retrieval, model calls, or post-processing. Instead of manually orchestrating each dependency, you can let the Flow manage how these pieces connect and run together. In Dataiku, you will focus on building the final item of your Flow. Every upstream buildable item is then treated as a dependency, meaning it must also be built if it isn’t already. Dataiku’s dependency resolver solves the recursion problem of determining which dependencies need to be built. Then the scheduler runs all the tasks required for those builds, enforcing concurrency and parallelism when possible.

The user is free to define the level of granularity in their project, which often translates to the number of intermediary states in a Flow. Dataiku also provides tools to structure the logic of a Flow and remove unnecessary intermediate data when needed, so expressiveness does not come at the cost of conciseness.

See also

Scheduling and automating a Flow#

Once your Flow has been developed, one important step towards its production-readiness is to define how its execution can be scheduled and automated. In Dataiku, the most frequent patterns are:

  • Using the native “scenario” feature to define steps to execute, triggers that will launch the execution, and reports to format the outcome of your runs.

  • Using Dataiku’s public API to connect to a third-party scheduler, which can either enforce its own scheduling rules or remotely start a Dataiku scenario.

See also