Agent logging#

After following the Creating and using a Code Agent tutorial, you will have a functional agent. From this point, you will need to log your agent’s behavior to understand better and control it. This tutorial shows you how to set up the proper environment and use the Agent logging feature.

Prerequisites#

  • Dataiku >= 14.6

  • Additionnally, you will need the requirements to build a code agent:

    • An OpenAI connection

    • Python >= 3.9

    • A code environment with the following packages:

      langchain_core  # tested with 1.2.29
langchain       # tested with 1.2.15
ddgs            # tested with 9.13.1

    • An SQL Dataset named pro_customers_sql. You can create this file by uploading this CSV file.

Preparing the logging dataset#

To activate and use the Agent logging feature, you will need a dedicated Dataset. You can create a Dataset for your Agent or reuse an existing one. From a Jupyter Notebook, run the following code to create a Dataset with the proper schema:

Create the logging Dataset#
import dataiku
from pprint import pprint as pp

client = dataiku.api_client()
project = client.get_default_project()

logging_dataset = "" # Fill with the name of the logging Dataset
connection_id = ""  # Fill with the connection id

if not project.get_dataset(logging_dataset).exists():
    project.create_llm_interaction_logging_dataset(logging_dataset, connection_id)

Note

You can refer to this code snippet to list the connections you can use.

Warning

When you create an empty Dataset, an error message will appear if you try to visualize the content. This behavior is expected and will disappear as soon as the first logging message is inserted.

Enabling Agent logging#

You can now enable the logging on your agent and set the parameters.

Enabling logging and setting parameters#
CODE_AGENT_ID = "" # Fill with your agent id
agent = project.get_agent(CODE_AGENT_ID)

agent_settings = agent.get_settings()
version = agent_settings.active_version
version_settings = agent_settings.get_version_settings(version)
agent_logging_selection = version_settings.interaction_logging_selection
agent_logging_selection.enable(
    logging_dataset,
    settings={
        "flushEveryS": 60,
        "flushEveryBytes": 1_000_000,
        "contentMode": "FULL",
    },
)

agent_settings.save()

You will need to provide the Dataset name where the agent will store the logging messages. You can also set the parameters that define the logging behavior. The Agent Interaction Logging documentation describes the parameters and their actions.

Using the logging feature#

Since you have created a Dataset and enabled logging, you will now benefit from it without modifying your code. The logging messages are not written synchronously to the dataset. You will have to wait the time defined by the flushEveryS parameter or until the size defined by the flushEveryBytes parameter is reached. Once the flush is complete, refresh your Dataset sample to visualize the messages. As an example, you can run the following code:

Using the Agent#

completion = agent.as_llm().new_completion()
completion.with_message('Give all the professional information you can about the customer with ID: fdouetteau. Also include information about the company if you can.')

resp = completion.execute()
pp(resp.text)

Running the code will produce a record in the Dataset you created. Let’s dive into some of the information you will find.

  • agent_id: the agent Id will be in the form of <your project Id>.<your agent Id in this project>

  • agent_version: the version of your agent when you ran your query. You can track your agent’s behavior evolution across versions.

  • agent_type: the type of agent. The possible values are:

  • user: the user that was used to run the query. It helps you understand potential permissions issues.

  • begin_time, end_time, and duration: temporal metrics to monitor your agent performance and detect abnormal queries that may last too long.

  • status and error_response: this will allow you to spot queries that may have problems. A status with the FAILED value will be completed with information in the error_response field.

  • input_messages: lists all the messages sent in your query in a JSON format. It follows the conventions of ChatMessage

  • response: response from the Agent using the ChatMessage format.

  • dku_trace: potential traces you added in your code. More information in the Adding traces to your Agent tutorial.

Note

Actions performed with the API are reflected in the Dataiku UI. And any modifications made with the help of the UI will behave the same way in future calls to your Agent, whether through code or other visual actions. Once you enable logging and define the parameters you need, all calls to the Agents will benefit from logging messages.

Complete code#

Here is the complete code of the Agent logging tutorial:

Complete code of the Agent logging tutorial
import dataiku
from pprint import pprint as pp

client = dataiku.api_client()
project = client.get_default_project()

logging_dataset = "" # Fill with the name of the logging Dataset
connection_id = ""  # Fill with the connection id

if not project.get_dataset(logging_dataset).exists():
    project.create_llm_interaction_logging_dataset(logging_dataset, connection_id)

CODE_AGENT_ID = "" # Fill with your agent id
agent = project.get_agent(CODE_AGENT_ID)

agent_settings = agent.get_settings()
version = agent_settings.active_version
version_settings = agent_settings.get_version_settings(version)
agent_logging_selection = version_settings.interaction_logging_selection
agent_logging_selection.enable(
    logging_dataset,
    settings={
        "flushEveryS": 60,
        "flushEveryBytes": 1_000_000,
        "contentMode": "FULL",
    },
)

agent_settings.save()

completion = agent.as_llm().new_completion()
completion.with_message('Give all the professional information you can about the customer with ID: fdouetteau. Also include information about the company if you can.')

resp = completion.execute()
pp(resp.text)

Wrapping up#

Congratulations! You’ve learned how to set up and use the Agent logging. You may follow up with the Adding traces to your Agent tutorial to improve your understanding of your agents.

Reference documentation#

Classes#

dataikuapi.dss.agent.DSSAgent(client, ...)

A handle to interact with a DSS-managed agent.

dataikuapi.dss.agent.DSSAgentInteractionLoggingSelection(...)

Selection for agent interaction logging.

dataikuapi.dss.agent.DSSAgentSettings(...)

Settings for a agent

dataikuapi.dss.agent.DSSAgentVersionSettings(...)

dataikuapi.DSSClient(host[, api_key, ...])

Entry point for the DSS API client

dataikuapi.dss.dataset.DSSDataset(client, ...)

A dataset on the DSS instance.

dataikuapi.dss.llm.DSSLLM(client, ...)

A handle to interact with a DSS-managed LLM.

dataikuapi.dss.llm.DSSLLMCompletionQuery(llm)

A handle to interact with a completion query.

dataikuapi.dss.project.DSSProject(client, ...)

A handle to interact with a project on the DSS instance.

Functions#

api_client()

Build an API client for the current DSS instance.

as_llm()

returns:

this agent as a usable dataikuapi.dss.llm.DSSLLM for querying

create_llm_interaction_logging_dataset(...)

Create a new LLM interaction logging dataset.

enable(dataset_name[, settings])

Enable interaction logging on this agent version with explicit settings.

execute()

Run the completion query and retrieve the LLM response.

exists()

Test if the dataset exists.

get_agent(id)

Get a handle to interact with a specific agent

get_dataset(dataset_name)

Get a handle to interact with a specific dataset

get_default_project()

Get a handle to the current default project, if available (i.e.

get_settings()

return:

a handle on the agent's definition

get_version_settings(version_id)

returns:

the settings of the given version of this agent

new_completion()

Create a new completion query.

save()

Saves the settings for this agent

with_message(message[, role])

Add a message to the completion query.