Agents Review#

class dataikuapi.dss.agent_review.DSSAgentReviewTrait(data=None, **kwargs)#

Represents the configuration of a trait for an agent review.

property id#: The unique identifier of the trait. :rtype: str

property name#: The name of the trait. :rtype: str

property description#: The description of the trait. :rtype: str

property llm_id#: The ID of the LLM used to compute this trait. :rtype: str

property criteria#: The criteria or prompt used by the LLM to evaluate this trait. :rtype: str

property enabled#: Whether this trait is enabled for the agent review. :rtype: bool

property needs_reference#: Whether this trait requires a reference answer to be computed. :rtype: bool

property needs_expectations#: Whether this trait requires expectations to be computed. :rtype: bool

class dataikuapi.dss.agent_review.DSSAgentReview(dss_client, project_key, data)#

A handle to interact with an agent review on the DSS instance.

Important

Do not instantiate directly, use dataikuapi.dss.project.DSSProject.get_agent_review() instead

property id#: Unique ID of the agent review. :rtype: str

property name#: Name of the agent review. :rtype: str

property owner#: The owner of the agent review. :return:

property creation_timestamp#: Timestamp of creation (epoch millis). :rtype: int

property agent_id#: ID of the associated agent (Saved Model smart ID). :rtype: str

property traits#: Traits of the agent review. :rtype: list of DSSAgentReviewTrait

property helper_llm_id#: Id of the “helper” LLM, used to compute expectations :rtype: str

property nb_executions#: Number of times a test is executed in a run :rtype: int

get_trait(trait_id)#: Get a specific trait by its ID. :param str trait_id: ID of the trait to retrieve. :returns: The requested trait, or None if not found. :rtype: DSSAgentReviewTrait

add_trait(trait)#: Add a trait to this agent review configuration. :param trait: The trait to add. :type trait: DSSAgentReviewTrait or dict

get_raw()#: Get the raw agent review data. :rtype: dict

property agent_version#: Version of the associated agent. :rtype: str

list_tests(as_type='listitems')#

List all tests linked to this agent review.

Parameters:: as_type (str) – Type of objects to return. Can be ‘listitems’ (default) or ‘objects’.
Returns:: List of tests defined for this agent review.
Return type:: if as_type=listitems, each test as a dataikuapi.dss.agent_review.DSSAgentReviewTestListItem. if as_type=objects, each test is returned as a dataikuapi.dss.agent_review.DSSAgentReviewTest.

get_test(test_id)#

Get a specific test by its ID.

Parameters:: test_id (str) – ID of the test to retrieve.
Returns:: The requested test.
Return type:: DSSAgentReviewTest

create_test(query=None, reference_answer=None, expectations=None)#

Create a new test for this agent review.

Parameters:

query (str) – Query to test the agent. Optional.
reference_answer (str) – Reference answer. Optional.
expectations (str) – Expectations on the agent answer. Optional.

Returns:

The created test object.

Return type:

DSSAgentReviewTest

create_tests_from_dataset(full_dataset_name, query_column, reference_answer_column=None, expectations_column=None, top_n=None, partitions=None, latest_partitions_n=None)#

Create new tests for this agent review by importing them from a dataset.

Parameters:

full_dataset_name (str) – Source dataset name.
query_column (str) – Name of the column containing the queries.
reference_answer_column (str) – Name of the column containing the reference answers. Optional.
expectations_column (str) – Name of the column containing the expectations. Optional.
top_n (int) – Only take the first n rows of the dataset. Optional.
partitions (list[str]) – For partitioned datasets, only consider the given partitions. Optional.
latest_partitions_n (int) – For partitioned datasets and if partitions is not set, only consider the latest n partitions. Optional.

Returns:

A dictionary with keys: - “createdTestIds”: list of ids of the created tests - “error”: The error message if any occurred

Return type:

dict

export_tests_to_dataset(full_dataset_name, create_new_dataset=False, target_connection=None, test_ids=None)#

Export tests of this agent review to a dataset.

Parameters:

full_dataset_name (str) – Target dataset name.
create_new_dataset (bool) – set to True to create a new dataset.
target_connection (str) – If creating a new dataset, ID of the connection to use. Optional.
test_ids (list[str]) – IDs of the tests to export. If None or empty, exports everything. Optional.

Returns:

A dictionary with keys: - “exportedTestCount”: count of exported tests - “error”: The error message if any occurred

Return type:

dict

perform_run(test_ids=None, wait=True, run_name=None)#

Execute a run with the specified tests.

Parameters:

test_ids (list[str]) – List of test IDs to run. Optional. If None or empty, all tests will be run.
wait (bool) – If True, the call blocks until the run is finished. If False, it returns a future history handle. Defaults to True.
run_name (str) – Optional name for the run.

Returns:

The run object if wait=True, or a future history handle if wait=False.

Return type:

DSSAgentReviewRun or dataikuapi.dss.future.DSSFutureWithHistory

list_runs(as_type='listitems')#

List all runs of this agent review.

Parameters:: as_type (str) – Type of objects to return. Can be ‘listitems’ (default) or ‘objects’.
Returns:: List of runs.
Return type:: if as_type=listitems, each run as a DSSAgentReviewRunListItem. if as_type=objects, each run is returned as a DSSAgentReviewRun.

get_run(run_id)#

Get a specific run by its ID.

Parameters:: run_id (str) – ID of the run to retrieve.
Returns:: The requested run.
Return type:: DSSAgentReviewRun

save()#: Save the agent review settings. :returns: The updated agent review. :rtype: DSSAgentReview

delete()#: Delete this agent review.

class dataikuapi.dss.agent_review.DSSAgentReviewListItem(dss_client, project_key, data)#

An item in a list of agent reviews.

Important

Do not instantiate this class directly. Instances are returned by dataikuapi.dss.project.DSSProject.list_agent_reviews().

to_agent_review()#

Get a handle to interact with this agent review.

Returns:: A handle on the agent review.
Return type:: DSSAgentReview

property id#: Unique ID of the agent review. :rtype: str

property name#: Name of the agent review. :rtype: str

property agent_id#: ID of the associated agent (Saved Model smart ID). :rtype: str

property agent_version#: Version of the associated agent. :rtype: str

clear() → None. Remove all items from D.#

copy() → a shallow copy of D#

fromkeys(value=None, /)#: Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)#: Return the value for key if key is in the dictionary, else default.

items() → a set-like object providing a view on D's items#

keys() → a set-like object providing a view on D's keys#

pop(k[, d]) → v, remove specified key and return the corresponding value.#: If key is not found, d is returned if given, otherwise KeyError is raised

popitem()#

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)#

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

property tags#

update([E, ]**F) → None. Update D from dict/iterable E and F.#: If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values() → an object providing a view on D's values#

class dataikuapi.dss.agent_review.DSSAgentReviewTest(dss_client, project_key, data)#

Represents a single test in an agent review.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReview.list_tests() or DSSAgentReview.create_test().

property id#: Unique ID of the test. :rtype: str

property agent_review_id#: ID of the associated agent review. :rtype: str

property query#: Test query. :rtype: str

property reference_answer#: Expected result of the query. :rtype: str

property expectations#: Expectations on the agent answer. :rtype: str

property creation_timestamp#: Timestamp of creation (epoch millis). :rtype: int

property created_by#: Login of the user who created the test. :rtype: str

property last_modification_timestamp#: Timestamp of last modification (epoch millis). :rtype: int

property last_modified_by#: Login of the user who modified last. :rtype: str

get_raw()#: Get the raw test data. :rtype: dict

run()#

Execute a run with this single test.

Returns:: The created run object.
Return type:: DSSAgentReviewRun

save()#

Save the test settings.

Returns:: The updated test.
Return type:: DSSAgentReviewTest

delete()#: Delete this test.

class dataikuapi.dss.agent_review.DSSAgentReviewTestListItem(dss_client, project_key, data)#

An item in a list of agent review tests.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReview.list_tests().

to_agent_review_test()#

Get a handle to interact with this agent review test.

Returns:: A handle on the agent review test.
Return type:: DSSAgentReviewTest

delete()#: Delete this test.

property id#: Unique ID of the test. :rtype: str

property query#: Test query. :rtype: str

property reference_answer#: Expected result of the query. :rtype: str

property expectations#: Expectations on the agent answer. :rtype: str

class dataikuapi.dss.agent_review.DSSAgentReviewRun(dss_client, project_key, data)#

Represents a run of an agent review (execution of tests).

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReview.get_run().

property id#: Unique ID of the run. :rtype: str

property name#: Name of the run. :rtype: str

property agent_review_id#: ID of the associated agent review. :rtype: str

property agent_id#: ID of the agent used in this run. :rtype: str

property agent_version#: Version of the agent used in this run. :rtype: str

property status#: Status of the run. :rtype: str

property error_message#: Error message of the run (nullable). :rtype: str

property created_by#: Login of the user who created the result. :rtype: str

get_raw()#: Get the raw run data. :rtype: dict

list_results(as_type='listitems')#

List all results produced by this run.

Parameters:: as_type (str) – Type of objects to return. Can be ‘listitems’ (default) or ‘objects’.
Returns:: List of results.
Return type:: if as_type=listitems, each run as a DSSAgentReviewResultListItem. if as_type=objects, each run is returned as a DSSAgentReviewResult.

get_result(result_id)#

Get a specific result by its ID.

Parameters:: result_id (str) – ID of the result to retrieve.
Returns:: The requested result.
Return type:: DSSAgentReviewResult

abort()#: Abort the run. :returns: The terminated run. :rtype: DSSAgentReviewRun

list_traits()#: Lists traits defined for this run. :return: List of traits. :rtype: list of DSSAgentReviewTrait

rerun(wait=True, run_name=None)#

Execute a new run with the same test selection.

Parameters:

wait (bool) – If True, the call blocks until the run is finished. If False, it returns a future history handle. Defaults to True.
run_name (str) – Optional name for the new run.

Returns:

The new run object if wait=True, or a future history handle if wait=False.

Return type:

DSSAgentReviewRun or dataikuapi.dss.future.DSSFutureWithHistory

rename(new_name)#: Rename the run. :param str new_name: The new name for the run. :returns: The updated run object. :rtype: DSSAgentReviewRun

delete()#: Delete this run.

class dataikuapi.dss.agent_review.DSSAgentReviewRunListItem(dss_client, project_key, data)#

An item in a list of agent review runs.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReview.list_runs().

to_agent_review_run()#

Get a handle to interact with this agent review run.

Returns:: A handle on the agent review run.
Return type:: DSSAgentReviewRun

delete()#: Delete this run.

property id#: Unique ID of the run. :rtype: str

property name#: Name of the run. :rtype: str

property agent_review_id#: ID of the associated agent review. :rtype: str

property agent_id#: ID of the agent used in this run. :rtype: str

property agent_version#: Version of the agent used in this run. :rtype: str

class dataikuapi.dss.agent_review.DSSAgentReviewHumanReview(dss_client, project_key, data)#

Represents a human review (manual evaluation) of a test result.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewResult.create_human_review() or DSSAgentReviewResult.list_human_reviews().

property id#: Unique ID of the human review. :rtype: str

property result_id#: ID of the result this human review refers to. :rtype: str

property comment#: Text comment of the human review. :rtype: str

property like#: Like of the human review (True for Pass, False for Fail). :rtype: bool

property created_by#: Login of the user who created the review. :rtype: str

property last_modification_timestamp#: Timestamp of last modification (epoch millis). :rtype: int

get_raw()#: Get the raw human review data. :rtype: dict

save()#: Save the human review. :returns: The updated human review. :rtype: DSSAgentReviewHumanReview

delete()#: Delete this human review.

class dataikuapi.dss.agent_review.DSSAgentReviewHumanReviewListItem(dss_client, project_key, data)#

An item in a list of result’s human reviews.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewResult.list_human_reviews().

delete()#: Delete this human review.

property id#: Unique ID of the human review. :rtype: str

property result_id#: ID of the result this human review refers to. :rtype: str

property comment#: Text comment of the human review. :rtype: str

property like#: Like of the human review (True for Pass, False for Fail). :rtype: bool

class dataikuapi.dss.agent_review.DSSAgentReviewTraitOverride(dss_client, project_key, data)#

Represents an trait override (manual evaluation) of a test result.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewResult.create_trait_override() or DSSAgentReviewResult.list_trait_overrides().

property id#: Unique ID of the trait override. :rtype: str

property result_id#: ID of the result this trait override refers to. :rtype: str

property like#: Like of the trait override (True for Pass, False for Fail). :rtype: bool

property created_by#: Login of the user who created the trait override. :rtype: str

property creation_timestamp#: Timestamp of creation (epoch millis). :rtype: int

property last_modification_timestamp#: Timestamp of last modification (epoch millis). :rtype: int

property last_modified_by#: Login of the user who modified last. :rtype: str

get_raw()#: Get the raw trait override data. :rtype: dict

save()#: Save the trait override. :returns: The updated trait override. :rtype: DSSAgentReviewTraitOverride

delete()#: Delete this trait override.

class dataikuapi.dss.agent_review.DSSAgentReviewTraitOverrideListItem(dss_client, project_key, data)#

An item in a list of agent review trait overrides.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewResult.list_trait_overrides().

delete()#: Delete this trait override.

property id#: Unique ID of the trait override. :rtype: str

property result_id#: ID of the result this trait override refers to. :rtype: str

property like#: Like/Dislike of the trait override (True for Pass, False for Fail). :rtype: bool

class dataikuapi.dss.agent_review.DSSAgentReviewTraitOutcome(dss_client, data)#

Represents the result of an evaluation of a trait during an agent execution.

property id#: Unique ID of the trait result. :rtype: str

property project_key#: Project key. :rtype: str

property justification#: Justification of the trait result. :rtype: str

property outcome#: Outcome of the trait result. :rtype: bool

property result_id#: ID of the associated result. :rtype: str

property trait_id#: ID of the trait. :rtype: str

get_raw()#: Get the raw trait result data. :rtype: dict

class dataikuapi.dss.agent_review.DSSAgentReviewExecutionResult(dss_client, project_key, data)#

Represents the execution result of an agent review test.

Important

Do not instantiate this class directly. Instances are created internally and exposed through the DSSAgentReviewResult.execution_results attribute.

property id#: Unique ID of the execution result. :rtype: str

property agent_review_id#: ID of the parent agent review. :rtype: str

property run_id#: ID of the run that produced this execution. :rtype: str

property test_id#: Test ID associated with this execution. :rtype: str

property result_id#: Result ID linking back to the review result. :rtype: str

property answer#: Answer produced during this specific execution. :rtype: str

property error#: Error during this specific execution. :rtype: str

property creation_timestamp#: Timestamp of creation (epoch millis). :rtype: int

property trait_outcomes_per_trait_id#

Trait results for this execution, keyed by trait ID.

Return type:: dict[str, DSSAgentReviewTraitOutcome]

get_raw()#: Get the raw data of this list object. :rtype: dict

class dataikuapi.dss.agent_review.DSSAgentReviewResult(dss_client, project_key, data)#

Represents the result of an execution of tests in a run.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewRun.list_results().

property id#: Unique ID of the result. :rtype: str

property test_id#: ID of the associated test. :rtype: str

property agent_review_id#: ID of the associated agent review. :rtype: str

property run_id#: ID of the associated run. :rtype: str

property query#: Query used in the test. :rtype: str

property raw_query#: Raw query (e.g. including system prompt if available). :rtype: str

property reference_answer#: Expected result of the query. :rtype: str

property expectations#: Expectations on the agent answer. :rtype: str

property tool_calls#: Tool calls performed by the agent :rtype: str

property creation_timestamp#: Timestamp of creation (epoch millis). :rtype: int

property agent_id#: ID of the associated agent. :rtype: str

property agent_version#: Version of the associated agent. :rtype: str

property created_by#: Login of the user who created the result. :rtype: str

property created_by_display_name#: Display name of the user who created the result. :rtype: str

property status#: Status of the result. :rtype: str

property human_reviews#: List of human reviews of this result. :rtype: list of DSSAgentReviewHumanReview

property trait_status_per_trait_id#: Status of each trait for this result. :rtype: dict[str, str]

property ai_status_per_trait_id#: AI-computed status of each trait for this result. :rtype: dict[str, str]

property trait_status_justification_per_trait_id#: Justification for the status of each trait. :rtype: dict[str, str]

property trait_overrides#: Trait overrides of human reviewers, grouped by trait ID. :rtype: dict[str, list of DSSAgentReviewTraitOverride]

property execution_results#: Execution results of the agent. :rtype: list of DSSAgentReviewExecutionResult

get_raw()#: Get the raw result data. :rtype: dict

get_trait_override(trait_override_id)#

Get a specific trait override by its ID.

Parameters:: trait_override_id (str) – ID of the trait override to retrieve.
Returns:: The requested trait override.
Return type:: DSSAgentReviewTraitOverride

create_trait_override(trait_id, like)#

Create a trait override for this trait result.

Parameters:

trait_id (str) – ID of the trait to override.
like (bool) – True for like (Pass), False for dislike (Fail).

Returns:

The created trait override.

Return type:

DSSAgentReviewTraitOverride

list_trait_overrides(as_type='listitems')#

List all trait overrides for this trait result.

Parameters:: as_type (str) – Type of objects to return. Can be ‘listitems’ (default) or ‘objects’.
Returns:: List of trait overrides.
Return type:: if as_type=listitems, each trait override is returned as a DSSAgentReviewTraitOverrideListItem. if as_type=objects, each trait override is returned as a DSSAgentReviewTraitOverride.

get_human_review(human_review_id)#

Get a specific human review by its ID.

Parameters:: human_review_id (str) – ID of the human review to retrieve.
Returns:: The requested human review.
Return type:: DSSAgentReviewHumanReview

create_human_review(comment=None, like=None)#

Create a human review for this trait result.

Parameters:

comment (str) – Text comment. Optional.
like (bool) – Like of the review (True for Pass, False for Fail). Optional.

Returns:

The created human review.

Return type:

DSSAgentReviewHumanReview

list_human_reviews(as_type='listitems')#

List all human reviews for this result.

Parameters:: as_type (str) – Type of objects to return. Can be ‘listitems’ (default) or ‘objects’.
Returns:: List of human reviews.
Return type:: if as_type=’listitems’, each human review is returned as a DSSAgentReviewHumanReviewListItem. if as_type=’objects’, each human review is returned as a DSSAgentReviewHumanReview.

class dataikuapi.dss.agent_review.DSSAgentReviewResultListItem(dss_client, project_key, data)#

An item in a list of agent review results.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewRun.list_results().

to_agent_review_result()#

Get a handle to interact with this agent review result.

Returns:: A handle on the agent review result.
Return type:: DSSAgentReviewResult

property id#: Unique ID of the result. :rtype: str

property test_id#: ID of the associated test. :rtype: str

property agent_review_id#: ID of the associated agent review. :rtype: str

property run_id#: ID of the associated run. :rtype: str

property query#: Query used in the test. :rtype: str

property raw_query#: Raw query (e.g. including system prompt if available). :rtype: str

property reference_answer#: Expected result of the query. :rtype: str

property expectations#: Expectations on the agent answer. :rtype: str

property creation_timestamp#: Timestamp of creation (epoch millis). :rtype: int

property agent_id#: ID of the associated agent. :rtype: str

property agent_version#: Version of the associated agent. :rtype: str

property created_by#: Login of the user who created the result. :rtype: str

property created_by_display_name#: Display name of the user who created the result. :rtype: str

property status#: Status of the result. :rtype: str

property trait_status_per_trait_id#: Status of each trait for this result. :rtype: dict[str, str]

property ai_status_per_trait_id#: AI-computed status of each trait for this result. :rtype: dict[str, str]

property trait_status_justification_per_trait_id#: Justification for the status of each trait. :rtype: dict[str, str]

get_raw()#: Get the raw data of this list item. :rtype: dict