Agents Review¶

class dataikuapi.dss.agent_review.DSSAgentReviewTrait(data=None, **kwargs)¶

Represents the configuration of a trait for an agent review.

property id¶: The unique identifier of the trait. :rtype: str

property name¶: The name of the trait. :rtype: str

property description¶: The description of the trait. :rtype: str

property llm_id¶: The ID of the LLM used to compute this trait. :rtype: str

property criteria¶: The criteria or prompt used by the LLM to evaluate this trait. :rtype: str

property enabled¶: Whether this trait is enabled for the agent review. :rtype: bool

property needs_reference¶: Whether this trait requires a reference answer to be computed. :rtype: bool

property needs_expectations¶: Whether this trait requires expectations to be computed. :rtype: bool

class dataikuapi.dss.agent_review.DSSAgentReview(dss_client, project_key, data)¶

A handle to interact with an agent review on the DSS instance.

Important

Do not instantiate directly, use dataikuapi.dss.project.DSSProject.get_agent_review() instead

property id¶: Unique ID of the agent review. :rtype: str

property name¶: Name of the agent review. :rtype: str

property owner¶: The owner of the agent review. :return:

property creation_timestamp¶: Timestamp of creation (epoch millis). :rtype: int

property agent_id¶: ID of the associated agent (Saved Model smart ID). :rtype: str

property traits¶: Traits of the agent review. :rtype: list of DSSAgentReviewTrait

property helper_llm_id¶: Id of the “helper” LLM, used to compute expectations :rtype: str

property nb_executions¶: Number of times a test is executed in a run :rtype: int

property hitl_validation_policy¶: Review-wide default HITL validation policy used when creating a test. Possible values are ACCEPT or REJECT. :rtype: str

property hitl_max_validation_turns¶: Maximum number of validation turns auto-answered during a test run before stopping. Possible values are integers greater than or equal to 1. :rtype: int

get_trait(trait_id)¶: Get a specific trait by its ID. :param str trait_id: ID of the trait to retrieve. :returns: The requested trait, or None if not found. :rtype: DSSAgentReviewTrait

add_trait(trait)¶: Add a trait to this agent review configuration. :param trait: The trait to add. :type trait: DSSAgentReviewTrait or dict

get_raw()¶: Get the raw agent review data. :rtype: dict

property agent_version¶: Version of the associated agent. :rtype: str

list_tests(as_type='listitems')¶

List all tests linked to this agent review.

Parameters:: as_type (str) – Type of objects to return. Can be ‘listitems’ (default) or ‘objects’.
Returns:: List of tests defined for this agent review.
Return type:: if as_type=listitems, each test as a dataikuapi.dss.agent_review.DSSAgentReviewTestListItem. if as_type=objects, each test is returned as a dataikuapi.dss.agent_review.DSSAgentReviewTest.

get_test(test_id)¶

Get a specific test by its ID.

Parameters:: test_id (str) – ID of the test to retrieve.
Returns:: The requested test.
Return type:: DSSAgentReviewTest

create_test(query=None, reference_answer=None, expectations=None, hitl_validation_policy=None)¶

Create a new test for this agent review.

Parameters:

query (str) – Query to test the agent. Optional.
reference_answer (str) – Reference answer. Optional.
expectations (str) – Expectations on the agent answer. Optional.
hitl_validation_policy (str) – Per-test HITL validation policy. Optional.

Returns:

The created test object.

Return type:

DSSAgentReviewTest

create_tests_from_dataset(full_dataset_name, query_column, reference_answer_column=None, expectations_column=None, top_n=None, partitions=None, latest_partitions_n=None, hitl_validation_policy_column=None)¶

Create new tests for this agent review by importing them from a dataset.

Parameters:

full_dataset_name (str) – Source dataset name.
query_column (str) – Name of the column containing the queries.
reference_answer_column (str) – Name of the column containing the reference answers. Optional.
expectations_column (str) – Name of the column containing the expectations. Optional.
top_n (int) – Only take the first n rows of the dataset. Optional.
partitions (list[str]) – For partitioned datasets, only consider the given partitions. Optional.
latest_partitions_n (int) – For partitioned datasets and if partitions is not set, only consider the latest n partitions. Optional.
hitl_validation_policy_column (str) – Name of the column containing per-test HITL validation policy. Optional.

Returns:

A dictionary with keys: - “createdTestIds”: list of ids of the created tests - “error”: The error message if any occurred

Return type:

dict

export_tests_to_dataset(full_dataset_name, create_new_dataset=False, target_connection=None, test_ids=None)¶

Export tests of this agent review to a dataset.

Parameters:

full_dataset_name (str) – Target dataset name.
create_new_dataset (bool) – set to True to create a new dataset.
target_connection (str) – If creating a new dataset, ID of the connection to use. Optional.
test_ids (list[str]) – IDs of the tests to export. If None or empty, exports everything. Optional.

Returns:

A dictionary with keys: - “exportedTestCount”: count of exported tests - “error”: The error message if any occurred

Return type:

dict

perform_run(test_ids=None, wait=True, run_name=None)¶

Execute a run with the specified tests.

Parameters:

test_ids (list[str]) – List of test IDs to run. Optional. If None or empty, all tests will be run.
wait (bool) – If True, the call blocks until the run is finished. If False, it returns a future history handle. Defaults to True.
run_name (str) – Optional name for the run.

Returns:

The run object if wait=True, or a future history handle if wait=False.

Return type:

DSSAgentReviewRun or dataikuapi.dss.future.DSSFutureWithHistory

list_runs(as_type='listitems')¶

List all runs of this agent review.

Parameters:: as_type (str) – Type of objects to return. Can be ‘listitems’ (default) or ‘objects’.
Returns:: List of runs.
Return type:: if as_type=listitems, each run as a DSSAgentReviewRunListItem. if as_type=objects, each run is returned as a DSSAgentReviewRun.

get_run(run_id)¶

Get a specific run by its ID.

Parameters:: run_id (str) – ID of the run to retrieve.
Returns:: The requested run.
Return type:: DSSAgentReviewRun

save()¶: Save the agent review settings. :returns: The updated agent review. :rtype: DSSAgentReview

delete()¶: Delete this agent review.

class dataikuapi.dss.agent_review.DSSAgentReviewListItem(dss_client, project_key, data)¶

An item in a list of agent reviews.

Important

Do not instantiate this class directly. Instances are returned by dataikuapi.dss.project.DSSProject.list_agent_reviews().

to_agent_review()¶

Get a handle to interact with this agent review.

Returns:: A handle on the agent review.
Return type:: DSSAgentReview

property id¶: Unique ID of the agent review. :rtype: str

property name¶: Name of the agent review. :rtype: str

property agent_id¶: ID of the associated agent (Saved Model smart ID). :rtype: str

property agent_version¶: Version of the associated agent. :rtype: str

clear() → None. Remove all items from D.¶

copy() → a shallow copy of D¶

classmethod fromkeys(iterable, value=None, /)¶: Create a new dictionary with keys from iterable and values set to value.

get(key, default=None, /)¶: Return the value for key if key is in the dictionary, else default.

items() → a set-like object providing a view on D's items¶

keys() → a set-like object providing a view on D's keys¶

pop(k[, d]) → v, remove specified key and return the corresponding value.¶: If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()¶

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault(key, default=None, /)¶

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

property tags¶

update([E, ]**F) → None. Update D from mapping/iterable E and F.¶: If E is present and has a .keys() method, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values() → an object providing a view on D's values¶

class dataikuapi.dss.agent_review.DSSAgentReviewTest(dss_client, project_key, data)¶

Represents a single test in an agent review.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReview.list_tests() or DSSAgentReview.create_test().

property id¶: Unique ID of the test. :rtype: str

property agent_review_id¶: ID of the associated agent review. :rtype: str

property query¶: Test query. :rtype: str

property reference_answer¶: Expected result of the query. :rtype: str

property expectations¶: Expectations on the agent answer. :rtype: str

property hitl_validation_policy¶: Per-test HITL validation policy. Possible values are ACCEPT or REJECT. :rtype: str

property creation_timestamp¶: Timestamp of creation (epoch millis). :rtype: int

property created_by¶: Login of the user who created the test. :rtype: str

property last_modification_timestamp¶: Timestamp of last modification (epoch millis). :rtype: int

property last_modified_by¶: Login of the user who modified last. :rtype: str

get_raw()¶: Get the raw test data. :rtype: dict

run()¶

Execute a run with this single test.

Returns:: The created run object.
Return type:: DSSAgentReviewRun

save()¶

Save the test settings.

Returns:: The updated test.
Return type:: DSSAgentReviewTest

delete()¶: Delete this test.

class dataikuapi.dss.agent_review.DSSAgentReviewTestListItem(dss_client, project_key, data)¶

An item in a list of agent review tests.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReview.list_tests().

to_agent_review_test()¶

Get a handle to interact with this agent review test.

Returns:: A handle on the agent review test.
Return type:: DSSAgentReviewTest

delete()¶: Delete this test.

property id¶: Unique ID of the test. :rtype: str

property query¶: Test query. :rtype: str

property reference_answer¶: Expected result of the query. :rtype: str

property expectations¶: Expectations on the agent answer. :rtype: str

property hitl_validation_policy¶: Per-test HITL validation policy. Possible values are ACCEPT or REJECT. :rtype: str

class dataikuapi.dss.agent_review.DSSAgentReviewRun(dss_client, project_key, data)¶

Represents a run of an agent review (execution of tests).

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReview.get_run().

property id¶: Unique ID of the run. :rtype: str

property name¶: Name of the run. :rtype: str

property agent_review_id¶: ID of the associated agent review. :rtype: str

property agent_id¶: ID of the agent used in this run. :rtype: str

property agent_version¶: Version of the agent used in this run. :rtype: str

property status¶: Status of the run. :rtype: str

property error_message¶: Error message of the run (nullable). :rtype: str

property created_by¶: Login of the user who created the result. :rtype: str

get_raw()¶: Get the raw run data. :rtype: dict

list_results(as_type='listitems')¶

List all results produced by this run.

Parameters:: as_type (str) – Type of objects to return. Can be ‘listitems’ (default) or ‘objects’.
Returns:: List of results.
Return type:: if as_type=listitems, each run as a DSSAgentReviewResultListItem. if as_type=objects, each run is returned as a DSSAgentReviewResult.

get_result(result_id)¶

Get a specific result by its ID.

Parameters:: result_id (str) – ID of the result to retrieve.
Returns:: The requested result.
Return type:: DSSAgentReviewResult

abort()¶: Abort the run. :returns: The terminated run. :rtype: DSSAgentReviewRun

list_traits()¶: Lists traits defined for this run. :return: List of traits. :rtype: list of DSSAgentReviewTrait

rerun(wait=True, run_name=None)¶

Execute a new run with the same test selection.

Parameters:

wait (bool) – If True, the call blocks until the run is finished. If False, it returns a future history handle. Defaults to True.
run_name (str) – Optional name for the new run.

Returns:

The new run object if wait=True, or a future history handle if wait=False.

Return type:

DSSAgentReviewRun or dataikuapi.dss.future.DSSFutureWithHistory

rename(new_name)¶: Rename the run. :param str new_name: The new name for the run. :returns: The updated run object. :rtype: DSSAgentReviewRun

delete()¶: Delete this run.

class dataikuapi.dss.agent_review.DSSAgentReviewRunListItem(dss_client, project_key, data)¶

An item in a list of agent review runs.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReview.list_runs().

to_agent_review_run()¶

Get a handle to interact with this agent review run.

Returns:: A handle on the agent review run.
Return type:: DSSAgentReviewRun

delete()¶: Delete this run.

property id¶: Unique ID of the run. :rtype: str

property name¶: Name of the run. :rtype: str

property agent_review_id¶: ID of the associated agent review. :rtype: str

property agent_id¶: ID of the agent used in this run. :rtype: str

property agent_version¶: Version of the agent used in this run. :rtype: str

class dataikuapi.dss.agent_review.DSSAgentReviewHumanReview(dss_client, project_key, data)¶

Represents a human review (manual evaluation) of a test result.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewResult.create_human_review() or DSSAgentReviewResult.list_human_reviews().

property id¶: Unique ID of the human review. :rtype: str

property result_id¶: ID of the result this human review refers to. :rtype: str

property comment¶: Text comment of the human review. :rtype: str

property like¶: Like of the human review (True for Pass, False for Fail). :rtype: bool

property created_by¶: Login of the user who created the review. :rtype: str

property last_modification_timestamp¶: Timestamp of last modification (epoch millis). :rtype: int

get_raw()¶: Get the raw human review data. :rtype: dict

save()¶: Save the human review. :returns: The updated human review. :rtype: DSSAgentReviewHumanReview

delete()¶: Delete this human review.

class dataikuapi.dss.agent_review.DSSAgentReviewHumanReviewListItem(dss_client, project_key, data)¶

An item in a list of result’s human reviews.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewResult.list_human_reviews().

delete()¶: Delete this human review.

property id¶: Unique ID of the human review. :rtype: str

property result_id¶: ID of the result this human review refers to. :rtype: str

property comment¶: Text comment of the human review. :rtype: str

property like¶: Like of the human review (True for Pass, False for Fail). :rtype: bool

class dataikuapi.dss.agent_review.DSSAgentReviewTraitOverride(dss_client, project_key, data)¶

Represents an trait override (manual evaluation) of a test result.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewResult.create_trait_override() or DSSAgentReviewResult.list_trait_overrides().

property id¶: Unique ID of the trait override. :rtype: str

property result_id¶: ID of the result this trait override refers to. :rtype: str

property like¶: Like of the trait override (True for Pass, False for Fail). :rtype: bool

property created_by¶: Login of the user who created the trait override. :rtype: str

property creation_timestamp¶: Timestamp of creation (epoch millis). :rtype: int

property last_modification_timestamp¶: Timestamp of last modification (epoch millis). :rtype: int

property last_modified_by¶: Login of the user who modified last. :rtype: str

get_raw()¶: Get the raw trait override data. :rtype: dict

save()¶: Save the trait override. :returns: The updated trait override. :rtype: DSSAgentReviewTraitOverride

delete()¶: Delete this trait override.

class dataikuapi.dss.agent_review.DSSAgentReviewTraitOverrideListItem(dss_client, project_key, data)¶

An item in a list of agent review trait overrides.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewResult.list_trait_overrides().

delete()¶: Delete this trait override.

property id¶: Unique ID of the trait override. :rtype: str

property result_id¶: ID of the result this trait override refers to. :rtype: str

property like¶: Like/Dislike of the trait override (True for Pass, False for Fail). :rtype: bool

class dataikuapi.dss.agent_review.DSSAgentReviewTraitOutcome(dss_client, data)¶

Represents the result of an evaluation of a trait during an agent execution.

property id¶: Unique ID of the trait result. :rtype: str

property project_key¶: Project key. :rtype: str

property justification¶: Justification of the trait result. :rtype: str

property outcome¶: Outcome of the trait result. :rtype: bool

property result_id¶: ID of the associated result. :rtype: str

property trait_id¶: ID of the trait. :rtype: str

get_raw()¶: Get the raw trait result data. :rtype: dict

class dataikuapi.dss.agent_review.DSSAgentReviewExecutionResult(dss_client, project_key, data)¶

Represents the execution result of an agent review test.

Important

Do not instantiate this class directly. Instances are created internally and exposed through the DSSAgentReviewResult.execution_results attribute.

property id¶: Unique ID of the execution result. :rtype: str

property agent_review_id¶: ID of the parent agent review. :rtype: str

property run_id¶: ID of the run that produced this execution. :rtype: str

property test_id¶: Test ID associated with this execution. :rtype: str

property result_id¶: Result ID linking back to the review result. :rtype: str

property answer¶: Answer produced during this specific execution. :rtype: str

property error¶: Error during this specific execution. :rtype: str

property creation_timestamp¶: Timestamp of creation (epoch millis). :rtype: int

property trait_outcomes_per_trait_id¶

Trait results for this execution, keyed by trait ID.

Return type:: dict[str, DSSAgentReviewTraitOutcome]

get_raw()¶: Get the raw data of this list object. :rtype: dict

class dataikuapi.dss.agent_review.DSSAgentReviewResult(dss_client, project_key, data)¶

Represents the result of an execution of tests in a run.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewRun.list_results().

property id¶: Unique ID of the result. :rtype: str

property test_id¶: ID of the associated test. :rtype: str

property agent_review_id¶: ID of the associated agent review. :rtype: str

property run_id¶: ID of the associated run. :rtype: str

property query¶: Query used in the test. :rtype: str

property raw_query¶: Raw query (e.g. including system prompt if available). :rtype: str

property reference_answer¶: Expected result of the query. :rtype: str

property expectations¶: Expectations on the agent answer. :rtype: str

property tool_calls¶: Tool calls performed by the agent :rtype: str

property creation_timestamp¶: Timestamp of creation (epoch millis). :rtype: int

property agent_id¶: ID of the associated agent. :rtype: str

property agent_version¶: Version of the associated agent. :rtype: str

property created_by¶: Login of the user who created the result. :rtype: str

property created_by_display_name¶: Display name of the user who created the result. :rtype: str

property status¶: Status of the result. :rtype: str

property human_reviews¶: List of human reviews of this result. :rtype: list of DSSAgentReviewHumanReview

property trait_status_per_trait_id¶: Status of each trait for this result. :rtype: dict[str, str]

property ai_status_per_trait_id¶: AI-computed status of each trait for this result. :rtype: dict[str, str]

property trait_status_justification_per_trait_id¶: Justification for the status of each trait. :rtype: dict[str, str]

property trait_overrides¶: Trait overrides of human reviewers, grouped by trait ID. :rtype: dict[str, list of DSSAgentReviewTraitOverride]

property execution_results¶: Execution results of the agent. :rtype: list of DSSAgentReviewExecutionResult

get_raw()¶: Get the raw result data. :rtype: dict

get_trait_override(trait_override_id)¶

Get a specific trait override by its ID.

Parameters:: trait_override_id (str) – ID of the trait override to retrieve.
Returns:: The requested trait override.
Return type:: DSSAgentReviewTraitOverride

create_trait_override(trait_id, like)¶

Create a trait override for this trait result.

Parameters:

trait_id (str) – ID of the trait to override.
like (bool) – True for like (Pass), False for dislike (Fail).

Returns:

The created trait override.

Return type:

DSSAgentReviewTraitOverride

list_trait_overrides(as_type='listitems')¶

List all trait overrides for this trait result.

Parameters:: as_type (str) – Type of objects to return. Can be ‘listitems’ (default) or ‘objects’.
Returns:: List of trait overrides.
Return type:: if as_type=listitems, each trait override is returned as a DSSAgentReviewTraitOverrideListItem. if as_type=objects, each trait override is returned as a DSSAgentReviewTraitOverride.

get_human_review(human_review_id)¶

Get a specific human review by its ID.

Parameters:: human_review_id (str) – ID of the human review to retrieve.
Returns:: The requested human review.
Return type:: DSSAgentReviewHumanReview

create_human_review(comment=None, like=None)¶

Create a human review for this trait result.

Parameters:

comment (str) – Text comment. Optional.
like (bool) – Like of the review (True for Pass, False for Fail). Optional.

Returns:

The created human review.

Return type:

DSSAgentReviewHumanReview

list_human_reviews(as_type='listitems')¶

List all human reviews for this result.

Parameters:: as_type (str) – Type of objects to return. Can be ‘listitems’ (default) or ‘objects’.
Returns:: List of human reviews.
Return type:: if as_type=’listitems’, each human review is returned as a DSSAgentReviewHumanReviewListItem. if as_type=’objects’, each human review is returned as a DSSAgentReviewHumanReview.

class dataikuapi.dss.agent_review.DSSAgentReviewResultListItem(dss_client, project_key, data)¶

An item in a list of agent review results.

Important

Do not instantiate this class directly. Instances are returned by DSSAgentReviewRun.list_results().

to_agent_review_result()¶

Get a handle to interact with this agent review result.

Returns:: A handle on the agent review result.
Return type:: DSSAgentReviewResult

property id¶: Unique ID of the result. :rtype: str

property test_id¶: ID of the associated test. :rtype: str

property agent_review_id¶: ID of the associated agent review. :rtype: str

property run_id¶: ID of the associated run. :rtype: str

property query¶: Query used in the test. :rtype: str

property raw_query¶: Raw query (e.g. including system prompt if available). :rtype: str

property reference_answer¶: Expected result of the query. :rtype: str

property expectations¶: Expectations on the agent answer. :rtype: str

property creation_timestamp¶: Timestamp of creation (epoch millis). :rtype: int

property agent_id¶: ID of the associated agent. :rtype: str

property agent_version¶: Version of the associated agent. :rtype: str

property created_by¶: Login of the user who created the result. :rtype: str

property created_by_display_name¶: Display name of the user who created the result. :rtype: str

property status¶: Status of the result. :rtype: str

property trait_status_per_trait_id¶: Status of each trait for this result. :rtype: dict[str, str]

property ai_status_per_trait_id¶: AI-computed status of each trait for this result. :rtype: dict[str, str]

property trait_status_justification_per_trait_id¶: Justification for the status of each trait. :rtype: dict[str, str]

get_raw()¶: Get the raw data of this list item. :rtype: dict