Model Evaluation Stores#

For usage information and examples, see Model Evaluation Stores

There are two main parts related to the handling of model evaluation stores in Dataiku’s Python APIs:

Both set of classes have fairly similar capabilities.

dataiku package API#

class dataiku.ModelEvaluationStore(lookup, project_key=None, ignore_flow=False)#

This is a handle to interact with a model evaluation store.

Note: this class is also available as dataiku.core.model_evaluation_store.ModelEvaluationStore

get_info(sensitive_info=False)#

Gets information about the location and settings of this model evaluation store.

Parameters:

sensitive_info (bool) – flag for sensitive information such as the Model Evaluation Store absolute path (defaults to False)

Return type:

dict

get_path()#

Gets the filesystem path of this model evaluation store.

Return type:

str

get_id()#

Gets the id of this model evaluation store.

Return type:

str

get_name()#

Gets the name of this model evaluation store.

Return type:

str

list_runs()#

Gets the list of runs of this model evaluation store.

Return type:

list of dataiku.core.model_evaluation_store.ModelEvaluation

get_evaluation(evaluation_id)#

Gets a model evaluation from the store based on its id.

Parameters:

evaluation_id (str) – the id of the model evaluation to retrieve

Returns:

a dataiku.core.model_evaluation_store.ModelEvaluation handle on the model evaluation

get_last_metric_values()#

Gets the set of last values of the metrics on this folder.

Returns:

a dataiku.core.ComputedMetrics object

get_metric_history(metric_lookup)#

Gets the set of all values a given metric took on this folder.

Parameters:

metric_lookup – metric name or unique identifier

Return type:

dict

class dataiku.core.model_evaluation_store.ModelEvaluation(store, evaluation_id)#

This is a handle to interact with a model evaluation from a model evaluation store.

set_preparation_steps(steps, requested_output_schema, context_project_key=None)#

Sets the preparation steps of the input dataset in a model evaluation.

Parameters:

  • steps (dict) – the steps of the preparation

  • requested_output_schema (dict) – the schema of the prepared input dataset as a list of objects like this one: { 'type': 'string', 'name': 'foo', 'maxLength': 1000 }

  • context_project_key (str) – a different project key to use instead of the current project key, because the preparation steps can live in a different project than the dataset (defaults to None)

get_schema()#

Gets the schema of the sample used for this model evaluation. There is more information for the map, array and object types.

Return type:

list of dict

Returns:

a schema as a list of objects like this one: { 'type': 'string', 'name': 'foo', 'maxLength': 1000 }

get_dataframe(columns=None, infer_with_pandas=True, parse_dates=True, bool_as_str=False, float_precision=None)#

Reads the sample in the run as a Pandas dataframe.

Pandas dataframes are fully in-memory, so you need to make sure that your dataset will fit in RAM before using this.

Inconsistent sampling parameter raise ValueError.

Note about encoding:

  • Column labels are “unicode” objects

  • When a column is of string type, the content is made of utf-8 encoded “str” objects

Parameters:

  • columns (list of dict) – the columns with information on type, names, etc. e.g. { 'type': 'string', 'name': 'foo', 'maxLength': 1000 } (defaults to None)

  • infer_with_pandas (bool) – uses the types detected by pandas rather than the types from the dataset schema as detected in DSS (defaults to True)

  • parse_dates (bool) – parses date columns in DSS’s dataset schema (defaults to True)

  • bool_as_str (bool) – leaves boolean values as strings (defaults to False)

  • float_precision (str) – float precision for pandas read_table (defaults to None)

Returns:

a pandas.Dataframe representing the sample used in the evaluation

iter_dataframes_forced_types(names, dtypes, parse_date_columns, sampling=None, chunksize=10000, float_precision=None)#

Reads the model evaluation sample as Pandas dataframes by chunks of fixed size with forced types.

Returns a generator over pandas dataframes.

Useful is the sample doesn’t fit in RAM.

Parameters:

  • names (list of str) – names of the columns of the dataset

  • dtypes (list of str or object) – data types of the columns

  • parse_date_columns (bool) – parses date columns in DSS’s dataset schema

  • sampling – ignored at the moment (defaults to None)

  • chunksize (int) – the size of the dataframes yielded by the iterator (defaults to 10000)

  • float_precision (str) – float precision for pandas read_table (defaults to None)

Returns:

a generator of pandas.Dataframe

iter_dataframes(chunksize=10000, infer_with_pandas=True, parse_dates=True, columns=None, bool_as_str=False, float_precision=None)#

Read the model evaluation sample to Pandas dataframes by chunks of fixed size.

Returns a generator over pandas dataframes.

Useful is the sample doesn’t fit in RAM.

Parameters:

  • chunksize (int) – the size of the dataframes yielded by the iterator (defaults to 10000)

  • infer_with_pandas (bool) – uses the types detected by pandas rather than the dataset schema as detected in DSS (defaults to True)

  • parse_dates (bool) – parses date columns in DSS’s dataset schema (defaults to True)

  • columns (list of dict) – columns of the dataset as dict with names and dtypes (defaults to None)

  • bool_as_str (bool) – leaves boolean values as strings (defaults to False)

  • float_precision (str) – float precision for pandas read_table. For more information on this parameter, please check pandas documentation: https://pandas.pydata.org/docs/reference/api/pandas.read_table.html (defaults to None)

Returns:

a generator of pandas.Dataframe

dataikuapi package API#

class dataikuapi.dss.modelevaluationstore.DSSModelEvaluationStore(client, project_key, mes_id)#

A handle to interact with a model evaluation store on the DSS instance.

Warning

Do not create this directly, use dataikuapi.dss.project.DSSProject.get_model_evaluation_store()

property mes_id#
get_settings()#

Returns the settings of this model evaluation store.

Return type:

DSSModelEvaluationStoreSettings

list_model_evaluations()#

List the model evaluations in this model evaluation store. The list is sorted by ME creation date.

Returns:

The list of the model evaluations

Return type:

list of dataikuapi.dss.modelevaluationstore.DSSModelEvaluation

get_model_evaluation(evaluation_id)#

Get a handle to interact with a specific model evaluation

Parameters:

evaluation_id (string) – the id of the desired model evaluation

Returns:

A dataikuapi.dss.modelevaluationstore.DSSModelEvaluation model evaluation handle

get_latest_model_evaluation()#

Get a handle to interact with the latest model evaluation computed

Returns:

A dataikuapi.dss.modelevaluationstore.DSSModelEvaluation model evaluation handle if the store is not empty, else None

delete_model_evaluations(evaluations)#
class MetricDefinition(code, value, name=None, description=None)#
class LabelDefinition(key, value)#
add_custom_model_evaluation(metrics, evaluation_id=None, name=None, labels=None, model=None)#

Adds a model evaluation with custom metrics to the model evaluation store.

Parameters:

Code sample:

import dataiku
from dataikuapi.dss.modelevaluationstore import DSSModelEvaluationStore

client=dataiku.api_client()
project=client.get_default_project()
mes=project.get_model_evaluation_store("7vFZWNck")

accuracy = DSSModelEvaluationStore.MetricDefinition("accuracy", 0.95, "Accuracy")
other = DSSModelEvaluationStore.MetricDefinition("other", 42, "Other", "Other metric desc")
label = DSSModelEvaluationStore.LabelDefinition("custom:myLabel", "myValue")

mes.add_custom_model_evaluation([accuracy, other], labels=[label])
mes.run_checks()
class dataikuapi.dss.modelevaluationstore.DSSModelEvaluationStoreSettings(model_evaluation_store, settings)#

A handle on the settings of a model evaluation store

Warning

Do not create this class directly, instead use dataikuapi.dss.modelevaluationstore.DSSModelEvaluationStore.get_settings()

property model_evaluation_store#
class dataikuapi.dss.modelevaluationstore.DSSModelEvaluation(model_evaluation_store, evaluation_id)#

A handle on a model evaluation

Warning

Do not create this class directly, instead use dataikuapi.dss.modelevaluationstore.DSSModelEvaluationStore.get_model_evaluation()

property mes_id#
get_full_info()#

Retrieve the model evaluation with its performance data

Returns:

the model evaluation full info, as a dataikuapi.dss.modelevaluationstore.DSSModelEvaluationFullInfo

class dataikuapi.dss.modelevaluationstore.DSSModelEvaluationFullInfo(model_evaluation, full_info)#

A handle on the full information on a model evaluation.

Includes information such as the full id of the evaluated model, the evaluation params, the performance and drift metrics, if any, etc.

Warning

Do not create this class directly, instead use dataikuapi.dss.modelevaluationstore.DSSModelEvaluation.get_full_info()

property model_evaluation#