streamsight.evaluators.Builder

class streamsight.evaluators.Builder(ignore_unknown_user: bool = True, ignore_unknown_item: bool = True, seed: int | None = None)

Bases: ABC

Base class for Builder objects.

Provides methods to set specific values for the builder and enforce checks such that the builder can be constructed correctly and to avoid possible errors when the builder is executed.

__init__(ignore_unknown_user: bool = True, ignore_unknown_item: bool = True, seed: int | None = None)

Methods

__init__([ignore_unknown_user, ...])
add_metric(metric)	Add metric to evaluate algorithm on.
add_setting(setting)	Add setting to the evaluator builder.
build()	Build object.
clear_metrics()	Clear all metrics from the builder.
set_metric_K(K)	Set K value for all metrics.

Attributes

metric_entries	Dict of metrics to evaluate algorithm on.
setting	Setting to evaluate the algorithms on
ignore_unknown_user	Ignore unknown user in the evaluation
ignore_unknown_item	Ignore unknown item in the evaluation

_abc_impl = <_abc._abc_data object>

_check_ready()

Check if the builder is ready to construct Evaluator.

Raises:

RuntimeError – If there are invalid configurations

_check_setting_exist()

Check if setting is already set.

Raises:

RuntimeError – If setting has not been set

add_metric(metric: str | type) → None

Add metric to evaluate algorithm on.

Metric will be added to the metric_entries dict where it will later be converted to a list when the evaluator is constructed.

Note

If K is not yet specified, the setting’s top_K value will be used. This requires the setting to be set before adding the metric.

Parameters:

• metric (Union[str, type]) – Metric to evaluate algorithm on

• K (Optional[int], optional) – Top K value to evaluate the prediction on, defaults to None

Raises:

• ValueError – If metric is not found in METRIC_REGISTRY

• RuntimeError – If setting is not set

add_setting(setting: Setting) → None

Add setting to the evaluator builder.

Note

The setting should be set before adding metrics or algorithms to the evaluator.

Parameters:

setting (Setting) – Setting to evaluate the algorithms on

Raises:

ValueError – If setting is not of instance Setting

abstract build()

Build object.

Raises:

NotImplementedError – If the method is not implemented

clear_metrics() → None

Clear all metrics from the builder.

ignore_unknown_item

Ignore unknown item in the evaluation

ignore_unknown_user

Ignore unknown user in the evaluation

metric_entries: Dict[str, MetricEntry]

Dict of metrics to evaluate algorithm on. Using Dict instead of List for fast lookup

set_metric_K(K: int) → None

Set K value for all metrics.

Parameters:

K (int) – K value to set for all metrics

setting: Setting

Setting to evaluate the algorithms on