streamsight.evaluators

Evaluator Builder

The evaluator module in streamsight contains the Builder class which is used to build an evaluator object. Builder allows the programmer to add algorithms, metrics and settings to the evaluator object before the Evaluator object is built.

It is recommended to initialise the Evaluator object with the Builder as it provides the API for the needed configurations. Beyond the API for the configurations, the builder checks for the validity of the configurations and raises exceptions if the configurations are invalid. The programmer can choose to build the Evaluator object without the Builder as described below but might face exceptions if the configurations are invalid.

Note

We use the term Builder here to refer to the class intendede to build the evaluator object, but this is a abstract class. The programmer should use either EvaluatorPipelineBuilder or EvaluatorStreamerBuilder as the builder.

The adding of new algorithms through add_algorithm() and metrics through add_metric() are made such that it can be done through the class type via importing the class or thorough specifying the class name as a string.

Builder([ignore_unknown_user, ...])	Base class for Builder objects.
EvaluatorPipelineBuilder([...])	Builder to facilitate construction of evaluator.
EvaluatorStreamerBuilder([...])	Builder to facilitate construction of evaluator.

Example

Below is a typical example of how to use the Builder to build an Evaluator object. This example also follows from the example in the python notebook.

from streamsight.evaluator import EvaluatorPipelineBuilder

builder = EvaluatorPipelineBuilder(item_user_based="item",
                    ignore_unknown_user=True,
                    ignore_unknown_item=True)
builder.add_setting(setting) # assuming setting is already defined
builder.add_algorithm("ItemKNNIncremental", {"K": 10})
builder.add_metric("PrecisionK")
evaluator = builder.build()

evaluator.run()

Evaluator

The evaluator module in streamsight contains the evaluators which allows the programmer to evaluate the algorithms on a fixed setting. The evaluator also aids the programmer in masking the shape of the dataset and then subsequently computing the metric for each prediction against the ground truth. There are 2 concrete implementation of Evaluator. The programmer can refer to the actual implementation of the Evaluator to understand the internal workings of the evaluator and to view the schematic diagram of the implementation.

EvaluatorBase(metric_entries, setting, metric_k)	Base class for evaluator.
EvaluatorPipeline(algorithm_entries, ...[, ...])	Evaluation via pipeline
EvaluatorStreamer(metric_entries, setting, ...)	Evaluation via streaming through API

Accumulator

The evaluator module in streamsight contains the Accumulator class which is used to accumulate the metrics.

MetricAccumulator()

Utility

The evaluator module in streamsight contains the utility classes which abstract some of the common functionalities used in the evaluator module.

MetricLevelEnum(value[, names, module, ...])
UserItemBaseStatus()	Unknown and known user/item base.
AlgorithmStatusWarning(algo_id, status, phase)