Evaluator module for algorithm evaluation in streaming settings.

This module provides classes and utilities for evaluating recommendation algorithms in streaming environments. It supports both batch pipeline evaluation and interactive streaming evaluation with comprehensive metric computation and result analysis.

Evaluator Builder¶

The evaluator module contains builder classes for constructing evaluator objects. The builders provide a fluent API for configuring evaluators with proper validation and error checking.

For detailed information about the builder classes and usage examples, see the recnexteval.evaluators.builder module.

Available Builders: - Builder: Abstract base class for all builder implementations - EvaluatorPipelineBuilder: Builder for pipeline evaluators - EvaluatorStreamerBuilder: Builder for streaming evaluators

Evaluator Classes¶

The core evaluator classes handle the evaluation of recommendation algorithms on streaming data. The evaluators manage data splitting, algorithm training, prediction generation, and metric computation.

EvaluatorPipeline¶

For batch evaluation of multiple algorithms on static or sliding window settings. This evaluator runs algorithms through a complete pipeline including training, prediction, and evaluation phases.

EvaluatorStreamer¶

For interactive streaming evaluation where algorithms can be registered and evaluated in real-time as data streams in. This allows for more flexible evaluation scenarios where algorithms can request data and submit predictions asynchronously.

Both evaluators inherit from EvaluatorBase which provides common functionality for metric computation, data masking, and result aggregation.

Key Features¶

• Multi-algorithm evaluation: Evaluate multiple algorithms simultaneously
• Streaming support: Handle temporal data streams with sliding windows
• Metric aggregation: Compute metrics at different levels (user, window, macro, micro)
• Data masking: Properly handle unknown users and items during evaluation
• Result analysis: Rich DataFrame outputs for metric analysis and comparison

Basic Usage¶

Pipeline Evaluation:

from recnexteval.evaluators import EvaluatorPipeline
from recnexteval.evaluators.builder import (
    EvaluatorPipelineBuilder,
)
from recnexteval.settings import Setting
from recnexteval.datasets import AmazonMusicDataset

# Load data and create setting
dataset = AmazonMusicDataset()
data = dataset.load()
setting = Setting(data=data, top_K=10)
setting.split()

# Build evaluator
builder = EvaluatorPipelineBuilder()
builder.add_setting(setting)
builder.set_metric_K(10)
builder.add_metric("PrecisionK")
builder.add_algorithm("MostPopular")
evaluator = builder.build()

# Run evaluation
evaluator.run()

# Get results
results = evaluator.metric_results(level="macro")

Streaming Evaluation:

from recnexteval.evaluators import EvaluatorStreamer
from recnexteval.evaluators.builder import (
    EvaluatorStreamerBuilder,
)
from recnexteval.algorithms import MostPopular

# Build streaming evaluator
builder = EvaluatorStreamerBuilder()
builder.add_setting(setting)
builder.set_metric_K(10)
builder.add_metric("HitK")
evaluator = builder.build()

# Start streaming
evaluator.start_stream()

# Register algorithm
algo_id = evaluator.register_algorithm(MostPopular())

# Stream evaluation loop
while True:
    try:
        # Get training data
        training_data = evaluator.get_training_data(algo_id)

        # Get unlabeled data
        unlabeled_data = evaluator.get_unlabeled_data(
            algo_id
        )

        # Algorithm makes predictions
        predictions = algorithm.predict(unlabeled_data)

        # Submit predictions
        evaluator.submit_prediction(algo_id, predictions)

    except EOWSettingError:
        break

Metric Levels¶

Evaluators support computing metrics at different aggregation levels:

• User level: Metrics computed per user across all timestamps
• Window level: Metrics computed per time window, averaging user scores within each window
• Macro level: Overall metrics averaging across all windows equally
• Micro level: Overall metrics averaging across all user-timestamp combinations equally

Available Evaluator Classes: - EvaluatorBase: Base class providing common evaluation functionality - EvaluatorPipeline: Pipeline-based batch evaluator - EvaluatorStreamer: Interactive streaming evaluator

Accumulator¶

The MetricAccumulator class accumulates and stores metric results during evaluation. The accumulator maintains a collection of metric objects organized by algorithm name and provides methods for retrieving results in various formats.

Features: - Storing metrics for multiple algorithms - Computing aggregated results at different levels - Exporting results to pandas DataFrames - Filtering results by algorithm, timestamp, or metric type

Utility Classes¶

Utility classes that support the evaluation process:

• MetricLevelEnum: Enumeration for different metric aggregation levels
• UserItemKnowledgeBase: Tracks known and unknown users/items during evaluation

These utilities handle the complex state management required for streaming evaluation scenarios.