core
MetricAccumulator ¶
Source code in src/recnexteval/evaluators/core/accumulator.py
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 | |
acc = defaultdict(dict) instance-attribute ¶
user_level_metrics property ¶
window_level_metrics property ¶
add(metric, algorithm_name) ¶
Add a metric to the accumulator.
Takes a Metric object and adds it under the algorithm name. If the specified metric already exists for the algorithm, it will be overwritten with the new metric.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
metric | Metric | Metric to store. | required |
algorithm_name | str | Name of the algorithm. | required |
Source code in src/recnexteval/evaluators/core/accumulator.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | |
df_user_level_metric() ¶
Get user-level metrics across all timestamps.
Returns:
| Type | Description |
|---|---|
DataFrame | DataFrame with user-level metric computations. |
Source code in src/recnexteval/evaluators/core/accumulator.py
78 79 80 81 82 83 84 85 86 87 88 89 | |
df_window_level_metric() ¶
Source code in src/recnexteval/evaluators/core/accumulator.py
91 92 93 94 95 96 97 | |
df_macro_level_metric() ¶
Get macro-level metrics across all timestamps.
Returns:
| Type | Description |
|---|---|
DataFrame | DataFrame with macro-level metric computations. |
Source code in src/recnexteval/evaluators/core/accumulator.py
99 100 101 102 103 104 105 106 107 108 109 110 111 112 | |
df_micro_level_metric() ¶
Get micro-level metrics across all timestamps.
Returns:
| Type | Description |
|---|---|
DataFrame | DataFrame with micro-level metric computations. |
Source code in src/recnexteval/evaluators/core/accumulator.py
114 115 116 117 118 119 120 121 122 123 124 125 126 127 | |
df_metric(filter_timestamp=None, filter_algo=None, level=MetricLevelEnum.MACRO) ¶
Get DataFrame representation of metrics.
Returns a DataFrame representation of the metrics. The DataFrame can be filtered based on algorithm name and timestamp.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filter_timestamp | None | int | Timestamp value to filter on. Defaults to None. | None |
filter_algo | None | str | Algorithm name to filter on. Defaults to None. | None |
level | MetricLevelEnum | Level of the metric to compute. Defaults to MetricLevelEnum.MACRO. | MACRO |
Returns:
| Type | Description |
|---|---|
DataFrame | DataFrame representation of the metrics. |
Source code in src/recnexteval/evaluators/core/accumulator.py
129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 | |
EvaluatorBase dataclass ¶
Base class for evaluator.
Provides the common methods and attributes for the evaluator classes. Should there be a need to create a new evaluator, it should inherit from this class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
metric_entries | list[MetricEntry] | List of metric entries to compute. | required |
setting | Setting | Setting object. | required |
metric_k | int | Value of K for the metrics. | required |
ignore_unknown_user | bool | Ignore unknown users, defaults to False. | False |
ignore_unknown_item | bool | Ignore unknown items, defaults to False. | False |
seed | int | Random seed for reproducibility. | 42 |
Source code in src/recnexteval/evaluators/core/base.py
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 | |
metric_entries instance-attribute ¶
setting instance-attribute ¶
metric_k instance-attribute ¶
ignore_unknown_user = False class-attribute instance-attribute ¶
ignore_unknown_item = False class-attribute instance-attribute ¶
seed = 42 class-attribute instance-attribute ¶
user_item_base = field(default_factory=UserItemKnowledgeBase) class-attribute instance-attribute ¶
metric_results(level=MetricLevelEnum.MACRO, only_current_timestamp=False, filter_timestamp=None, filter_algo=None) ¶
Results of the metrics computed.
Computes the metrics of all algorithms based on the level specified and return the results in a pandas DataFrame. The results can be filtered based on the algorithm name and the current timestamp.
Specifics¶
- User level: User level metrics computed across all timestamps.
- Window level: Window level metrics computed across all timestamps. This can be viewed as a macro level metric in the context of a single window, where the scores of each user is averaged within the window.
- Macro level: Macro level metrics computed for entire timeline. This score is computed by averaging the scores of all windows, treating each window equally.
- Micro level: Micro level metrics computed for entire timeline. This score is computed by averaging the scores of all users, treating each user and the timestamp the user is in as unique contribution to the overall score.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
level | MetricLevelEnum | Literal['macro', 'micro', 'window', 'user'] | Level of the metric to compute, defaults to "macro". | MACRO |
only_current_timestamp | None | bool | Filter only the current timestamp, defaults to False. | False |
filter_timestamp | None | int | Timestamp value to filter on, defaults to None. If both | None |
filter_algo | None | str | Algorithm name to filter on, defaults to None. | None |
Returns:
| Type | Description |
|---|---|
DataFrame | Dataframe representation of the metric. |
Source code in src/recnexteval/evaluators/core/base.py
156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 | |
plot_macro_level_metric() ¶
Source code in src/recnexteval/evaluators/core/base.py
210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 | |
plot_micro_level_metric() ¶
Source code in src/recnexteval/evaluators/core/base.py
235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 | |
plot_window_level_metric() ¶
Source code in src/recnexteval/evaluators/core/base.py
260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 | |
restore() ¶
Restore the generators before pickling.
This method is used to restore the generators after loading the object from a pickle file.
Source code in src/recnexteval/evaluators/core/base.py
310 311 312 313 314 315 316 317 | |
current_step() ¶
Return the current step of the evaluator.
Returns:
| Type | Description |
|---|---|
int | Current step of the evaluator. |
Source code in src/recnexteval/evaluators/core/base.py
319 320 321 322 323 324 325 | |
AlgorithmStateEnum ¶
Bases: StrEnum
Enum for the state of the algorithm.
Used to keep track of the state of the algorithm during the streaming process in the EvaluatorStreamer.
Source code in src/recnexteval/evaluators/core/constant.py
4 5 6 7 8 9 10 11 12 13 14 15 | |
MetricLevelEnum ¶
Bases: StrEnum
Source code in src/recnexteval/evaluators/core/constant.py
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | |
MICRO = 'micro' class-attribute instance-attribute ¶
MACRO = 'macro' class-attribute instance-attribute ¶
WINDOW = 'window' class-attribute instance-attribute ¶
USER = 'user' class-attribute instance-attribute ¶
has_value(value) classmethod ¶
Check valid value for MetricLevelEnum.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value | str | String value input. | required |
Returns:
| Type | Description |
|---|---|
bool | Whether the value is valid. |
Source code in src/recnexteval/evaluators/core/constant.py
24 25 26 27 28 29 30 31 32 33 34 | |
AlgorithmStateEntry dataclass ¶
Entry for the algorithm status registry.
This dataclass stores the status of an algorithm for use by AlgorithmStateManager. It contains the algorithm name, unique identifier, current state, associated data segment, and an optional pointer to the algorithm object.
Attributes:
| Name | Type | Description |
|---|---|---|
name | str | Name of the algorithm. |
algorithm_uuid | UUID | Unique identifier for the algorithm. |
algorithm_ptr | Algorithm | Pointer to the algorithm object. |
state | AlgorithmStateEnum | State of the algorithm. |
data_segment | int | Data segment the algorithm is associated with. |
params | dict[str, Any] | Parameters for the algorithm. |
Source code in src/recnexteval/evaluators/core/state_management.py
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | |
AlgorithmStateManager ¶
Source code in src/recnexteval/evaluators/core/state_management.py
41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 | |
values() ¶
Return an iterator over registered AlgorithmStateEntry objects.
Allows iteration over the registered entries.
Returns:
| Type | Description |
|---|---|
Iterator[AlgorithmStateEntry] | An iterator over the registered entries. |
Source code in src/recnexteval/evaluators/core/state_management.py
63 64 65 66 67 68 69 70 71 | |
get(algo_id) ¶
Get the :class:AlgorithmStateEntry for algo_id.
Source code in src/recnexteval/evaluators/core/state_management.py
109 110 111 | |
get_state(algo_id) ¶
Get the current state of the algorithm with algo_id.
Source code in src/recnexteval/evaluators/core/state_management.py
113 114 115 | |
register(algorithm_ptr, name=None, params={}, algo_uuid=None) ¶
Register new algorithm
Source code in src/recnexteval/evaluators/core/state_management.py
117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | |
can_request_training_data(algo_id) ¶
Check if algorithm can request training data
Source code in src/recnexteval/evaluators/core/state_management.py
143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 | |
can_request_unlabeled_data(algo_id) ¶
Check if algorithm can request unlabeled data
Source code in src/recnexteval/evaluators/core/state_management.py
163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 | |
can_submit_prediction(algo_id) ¶
Check if algorithm can submit prediction
Source code in src/recnexteval/evaluators/core/state_management.py
186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 | |
transition(algo_id, new_state, data_segment=None) ¶
Transition algorithm to new state with validation
Source code in src/recnexteval/evaluators/core/state_management.py
206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 | |
is_all_predicted() ¶
Return whether every registered algorithm is in PREDICTED state.
Returns:
| Type | Description |
|---|---|
bool | True if all registered entries have state |
bool |
|
Source code in src/recnexteval/evaluators/core/state_management.py
233 234 235 236 237 238 239 240 241 242 | |
get_all_states() ¶
Get state of all algorithms
Source code in src/recnexteval/evaluators/core/state_management.py
244 245 246 | |
is_all_same_data_segment() ¶
Return whether all registered entries share the same data segment.
Returns:
| Type | Description |
|---|---|
bool | True if there is exactly one distinct data segment across all |
bool | registered entries, False otherwise. |
Source code in src/recnexteval/evaluators/core/state_management.py
248 249 250 251 252 253 254 255 256 257 258 | |
all_algo_states() ¶
Return a mapping of identifier strings to algorithm states.
The identifier used is "{name}_{uuid}" for each registered entry.
Returns:
| Type | Description |
|---|---|
dict[str, AlgorithmStateEnum] | Mapping from identifier string to the entry's |
dict[str, AlgorithmStateEnum] | class: |
Source code in src/recnexteval/evaluators/core/state_management.py
260 261 262 263 264 265 266 267 268 269 270 271 272 | |
set_all_ready(data_segment) ¶
Set all registered algorithms to the READY state.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_segment | int | Data segment to assign to every algorithm. | required |
Source code in src/recnexteval/evaluators/core/state_management.py
274 275 276 277 278 279 280 281 | |
get_algorithm_identifier(algo_id) ¶
Return a stable identifier string for the algorithm.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
algo_id | UUID | UUID of the algorithm. | required |
Returns:
| Type | Description |
|---|---|
str | Identifier in the format "{name}_{uuid}". |
Raises:
| Type | Description |
|---|---|
AttributeError | If |
Source code in src/recnexteval/evaluators/core/state_management.py
283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 | |
UserItemKnowledgeBase dataclass ¶
Unknown and known user/item base.
This class is used to store the status of the user and item base. The class stores the known and unknown user and item set. The class also provides methods to update the known and unknown user and item set.
Source code in src/recnexteval/evaluators/core/user_item_knowledge_base.py
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | |
unknown_user = field(default_factory=set) class-attribute instance-attribute ¶
known_user = field(default_factory=set) class-attribute instance-attribute ¶
unknown_item = field(default_factory=set) class-attribute instance-attribute ¶
known_item = field(default_factory=set) class-attribute instance-attribute ¶
known_shape property ¶
Known number of user id and item id.
id are zero-indexed and the shape returns the max id + 1.
Note
max is used over len as there may be gaps in the id sequence and we are only concerned with the shape of the user-item interaction matrix.
Returns:
| Type | Description |
|---|---|
tuple[int, int] | Tuple of (|user|, |item|). |
global_shape property ¶
Global number of user id and item id.
This is the shape of the user-item interaction matrix considering all the users and items that has been possibly exposed. The global shape considers the fact that an unknown user/item can be exposed during the prediction stage when an unknown user/item id is requested for prediction on the algorithm.
Returns:
| Type | Description |
|---|---|
tuple[int, int] | Tuple of (|user|, |item|). |
update_known_user_item_base(data) ¶
Updates the known user and item set with the data.
Source code in src/recnexteval/evaluators/core/user_item_knowledge_base.py
58 59 60 61 | |
update_unknown_user_item_base(data) ¶
Updates the unknown user and item set with the data.
Source code in src/recnexteval/evaluators/core/user_item_knowledge_base.py
63 64 65 66 | |
reset_unknown_user_item_base() ¶
Clears the unknown user and item set.
Source code in src/recnexteval/evaluators/core/user_item_knowledge_base.py
68 69 70 71 | |