Skip to content

RollingROCAUC

Rolling version of the Receiving Operating Characteristic Area Under the Curve.

The RollingROCAUC calculates the metric using the instances in its window of size S. It keeps a queue of the instances, when an instance is added and the queue length is equal to S, the last instance is removed. The metric has a tree with ordered instances, in order to calculate the AUC efficiently. It was implemented based on the algorithm presented in Brzezinski and Stefanowski, 2017.

The difference between this metric and the standard ROCAUC is that the latter calculates an approximation of the real metric considering all data from the beginning of the stream, while the RollingROCAUC calculates the exact value considering only the last S instances. This approach may be beneficial if it's necessary to evaluate the model's performance over time, since calculating the metric using the entire stream may hide the current performance of the classifier.

Parameters

  • window_size – defaults to 1000

    The max length of the window.

  • pos_val – defaults to True

    Value to treat as "positive".

Attributes

  • bigger_is_better

    Indicate if a high value is better than a low one or not.

  • requires_labels

    Indicates if labels are required, rather than probabilities.

  • works_with_weights

    Indicate whether the model takes into consideration the effect of sample weights

Examples

>>> from river import metrics

>>> y_true = [ 0,  1,  0,  1,  0,  1,  0,  0,   1,  1]
>>> y_pred = [.3, .5, .5, .7, .1, .3, .1, .4, .35, .8]

>>> metric = metrics.RollingROCAUC(window_size=4)

>>> for yt, yp in zip(y_true, y_pred):
...     metric = metric.update(yt, yp)

>>> metric
RollingROCAUC: 75.00%

Methods

get

Return the current value of the metric.

is_better_than
revert

Revert the metric.

Parameters

  • y_true (bool)
  • y_pred (Union[bool, float, Dict[bool, float]])
update

Update the metric.

Parameters

  • y_true (bool)
  • y_pred (Union[bool, float, Dict[bool, float]])
works_with

Indicates whether or not a metric can work with a given model.

Parameters

  • model (river.base.estimator.Estimator)