CohenKappa

Cohen's Kappa score.

Cohen's Kappa expresses the level of agreement between two annotators on a classification problem. It is defined as

$$\kappa =(p_o-p_e)/(1-p_e)$$

where $p_o$ is the empirical probability of agreement on the label assigned to any sample (prequential accuracy), and $p_e$ is the expected agreement when both annotators assign labels randomly.

Parameters

• cm (river.metrics.confusion.ConfusionMatrix) – defaults to None

  This parameter allows sharing the same confusion matrix between multiple metrics. Sharing a confusion matrix reduces the amount of storage and computation time.

Attributes

• bigger_is_better

  Indicate if a high value is better than a low one or not.

• requires_labels

  Indicates if labels are required, rather than probabilities.

• works_with_weights

  Indicate whether the model takes into consideration the effect of sample weights

Examples

>>> from river import metrics

>>> y_true = ['cat', 'ant', 'cat', 'cat', 'ant', 'bird']
>>> y_pred = ['ant', 'ant', 'cat', 'cat', 'ant', 'cat']

>>> metric = metrics.CohenKappa()

>>> for yt, yp in zip(y_true, y_pred):
...     metric = metric.update(yt, yp)

>>> metric
CohenKappa: 42.86%

Methods

get

Return the current value of the metric.

is_better_than

revert

Revert the metric.

Parameters

• y_true
• y_pred
• sample_weight – defaults to 1.0

update

Update the metric.

Parameters

• y_true
• y_pred
• sample_weight – defaults to 1.0

works_with

Indicates whether or not a metric can work with a given model.

Parameters

• model (river.base.estimator.Estimator)

References

---

1. J. Cohen (1960). "A coefficient of agreement for nominal scales". Educational and Psychological Measurement 20(1):37-46. doi:10.1177/001316446002000104. ↩