Skip to content

CohenKappa

Cohen's Kappa score.

Cohen's Kappa expresses the level of agreement between two annotators on a classification problem. It is defined as

\[ \kappa = (p_o - p_e) / (1 - p_e) \]

where \(p_o\) is the empirical probability of agreement on the label assigned to any sample (prequential accuracy), and \(p_e\) is the expected agreement when both annotators assign labels randomly.

Parameters

  • cm

    Default → None

    This parameter allows sharing the same confusion matrix between multiple metrics. Sharing a confusion matrix reduces the amount of storage and computation time.

Attributes

  • bigger_is_better

    Indicate if a high value is better than a low one or not.

  • requires_labels

    Indicates if labels are required, rather than probabilities.

  • works_with_weights

    Indicate whether the model takes into consideration the effect of sample weights

Examples

from river import metrics

y_true = ['cat', 'ant', 'cat', 'cat', 'ant', 'bird']
y_pred = ['ant', 'ant', 'cat', 'cat', 'ant', 'cat']

metric = metrics.CohenKappa()

for yt, yp in zip(y_true, y_pred):
    metric.update(yt, yp)

metric

CohenKappa: 42.86%

Methods

get

Return the current value of the metric.

is_better_than

Indicate if the current metric is better than another one.

Parameters

  • other

revert

Revert the metric.

Parameters

  • y_true
  • y_pred
  • w — defaults to 1.0

update

Update the metric.

Parameters

  • y_true
  • y_pred
  • w — defaults to 1.0

works_with

Indicates whether or not a metric can work with a given model.

Parameters

  1. J. Cohen (1960). "A coefficient of agreement for nominal scales". Educational and Psychological Measurement 20(1):37-46. doi:10.1177/001316446002000104. ↩