TextClust¶

textClust, a clustering algorithm for text data.

textClust ¹² is a stream clustering algorithm for textual data that can identify and track topics over time in a stream of texts. The algorithm uses a widely popular two-phase clustering approach where the stream is first summarised in real-time.

The result is many small preliminary clusters in the stream called micro-clusters. Micro-clusters maintain enough information to update and efficiently calculate the cosine similarity between them over time, based on the TF-IDF vector of their texts. Upon request, the miro-clusters can be reclustered to generate the final result using any distance-based clustering algorithm, such as hierarchical clustering. To keep the micro-clusters up-to-date, our algorithm applies a fading strategy where micro-clusters that are not updated regularly lose relevance and are eventually removed.

Parameters¶

• radius

  Type → float

  Default → 0.3

  Distance threshold to merge two micro-clusters. The value is in the range (0, 1].

• fading_factor

  Type → float

  Default → 0.0005

  Non-negative fading factor for micro-clusters.

• tgap

  Type → int

  Default → 100

  Positive number of observations or time units between outlier removal operations.

• term_fading

  Type → bool

  Default → True

  Determines whether individual terms also fade.

• real_time_fading

  Type → bool

  Default → False

  Determines whether fading uses natural time or the number of observations.

• micro_distance

  Type → str

  Default → tfidf_cosine_distance

  Distance metric for micro-clusters. The available metric is tfidf_cosine_distance.

• macro_distance

  Type → str

  Default → tfidf_cosine_distance

  Distance metric for macro-clusters. The available metric is tfidf_cosine_distance.

• num_macro

  Type → int

  Default → 3

  Positive number of macro-clusters to identify during the reclustering phase.

• min_weight

  Type → float

  Default → 0

  Non-negative minimum weight of the micro-clusters used for reclustering.

• auto_r

  Type → bool

  Default → False

  Determines whether TextClust updates radius automatically.

• auto_merge

  Type → bool

  Default → True

  Determines whether TextClust merges close observations.

• sigma

  Type → float

  Default → 1

  Non-negative parameter for automatic threshold adaptation.

Attributes¶

• micro_clusters

  Micro-clusters generated by the algorithm. Micro-clusters are of type textclust.MicroCluster

Examples¶

from river import compose
from river import feature_extraction
from river import metrics
from river import cluster

corpus = [
   {"text":'This is the first document.',"idd":1, "cluster": 1, "cluster":1},
   {"text":'This document is the second document.',"idd":2,"cluster": 1},
   {"text":'And this is super unrelated.',"idd":3,"cluster": 2},
   {"text":'Is this the first document?',"idd":4,"cluster": 1},
   {"text":'This is super unrelated as well',"idd":5,"cluster": 2},
   {"text":'Test text',"idd":6,"cluster": 5}
]

stopwords = [ 'stop', 'the', 'to', 'and', 'a', 'in', 'it', 'is', 'I']

metric = metrics.AdjustedRand()

model = compose.Pipeline(
    feature_extraction.BagOfWords(lowercase=True, ngram_range=(1, 2), stop_words=stopwords),
    cluster.TextClust(real_time_fading=False, fading_factor=0.001, tgap=100, auto_r=True,
    radius=0.9)
)

for x in corpus:
    y_pred = model.predict_one(x["text"])
    y = x["cluster"]
    metric.update(y,y_pred)
    model.learn_one(x["text"])

print(metric)

AdjustedRand: -0.17647058823529413

Methods¶