# TextClust

textClust, a clustering algorithm for text data.

textClust [^/dev/api/cluster/TextClust/#1][^/dev/api/cluster/TextClust/#2] is a stream clustering algorithm for textual data that can identify and track topics over time in a stream of texts. The algorithm uses a widely popular two-phase clustering approach where the stream is first summarised in real-time.

The result is many small preliminary clusters in the stream called `micro-clusters`. Micro-clusters maintain enough information to update and efficiently calculate the cosine similarity between them over time, based on the TF-IDF vector of their texts. Upon request, the miro-clusters can be reclustered to generate the final result using any distance-based clustering algorithm, such as hierarchical clustering. To keep the micro-clusters up-to-date, our algorithm applies a fading strategy where micro-clusters that are not updated regularly lose relevance and are eventually removed.

## Parameters

- **radius**

  *Type* → `float`

  *Default* → `0.3`

  Distance threshold to merge two micro-clusters. The value is in the range `(0, 1]`.
- **fading\_factor**

  *Type* → `float`

  *Default* → `0.0005`

  Non-negative fading factor for micro-clusters.
- **tgap**

  *Type* → `int`

  *Default* → `100`

  Positive number of observations or time units between outlier removal operations.
- **term\_fading**

  *Type* → `bool`

  *Default* → `True`

  Determines whether individual terms also fade.
- **real\_time\_fading**

  *Type* → `bool`

  *Default* → `False`

  Determines whether fading uses natural time or the number of observations.
- **micro\_distance**

  *Type* → `str`

  *Default* → `tfidf_cosine_distance`

  Distance metric for micro-clusters. The available metric is `tfidf_cosine_distance`.
- **macro\_distance**

  *Type* → `str`

  *Default* → `tfidf_cosine_distance`

  Distance metric for macro-clusters. The available metric is `tfidf_cosine_distance`.
- **num\_macro**

  *Type* → `int`

  *Default* → `3`

  Positive number of macro-clusters to identify during the reclustering phase.
- **min\_weight**

  *Type* → `float`

  *Default* → `0`

  Non-negative minimum weight of the micro-clusters used for reclustering.
- **auto\_r**

  *Type* → `bool`

  *Default* → `False`

  Determines whether TextClust updates `radius` automatically.
- **auto\_merge**

  *Type* → `bool`

  *Default* → `True`

  Determines whether TextClust merges close observations.
- **sigma**

  *Type* → `float`

  *Default* → `1`

  Non-negative parameter for automatic threshold adaptation.

## Attributes

- **micro\_clusters**

  Micro-clusters generated by the algorithm. Micro-clusters are of type `textclust.MicroCluster`

## Examples

```python
from river import compose
from river import feature_extraction
from river import metrics
from river import cluster

corpus = [
   {"text":'This is the first document.',"idd":1, "cluster": 1, "cluster":1},
   {"text":'This document is the second document.',"idd":2,"cluster": 1},
   {"text":'And this is super unrelated.',"idd":3,"cluster": 2},
   {"text":'Is this the first document?',"idd":4,"cluster": 1},
   {"text":'This is super unrelated as well',"idd":5,"cluster": 2},
   {"text":'Test text',"idd":6,"cluster": 5}
]

stopwords = [ 'stop', 'the', 'to', 'and', 'a', 'in', 'it', 'is', 'I']

metric = metrics.AdjustedRand()

model = compose.Pipeline(
    feature_extraction.BagOfWords(lowercase=True, ngram_range=(1, 2), stop_words=stopwords),
    cluster.TextClust(real_time_fading=False, fading_factor=0.001, tgap=100, auto_r=True,
    radius=0.9)
)

for x in corpus:
    y_pred = model.predict_one(x["text"])
    y = x["cluster"]
    metric.update(y,y_pred)
    model.learn_one(x["text"])

print(metric)
```

```text
AdjustedRand: -0.17647058823529413
```

## Methods

> [!NOTE]
>
> **Distances**

> [!NOTE]
>
> **MicroCluster**

> [!NOTE]
>
> **TfContainer**

> [!NOTE]
>
> **get\_assignment**

> [!NOTE]
>
> **get\_macroclusters**

> [!NOTE]
>
> **learn\_one**
>
> Update the model with a set of features `x`.
>
> **Parameters**
>
> - **x** — `dict[typing.FeatureName, Any]`
> - **t** — defaults to `None`
> - **w** — defaults to `None`

> [!NOTE]
>
> **predict\_one**
>
> Predicts the cluster number for a set of features `x`.
>
> **Parameters**
>
> - **x** — `dict[typing.FeatureName, Any]`
> - **w** — defaults to `None`
> - **type** — defaults to `micro`
>
> **Returns**
>
> `int`: A cluster number.

> [!NOTE]
>
> **showclusters**

> [!NOTE]
>
> **update\_macro\_clusters**

[^/dev/api/cluster/TextClust/#1]: Assenmacher, D. und Trautmann, H. (2022). Textual One-Pass Stream Clustering with Automated Distance Threshold Adaption. In: Asian Conference on Intelligent Information and Database Systems (Accepted)

[^/dev/api/cluster/TextClust/#2]: Carnein, M., Assenmacher, D., Trautmann, H. (2017). Stream Clustering of Chat Messages with Applications to Twitch Streams. In: Advances in Conceptual Modeling. ER 2017.