OXTRegressor

Online Extra Trees regressor.

The online Extra Trees¹ ensemble takes some steps further into randomization when compared to Adaptive Random Forests (ARF). A subspace of the feature space is considered at each split attempt, as ARF does, and online bagging or subbagging can also be (optionally) used. Nonetheless, Extra Trees randomizes the split candidates evaluated by each leaf node (just a single split is tested by numerical feature, which brings significant speedups to the ensemble), and might also randomize the maximum depth of the forest members, as well as the size of the feature subspace processed by each of its trees' leaves.

On the other hand, OXT suffers from a cold-start problem. As the splits are random, the predictive performance in small samples is usually worse than using a deterministic split approach, such as the one used by ARF.

Parameters

• n_models ('int') – defaults to 10

  The number of trees in the ensemble.

• max_features ('bool | str | int') – defaults to sqrt

  Max number of attributes for each node split.
  - If int, then consider max_features at each split.
  - If float, then max_features is a percentage and int(max_features * n_features) features are considered per split.
  - If "sqrt", then max_features=sqrt(n_features).
  - If "log2", then max_features=log2(n_features).
  - If "random", then max_features will assume a different random number in the interval [2, n_features] for each tree leaf.
  - If None, then max_features=n_features.

• resampling_strategy ('str | None') – defaults to subbagging

  The chosen instance resampling strategy:
  - If None, no resampling will be done and the trees will process all instances. - If 'baggging', online bagging will be performed (sampling with replacement). - If 'subbagging', online subbagging will be performed (sampling without replacement).

• resampling_rate ('int | float') – defaults to 0.5

  Only valid if resampling_strategy is not None. Controls the parameters of the resampling strategy.
  . - If resampling_strategy='bagging', must be an integer greater than or equal to 1 that parameterizes the poisson distribution used to simulate bagging in online learning settings. It acts as the lambda parameter of Oza Bagging and Leveraging Bagging.
  - If resampling_strategy='subbagging', must be a float in the interval \((0, 1]\) that controls the chance of each instance being used by a tree for learning.

• detection_mode ('str') – defaults to all

  The concept drift detection mode in which the forest operates. Valid values are:
  - "all": creates both warning and concept drift detectors. If a warning is detected, an alternate tree starts being trained in the background. If the warning trigger escalates to a concept drift, the affected tree is replaced by the alternate tree.
  - "drop": only the concept drift detectors are created. If a drift is detected, the affected tree is dropped and replaced by a new tree.
  - "off": disables the concept drift adaptation capabilities. The forest will act as if the processed stream is stationary.

• warning_detector ('base.DriftDetector | None') – defaults to None

  The detector that will be used to trigger concept drift warnings. Defaults to drift.ADWIN(0.01).

• drift_detector ('base.DriftDetector | None') – defaults to None

  The detector used to detect concept drifts. Defaults to drift.ADWIN(0.001).

• max_depth ('int | None') – defaults to None

  The maximum depth the ensemble members might reach. If None, the trees will grow indefinitely.

• randomize_tree_depth ('bool') – defaults to False

  Whether or not randomize the maximum depth of each tree in the ensemble. If max_depth is provided, it is going to act as an upper bound to generate the maximum depth for each tree.

• track_metric ('metrics.base.RegressionMetric | None') – defaults to None

  The performance metric used to weight predictions. Defaults to metrics.MAE().

• disable_weighted_vote ('bool') – defaults to True

  Defines whether or not to use predictions weighted by each trees' prediction performance.

• split_buffer_size ('int') – defaults to 5

  Defines the size of the buffer used by the tree splitters when determining the feature range and a random split point in this interval.

• seed ('int | None') – defaults to None

  Random seed to support reproducibility.

• grace_period ('int') – defaults to 50

  [Tree parameter] Number of instances a leaf should observe between split attempts.

• delta ('float') – defaults to 0.01

  [Tree parameter] Allowed error in split decision, a value closer to 0 takes longer to decide.

• tau ('float') – defaults to 0.05

  [Tree parameter] Threshold below which a split will be forced to break ties.

• leaf_prediction ('str') – defaults to adaptive

  [Tree parameter] Prediction mechanism used at leaves.
  - 'mean' - Target mean
  - 'model' - Uses the model defined in leaf_model
  - 'adaptive' - Chooses between 'mean' and 'model' dynamically
  

• leaf_model ('base.Regressor') – defaults to None

  [Tree parameter] The regression model used to provide responses if leaf_prediction='model'. If not provided, an instance of river.linear_model.LinearRegression with the default hyperparameters is used.

• model_selector_decay ('float') – defaults to 0.95

  [Tree parameter] The exponential decaying factor applied to the learning models' squared errors, that are monitored if leaf_prediction='adaptive'. Must be between 0 and 1. The closer to 1, the more importance is going to be given to past observations. On the other hand, if its value approaches 0, the recent observed errors are going to have more influence on the final decision.

• nominal_attributes ('list') – defaults to None

  [Tree parameter] List of Nominal attributes. If empty, then assume that all attributes are numerical.

• min_samples_split ('int') – defaults to 5

  [Tree parameter] The minimum number of samples every branch resulting from a split candidate must have to be considered valid.

• binary_split ('bool') – defaults to False

  [Tree parameter] If True, only allow binary splits.

• max_size ('int') – defaults to 500

  [Tree parameter] Maximum memory (MB) consumed by the tree.

• memory_estimate_period ('int') – defaults to 2000000

  [Tree parameter] Number of instances between memory consumption checks.

• stop_mem_management ('bool') – defaults to False

  [Tree parameter] If True, stop growing as soon as memory limit is hit.

• remove_poor_attrs ('bool') – defaults to False

  [Tree parameter] If True, disable poor attributes to reduce memory usage.

• merit_preprune ('bool') – defaults to True

  [Tree parameter] If True, enable merit-based tree pre-pruning.

Attributes

• instances_per_tree

  The number of instances processed by each one of the current forest members. Each time a concept drift is detected, the count corresponding to the affected tree is reset.

• models

• n_drifts

  The number of concept drifts detected per ensemble member.

• n_tree_swaps

  The number of performed alternate tree swaps. Not applicable if the warning detectors are disabled.

• n_warnings

  The number of warnings detected per ensemble member.

• total_instances

  The total number of instances processed by the ensemble.

Examples

>>> from river import datasets
>>> from river import evaluate
>>> from river import metrics
>>> from river import forest

>>> dataset = datasets.synth.Friedman(seed=42).take(5000)

>>> model = forest.OXTRegressor(n_models=3, seed=42)

>>> metric = metrics.RMSE()

>>> evaluate.progressive_val_score(dataset, model, metric)
RMSE: 3.127311

Methods

append

S.append(value) -- append value to the end of the sequence

Parameters

• item

clear

S.clear() -> None -- remove all items from S

copy

count

S.count(value) -> integer -- return number of occurrences of value

Parameters

• item

extend

S.extend(iterable) -- extend sequence by appending elements from the iterable

Parameters

• other

index

S.index(value, [start, [stop]]) -> integer -- return first index of value. Raises ValueError if the value is not present.

Supporting start and stop arguments is optional, but recommended.

Parameters

• item
• args

insert

S.insert(index, value) -- insert value before index

Parameters

• i
• item

learn_one

pop

S.pop([index]) -> item -- remove and return item at index (default last). Raise IndexError if list is empty or index is out of range.

Parameters

• i – defaults to -1

predict_one

Predict the output of features x.

Parameters

• x ('dict')

Returns

base.typing.RegTarget: The prediction.

remove

S.remove(value) -- remove first occurrence of value. Raise ValueError if the value is not present.

Parameters

• item

reverse

S.reverse() -- reverse IN PLACE

sort

Notes

As the Online Extra Trees change the way in which Hoeffding Trees perform split attempts and monitor numerical input features, some of the parameters of the vanilla Hoeffding Tree algorithms are not available.

References

---

1. Mastelini, S. M., Nakano, F. K., Vens, C., & de Leon Ferreira, A. C. P. (2022). Online Extra Trees Regressor. IEEE Transactions on Neural Networks and Learning Systems. ↩