# Multi-armed bandits

River has a `bandit` module. It contains several multi-armed bandit policies, bandit environments, and utilities to benchmark policies on bandit problems.

Bandit environments in River implement the [Gym](<https://www.gymlibrary.dev/>) interface. You can thus load them with `gym.make`. Note that Gym is intended for reinforcement learning algorithms, while bandit policies are the simplest form of reinforcement learning. Bandit policies learn by receiving a reward after each step, while reinforcement learning algorithms have to learn from feedback that may arrive at the end of a (long) sequence of steps.

```python
import gymnasium as gym

for k in gym.envs.registry:
    if k.startswith('river_bandits'):
        print(k)
```

River's bandit module offers the [`bandit.evaluate`](<https://riverml.xyz/dev/api/bandit/evaluate>) function to benchmark several policies on a given environment. It takes as input a list of bandit policies, a bandit environment (the problem to solve), and a reward object.

```python
import gymnasium as gym
from river import bandit
import pandas as pd
from tqdm import tqdm
from river import stats

policies=[
    bandit.EpsilonGreedy(epsilon=0.1),
    bandit.EpsilonGreedy(epsilon=0.01),
    bandit.EpsilonGreedy(epsilon=0),
]

env = gym.make(
    'river_bandits/KArmedTestbed-v0',
    max_episode_steps=1000
)

trace = bandit.evaluate(
    policies=policies,
    env=env,
    reward_stat=stats.Mean(),
    n_episodes=(n_episodes := 2000),
)
```

The [`bandit.evaluate`](<https://riverml.xyz/dev/api/bandit/evaluate>) function returns a generator containing the results at each step of the benchmark. This can be wrapped with a `pandas.DataFrame` to gather all the results.

```python
trace_df = pd.DataFrame(tqdm(
    trace, position=0, total=(
        n_episodes *
        len(policies) *
        env._max_episode_steps
    )
))
trace_df.sample(5, random_state=42)
```

```text
0%|          | 0/6000000 [00:00<?, ?it/s]
```

```text
1%|          | 35757/6000000 [00:00<00:16, 357559.26it/s]
```

```text
1%|          | 72819/6000000 [00:00<00:16, 365236.01it/s]
```

```text
2%|▏         | 110733/6000000 [00:00<00:15, 371579.67it/s]
```

```text
2%|▏         | 147891/6000000 [00:00<00:17, 326662.57it/s]
```

```text
3%|▎         | 185413/6000000 [00:00<00:16, 342755.43it/s]
```

```text
4%|▎         | 222290/6000000 [00:00<00:16, 351111.82it/s]
```

```text
4%|▍         | 259892/6000000 [00:00<00:15, 358946.37it/s]
```

```text
5%|▍         | 297001/6000000 [00:00<00:15, 362622.05it/s]
```

```text
6%|▌         | 334941/6000000 [00:00<00:15, 367769.85it/s]
```

```text
6%|▌         | 372323/6000000 [00:01<00:15, 369610.43it/s]
```

```text
7%|▋         | 409933/6000000 [00:01<00:15, 371578.13it/s]
```

```text
7%|▋         | 447328/6000000 [00:01<00:14, 372291.06it/s]
```

```text
8%|▊         | 484617/6000000 [00:01<00:14, 371554.78it/s]
```

```text
9%|▊         | 522080/6000000 [00:01<00:14, 372478.04it/s]
```

```text
9%|▉         | 559358/6000000 [00:01<00:14, 372204.41it/s]
```

```text
10%|▉         | 597110/6000000 [00:01<00:14, 373798.38it/s]
```

```text
11%|█         | 634547/6000000 [00:01<00:14, 373967.61it/s]
```

```text
11%|█         | 672022/6000000 [00:01<00:14, 374200.06it/s]
```

```text
12%|█▏        | 709601/6000000 [00:01<00:14, 374672.91it/s]
```

```text
12%|█▏        | 747074/6000000 [00:02<00:14, 370761.11it/s]
```

```text
13%|█▎        | 785045/6000000 [00:02<00:13, 373424.91it/s]
```

```text
14%|█▎        | 822399/6000000 [00:02<00:13, 372122.15it/s]
```

```text
14%|█▍        | 859692/6000000 [00:02<00:13, 372359.69it/s]
```

```text
15%|█▍        | 896934/6000000 [00:02<00:13, 370586.53it/s]
```

```text
16%|█▌        | 934221/6000000 [00:02<00:13, 371263.88it/s]
```

```text
16%|█▌        | 971559/6000000 [00:02<00:13, 371893.60it/s]
```

```text
17%|█▋        | 1008752/6000000 [00:02<00:13, 371425.78it/s]
```

```text
17%|█▋        | 1045897/6000000 [00:02<00:13, 371350.78it/s]
```

```text
18%|█▊        | 1083034/6000000 [00:02<00:13, 371117.77it/s]
```

```text
19%|█▊        | 1120359/6000000 [00:03<00:13, 371753.82it/s]
```

```text
19%|█▉        | 1158001/6000000 [00:03<00:12, 373080.06it/s]
```

```text
20%|█▉        | 1195310/6000000 [00:03<00:12, 372663.02it/s]
```

```text
21%|██        | 1232737/6000000 [00:03<00:12, 373140.72it/s]
```

```text
21%|██        | 1270052/6000000 [00:03<00:12, 369485.39it/s]
```

```text
22%|██▏       | 1307009/6000000 [00:03<00:12, 369391.91it/s]
```

```text
22%|██▏       | 1344001/6000000 [00:03<00:12, 369161.75it/s]
```

```text
23%|██▎       | 1381000/6000000 [00:03<00:12, 369405.75it/s]
```

```text
24%|██▎       | 1418048/6000000 [00:03<00:12, 369723.65it/s]
```

```text
24%|██▍       | 1455023/6000000 [00:03<00:12, 368530.32it/s]
```

```text
25%|██▍       | 1492402/6000000 [00:04<00:12, 370100.09it/s]
```

```text
25%|██▌       | 1529867/6000000 [00:04<00:12, 371459.06it/s]
```

```text
26%|██▌       | 1567015/6000000 [00:04<00:11, 370982.10it/s]
```

```text
27%|██▋       | 1604457/6000000 [00:04<00:11, 372007.81it/s]
```

```text
27%|██▋       | 1641659/6000000 [00:04<00:11, 371378.51it/s]
```

```text
28%|██▊       | 1678798/6000000 [00:04<00:11, 371276.34it/s]
```

```text
29%|██▊       | 1715950/6000000 [00:04<00:11, 371347.03it/s]
```

```text
29%|██▉       | 1753670/6000000 [00:04<00:11, 373098.39it/s]
```

```text
30%|██▉       | 1791002/6000000 [00:04<00:11, 373149.99it/s]
```

```text
30%|███       | 1828489/6000000 [00:04<00:11, 373661.22it/s]
```

```text
31%|███       | 1866001/6000000 [00:05<00:11, 373855.53it/s]
```

```text
32%|███▏      | 1903498/6000000 [00:05<00:10, 374184.91it/s]
```

```text
32%|███▏      | 1941106/6000000 [00:05<00:10, 374748.40it/s]
```

```text
33%|███▎      | 1978581/6000000 [00:05<00:10, 374080.35it/s]
```

```text
34%|███▎      | 2015990/6000000 [00:05<00:10, 372731.36it/s]
```

```text
34%|███▍      | 2053265/6000000 [00:05<00:10, 365582.52it/s]
```

```text
35%|███▍      | 2090788/6000000 [00:05<00:10, 368428.17it/s]
```

```text
35%|███▌      | 2127656/6000000 [00:05<00:10, 368484.21it/s]
```

```text
36%|███▌      | 2164952/6000000 [00:05<00:10, 369814.01it/s]
```

```text
37%|███▋      | 2202045/6000000 [00:05<00:10, 370144.68it/s]
```

```text
37%|███▋      | 2239400/6000000 [00:06<00:10, 371161.05it/s]
```

```text
38%|███▊      | 2276524/6000000 [00:06<00:10, 368644.68it/s]
```

```text
39%|███▊      | 2313397/6000000 [00:06<00:10, 368395.66it/s]
```

```text
39%|███▉      | 2350573/6000000 [00:06<00:09, 369397.42it/s]
```

```text
40%|███▉      | 2387748/6000000 [00:06<00:09, 370097.28it/s]
```

```text
40%|████      | 2424762/6000000 [00:06<00:09, 369193.32it/s]
```

```text
41%|████      | 2461728/6000000 [00:06<00:09, 369329.51it/s]
```

```text
42%|████▏     | 2498663/6000000 [00:06<00:09, 368993.24it/s]
```

```text
42%|████▏     | 2535658/6000000 [00:06<00:09, 369276.52it/s]
```

```text
43%|████▎     | 2572587/6000000 [00:06<00:09, 368596.09it/s]
```

```text
43%|████▎     | 2609448/6000000 [00:07<00:09, 368525.16it/s]
```

```text
44%|████▍     | 2646625/6000000 [00:07<00:09, 369494.78it/s]
```

```text
45%|████▍     | 2683576/6000000 [00:07<00:08, 369126.35it/s]
```

```text
45%|████▌     | 2720490/6000000 [00:07<00:08, 365618.69it/s]
```

```text
46%|████▌     | 2757060/6000000 [00:07<00:08, 362907.63it/s]
```

```text
47%|████▋     | 2794381/6000000 [00:07<00:08, 365964.03it/s]
```

```text
47%|████▋     | 2831461/6000000 [00:07<00:08, 367399.83it/s]
```

```text
48%|████▊     | 2868865/6000000 [00:07<00:08, 369379.09it/s]
```

```text
48%|████▊     | 2906395/6000000 [00:07<00:08, 371145.44it/s]
```

```text
49%|████▉     | 2943637/6000000 [00:07<00:08, 371523.73it/s]
```

```text
50%|████▉     | 2981552/6000000 [00:08<00:08, 373804.06it/s]
```

```text
50%|█████     | 3018936/6000000 [00:08<00:07, 373452.13it/s]
```

```text
51%|█████     | 3056284/6000000 [00:08<00:07, 372619.98it/s]
```

```text
52%|█████▏    | 3093548/6000000 [00:08<00:07, 372544.76it/s]
```

```text
52%|█████▏    | 3130804/6000000 [00:08<00:07, 372457.97it/s]
```

```text
53%|█████▎    | 3168051/6000000 [00:08<00:07, 372449.18it/s]
```

```text
53%|█████▎    | 3205297/6000000 [00:08<00:07, 370081.12it/s]
```

```text
54%|█████▍    | 3242309/6000000 [00:08<00:07, 364794.40it/s]
```

```text
55%|█████▍    | 3278806/6000000 [00:08<00:07, 358176.48it/s]
```

```text
55%|█████▌    | 3316032/6000000 [00:08<00:07, 362305.53it/s]
```

```text
56%|█████▌    | 3352991/6000000 [00:09<00:07, 364454.09it/s]
```

```text
56%|█████▋    | 3389462/6000000 [00:09<00:07, 362133.20it/s]
```

```text
57%|█████▋    | 3426001/6000000 [00:09<00:07, 362474.24it/s]
```

```text
58%|█████▊    | 3463655/6000000 [00:09<00:06, 366647.55it/s]
```

```text
58%|█████▊    | 3500334/6000000 [00:09<00:06, 365128.09it/s]
```

```text
59%|█████▉    | 3536857/6000000 [00:09<00:06, 363133.85it/s]
```

```text
60%|█████▉    | 3573179/6000000 [00:09<00:06, 359157.16it/s]
```

```text
60%|██████    | 3609624/6000000 [00:09<00:06, 360721.13it/s]
```

```text
61%|██████    | 3646986/6000000 [00:09<00:06, 364548.61it/s]
```

```text
61%|██████▏   | 3684075/6000000 [00:09<00:06, 366436.38it/s]
```

```text
62%|██████▏   | 3721268/6000000 [00:10<00:06, 368072.99it/s]
```

```text
63%|██████▎   | 3758506/6000000 [00:10<00:06, 369358.33it/s]
```

```text
63%|██████▎   | 3795448/6000000 [00:10<00:05, 368421.88it/s]
```

```text
64%|██████▍   | 3832295/6000000 [00:10<00:05, 367435.45it/s]
```

```text
64%|██████▍   | 3869042/6000000 [00:10<00:05, 364214.91it/s]
```

```text
65%|██████▌   | 3906001/6000000 [00:10<00:05, 365127.76it/s]
```

```text
66%|██████▌   | 3943064/6000000 [00:10<00:05, 366764.39it/s]
```

```text
66%|██████▋   | 3980236/6000000 [00:10<00:05, 368241.75it/s]
```

```text
67%|██████▋   | 4017325/6000000 [00:10<00:05, 369029.87it/s]
```

```text
68%|██████▊   | 4054945/6000000 [00:10<00:05, 371174.17it/s]
```

```text
68%|██████▊   | 4092136/6000000 [00:11<00:05, 371389.83it/s]
```

```text
69%|██████▉   | 4129278/6000000 [00:11<00:05, 370600.58it/s]
```

```text
69%|██████▉   | 4166819/6000000 [00:11<00:04, 372038.93it/s]
```

```text
70%|███████   | 4204741/6000000 [00:11<00:04, 374184.87it/s]
```

```text
71%|███████   | 4242161/6000000 [00:11<00:04, 372325.47it/s]
```

```text
71%|███████▏  | 4279413/6000000 [00:11<00:04, 372381.30it/s]
```

```text
72%|███████▏  | 4316654/6000000 [00:11<00:04, 369006.91it/s]
```

```text
73%|███████▎  | 4353563/6000000 [00:11<00:04, 356194.85it/s]
```

```text
73%|███████▎  | 4389275/6000000 [00:11<00:04, 351959.78it/s]
```

```text
74%|███████▎  | 4424536/6000000 [00:12<00:04, 350603.68it/s]
```

```text
74%|███████▍  | 4459639/6000000 [00:12<00:04, 346635.10it/s]
```

```text
75%|███████▍  | 4494334/6000000 [00:12<00:04, 344523.91it/s]
```

```text
75%|███████▌  | 4528806/6000000 [00:12<00:04, 339414.75it/s]
```

```text
76%|███████▌  | 4564323/6000000 [00:12<00:04, 344015.63it/s]
```

```text
77%|███████▋  | 4601512/6000000 [00:12<00:03, 352219.23it/s]
```

```text
77%|███████▋  | 4638394/6000000 [00:12<00:03, 357132.54it/s]
```

```text
78%|███████▊  | 4675353/6000000 [00:12<00:03, 360831.62it/s]
```

```text
79%|███████▊  | 4711481/6000000 [00:12<00:03, 360964.09it/s]
```

```text
79%|███████▉  | 4747595/6000000 [00:12<00:03, 358667.72it/s]
```

```text
80%|███████▉  | 4784547/6000000 [00:13<00:03, 361893.77it/s]
```

```text
80%|████████  | 4821143/6000000 [00:13<00:03, 363106.09it/s]
```

```text
81%|████████  | 4857975/6000000 [00:13<00:03, 364663.38it/s]
```

```text
82%|████████▏ | 4895272/6000000 [00:13<00:03, 367145.68it/s]
```

```text
82%|████████▏ | 4932001/6000000 [00:13<00:02, 367117.45it/s]
```

```text
83%|████████▎ | 4969498/6000000 [00:13<00:02, 369468.59it/s]
```

```text
83%|████████▎ | 5006449/6000000 [00:13<00:02, 368087.42it/s]
```

```text
84%|████████▍ | 5043262/6000000 [00:13<00:02, 368027.85it/s]
```

```text
85%|████████▍ | 5080068/6000000 [00:13<00:02, 366846.15it/s]
```

```text
85%|████████▌ | 5116755/6000000 [00:13<00:02, 352961.50it/s]
```

```text
86%|████████▌ | 5154001/6000000 [00:14<00:02, 358187.64it/s]
```

```text
87%|████████▋ | 5190588/6000000 [00:14<00:02, 360440.03it/s]
```

```text
87%|████████▋ | 5227873/6000000 [00:14<00:02, 364105.01it/s]
```

```text
88%|████████▊ | 5265001/6000000 [00:14<00:02, 366023.59it/s]
```

```text
88%|████████▊ | 5301643/6000000 [00:14<00:01, 365529.05it/s]
```

```text
89%|████████▉ | 5338224/6000000 [00:14<00:01, 364936.47it/s]
```

```text
90%|████████▉ | 5375267/6000000 [00:14<00:01, 366571.56it/s]
```

```text
90%|█████████ | 5412191/6000000 [00:14<00:01, 367368.54it/s]
```

```text
91%|█████████ | 5449178/6000000 [00:14<00:01, 368113.44it/s]
```

```text
91%|█████████▏| 5486294/6000000 [00:14<00:01, 369024.29it/s]
```

```text
92%|█████████▏| 5523689/6000000 [00:15<00:01, 370497.56it/s]
```

```text
93%|█████████▎| 5561261/6000000 [00:15<00:01, 372060.39it/s]
```

```text
93%|█████████▎| 5599087/6000000 [00:15<00:01, 373915.56it/s]
```

```text
94%|█████████▍| 5636481/6000000 [00:15<00:00, 373602.49it/s]
```

```text
95%|█████████▍| 5673901/6000000 [00:15<00:00, 373779.28it/s]
```

```text
95%|█████████▌| 5711281/6000000 [00:15<00:00, 372774.16it/s]
```

```text
96%|█████████▌| 5748560/6000000 [00:15<00:00, 371152.62it/s]
```

```text
96%|█████████▋| 5785876/6000000 [00:15<00:00, 371751.04it/s]
```

```text
97%|█████████▋| 5823053/6000000 [00:15<00:00, 371383.78it/s]
```

```text
98%|█████████▊| 5860550/6000000 [00:15<00:00, 372454.11it/s]
```

```text
98%|█████████▊| 5897877/6000000 [00:16<00:00, 372695.01it/s]
```

```text
99%|█████████▉| 5935148/6000000 [00:16<00:00, 370904.14it/s]
```

```text
100%|█████████▉| 5972723/6000000 [00:16<00:00, 372350.25it/s]
```

```text
100%|██████████| 6000000/6000000 [00:16<00:00, 367420.57it/s]
```

```text

```

|  | episode | step | policy\_idx | arm | reward | reward\_stat |
| --- | --- | --- | --- | --- | --- | --- |
| 1324896 | 441 | 632 | 0 | 5 | 0.235309 | 2.214089 |
| 3566176 | 1188 | 725 | 1 | 0 | 2.311906 | 1.419877 |
| 1109043 | 369 | 681 | 0 | 3 | 2.536967 | 1.663608 |
| 4286042 | 1428 | 680 | 2 | 0 | 0.432460 | 0.559864 |
| 5395174 | 1798 | 391 | 1 | 1 | 0.504419 | 0.965629 |

It is then straightforward to plot the average reward each policy obtains at each step, by averaging over episodes.

```python
policy_names = {
    0: 'ε = 0.1',
    1: 'ε = 0.01',
    2: 'ε = 0 (greedy)'
}

(
    trace_df
    .assign(policy=trace_df.policy_idx.map(policy_names))
    .groupby(['step', 'policy'])
    ['reward'].mean()
    .unstack()
    .plot()
)
```

## Controlling the evaluation loop

The [`bandit.evaluate`](<https://riverml.xyz/dev/api/bandit/evaluate>) function is useful for benchmarking. But in practice, you'll want to have control over your bandit policy. Indeed you'll want the freedom to pull arms (with the `pull` method) and update the policy (with the `update` method) at your discretion.

As an example, the following is a possible reimplementation of the [`bandit.evaluate`](<https://riverml.xyz/dev/api/bandit/evaluate>) function. Here we'll be measuring the rate at which each policy selects the optimal arm.

Note how the `pull` and `update` methods are used.

```python
import copy

policies=[
    bandit.EpsilonGreedy(epsilon=0.1),
    bandit.EpsilonGreedy(epsilon=0.01),
    bandit.EpsilonGreedy(epsilon=0),
]

env = gym.make(
    'river_bandits/KArmedTestbed-v0',
    max_episode_steps=1000
)
n_episodes = 2000

trace = []

with tqdm(total=len(policies) * n_episodes * env._max_episode_steps, position=0) as progress:
    for policy in policies:
        for episode in range(n_episodes):
            episode_policy = policy.clone()
            episode_env = copy.deepcopy(env)
            episode_env.reset()
            step = 0
            while True:
                action = episode_policy.pull(range(episode_env.action_space.n))
                observation, reward, terminated, truncated, info = episode_env.step(action)
                best_action = observation
                episode_policy.update(action, reward)

                trace.append({
                    "episode": episode,
                    "step": step,
                    "policy": f"ε = {policy.epsilon}",
                    "is_action_optimal": action == best_action
                })
                step += 1
                progress.update()

                if terminated or truncated:
                    break

trace_df = pd.DataFrame(trace)
```

```text
0%|          | 0/6000000 [00:00<?, ?it/s]
```

```text
0%|          | 16373/6000000 [00:00<00:36, 163728.59it/s]
```

```text
1%|          | 43778/6000000 [00:00<00:26, 228620.55it/s]
```

```text
1%|▏         | 77027/6000000 [00:00<00:21, 276045.34it/s]
```

```text
2%|▏         | 110001/6000000 [00:00<00:19, 297059.24it/s]
```

```text
2%|▏         | 146256/6000000 [00:00<00:18, 320663.77it/s]
```

```text
3%|▎         | 183098/6000000 [00:00<00:17, 336895.25it/s]
```

```text
4%|▎         | 219407/6000000 [00:00<00:16, 345456.11it/s]
```

```text
4%|▍         | 256012/6000000 [00:00<00:16, 352010.42it/s]
```

```text
5%|▍         | 291214/6000000 [00:00<00:16, 349495.07it/s]
```

```text
5%|▌         | 327848/6000000 [00:01<00:15, 354668.15it/s]
```

```text
6%|▌         | 365254/6000000 [00:01<00:15, 360581.40it/s]
```

```text
7%|▋         | 402950/6000000 [00:01<00:15, 365550.60it/s]
```

```text
7%|▋         | 440824/6000000 [00:01<00:15, 369537.32it/s]
```

```text
8%|▊         | 477783/6000000 [00:01<00:14, 369109.94it/s]
```

```text
9%|▊         | 514698/6000000 [00:01<00:14, 366349.36it/s]
```

```text
9%|▉         | 551340/6000000 [00:01<00:14, 366149.46it/s]
```

```text
10%|▉         | 587999/6000000 [00:01<00:14, 366281.12it/s]
```

```text
10%|█         | 624631/6000000 [00:01<00:14, 364624.45it/s]
```

```text
11%|█         | 661097/6000000 [00:01<00:14, 364563.01it/s]
```

```text
12%|█▏        | 697556/6000000 [00:02<00:14, 363973.85it/s]
```

```text
12%|█▏        | 733956/6000000 [00:02<00:14, 363214.91it/s]
```

```text
13%|█▎        | 770279/6000000 [00:02<00:14, 363084.75it/s]
```

```text
13%|█▎        | 806758/6000000 [00:02<00:14, 363593.09it/s]
```

```text
14%|█▍        | 843577/6000000 [00:02<00:14, 364969.12it/s]
```

```text
15%|█▍        | 880075/6000000 [00:02<00:14, 364124.95it/s]
```

```text
15%|█▌        | 916812/6000000 [00:02<00:13, 365095.72it/s]
```

```text
16%|█▌        | 953323/6000000 [00:02<00:13, 362778.45it/s]
```

```text
16%|█▋        | 989605/6000000 [00:02<00:13, 359780.83it/s]
```

```text
17%|█▋        | 1026078/6000000 [00:02<00:13, 361249.78it/s]
```

```text
18%|█▊        | 1063270/6000000 [00:03<00:13, 364427.52it/s]
```

```text
18%|█▊        | 1100091/6000000 [00:03<00:13, 365556.60it/s]
```

```text
19%|█▉        | 1136652/6000000 [00:03<00:13, 363805.35it/s]
```

```text
20%|█▉        | 1173038/6000000 [00:03<00:13, 358485.06it/s]
```

```text
20%|██        | 1208905/6000000 [00:03<00:13, 357109.83it/s]
```

```text
21%|██        | 1245396/6000000 [00:03<00:13, 359417.61it/s]
```

```text
21%|██▏       | 1281873/6000000 [00:03<00:13, 361005.59it/s]
```

```text
22%|██▏       | 1317984/6000000 [00:03<00:13, 355166.00it/s]
```

```text
23%|██▎       | 1353549/6000000 [00:03<00:13, 355306.75it/s]
```

```text
23%|██▎       | 1390153/6000000 [00:03<00:12, 358489.87it/s]
```

```text
24%|██▍       | 1426018/6000000 [00:04<00:12, 357680.46it/s]
```

```text
24%|██▍       | 1461879/6000000 [00:04<00:12, 357955.14it/s]
```

```text
25%|██▍       | 1498178/6000000 [00:04<00:12, 359455.79it/s]
```

```text
26%|██▌       | 1534239/6000000 [00:04<00:12, 359798.63it/s]
```

```text
26%|██▌       | 1570224/6000000 [00:04<00:12, 358017.35it/s]
```

```text
27%|██▋       | 1606277/6000000 [00:04<00:12, 358765.26it/s]
```

```text
27%|██▋       | 1642669/6000000 [00:04<00:12, 360303.79it/s]
```

```text
28%|██▊       | 1679231/6000000 [00:04<00:11, 361892.32it/s]
```

```text
29%|██▊       | 1716188/6000000 [00:04<00:11, 364189.22it/s]
```

```text
29%|██▉       | 1752610/6000000 [00:04<00:11, 363411.17it/s]
```

```text
30%|██▉       | 1789436/6000000 [00:05<00:11, 364861.13it/s]
```

```text
30%|███       | 1826338/6000000 [00:05<00:11, 366103.65it/s]
```

```text
31%|███       | 1863813/6000000 [00:05<00:11, 368694.06it/s]
```

```text
32%|███▏      | 1900684/6000000 [00:05<00:11, 365240.64it/s]
```

```text
32%|███▏      | 1937826/6000000 [00:05<00:11, 367079.97it/s]
```

```text
33%|███▎      | 1974541/6000000 [00:05<00:10, 366914.80it/s]
```

```text
34%|███▎      | 2011238/6000000 [00:05<00:10, 362867.98it/s]
```

```text
34%|███▍      | 2047537/6000000 [00:05<00:10, 361466.54it/s]
```

```text
35%|███▍      | 2083692/6000000 [00:05<00:10, 360927.45it/s]
```

```text
35%|███▌      | 2119791/6000000 [00:05<00:10, 360506.03it/s]
```

```text
36%|███▌      | 2155846/6000000 [00:06<00:10, 359222.88it/s]
```

```text
37%|███▋      | 2191772/6000000 [00:06<00:10, 358624.09it/s]
```

```text
37%|███▋      | 2227637/6000000 [00:06<00:10, 354643.37it/s]
```

```text
38%|███▊      | 2263111/6000000 [00:06<00:11, 338765.25it/s]
```

```text
38%|███▊      | 2297737/6000000 [00:06<00:10, 340920.36it/s]
```

```text
39%|███▉      | 2332269/6000000 [00:06<00:10, 342201.56it/s]
```

```text
39%|███▉      | 2367117/6000000 [00:06<00:10, 344045.14it/s]
```

```text
40%|████      | 2401581/6000000 [00:06<00:10, 340529.16it/s]
```

```text
41%|████      | 2436130/6000000 [00:06<00:10, 341989.38it/s]
```

```text
41%|████      | 2471108/6000000 [00:06<00:10, 344294.79it/s]
```

```text
42%|████▏     | 2507105/6000000 [00:07<00:10, 348956.60it/s]
```

```text
42%|████▏     | 2543173/6000000 [00:07<00:09, 352451.40it/s]
```

```text
43%|████▎     | 2579173/6000000 [00:07<00:09, 354705.19it/s]
```

```text
44%|████▎     | 2614918/6000000 [00:07<00:09, 355525.26it/s]
```

```text
44%|████▍     | 2650899/6000000 [00:07<00:09, 356807.84it/s]
```

```text
45%|████▍     | 2686753/6000000 [00:07<00:09, 357322.49it/s]
```

```text
45%|████▌     | 2722590/6000000 [00:07<00:09, 357635.24it/s]
```

```text
46%|████▌     | 2758491/6000000 [00:07<00:09, 358046.19it/s]
```

```text
47%|████▋     | 2794299/6000000 [00:07<00:08, 358045.43it/s]
```

```text
47%|████▋     | 2830106/6000000 [00:07<00:08, 354590.48it/s]
```

```text
48%|████▊     | 2865574/6000000 [00:08<00:08, 353728.93it/s]
```

```text
48%|████▊     | 2901365/6000000 [00:08<00:08, 354971.67it/s]
```

```text
49%|████▉     | 2937068/6000000 [00:08<00:08, 355585.10it/s]
```

```text
50%|████▉     | 2973001/6000000 [00:08<00:08, 356680.30it/s]
```

```text
50%|█████     | 3008921/6000000 [00:08<00:08, 357433.51it/s]
```

```text
51%|█████     | 3044667/6000000 [00:08<00:08, 357388.72it/s]
```

```text
51%|█████▏    | 3080568/6000000 [00:08<00:08, 357872.29it/s]
```

```text
52%|█████▏    | 3116442/6000000 [00:08<00:08, 358131.71it/s]
```

```text
53%|█████▎    | 3152257/6000000 [00:08<00:07, 358059.49it/s]
```

```text
53%|█████▎    | 3188064/6000000 [00:08<00:07, 357674.62it/s]
```

```text
54%|█████▎    | 3223907/6000000 [00:09<00:07, 357900.82it/s]
```

```text
54%|█████▍    | 3259781/6000000 [00:09<00:07, 358151.44it/s]
```

```text
55%|█████▍    | 3295961/6000000 [00:09<00:07, 359243.43it/s]
```

```text
56%|█████▌    | 3332005/6000000 [00:09<00:07, 359599.01it/s]
```

```text
56%|█████▌    | 3368095/6000000 [00:09<00:07, 359986.98it/s]
```

```text
57%|█████▋    | 3404351/6000000 [00:09<00:07, 360757.54it/s]
```

```text
57%|█████▋    | 3440427/6000000 [00:09<00:07, 359251.81it/s]
```

```text
58%|█████▊    | 3476354/6000000 [00:09<00:07, 358995.47it/s]
```

```text
59%|█████▊    | 3512356/6000000 [00:09<00:06, 359299.81it/s]
```

```text
59%|█████▉    | 3548287/6000000 [00:09<00:06, 358271.65it/s]
```

```text
60%|█████▉    | 3584116/6000000 [00:10<00:06, 355236.27it/s]
```

```text
60%|██████    | 3619887/6000000 [00:10<00:06, 355971.10it/s]
```

```text
61%|██████    | 3655788/6000000 [00:10<00:06, 356876.56it/s]
```

```text
62%|██████▏   | 3691953/6000000 [00:10<00:06, 358302.61it/s]
```

```text
62%|██████▏   | 3727787/6000000 [00:10<00:06, 357788.23it/s]
```

```text
63%|██████▎   | 3763639/6000000 [00:10<00:06, 358006.18it/s]
```

```text
63%|██████▎   | 3799442/6000000 [00:10<00:06, 357713.42it/s]
```

```text
64%|██████▍   | 3835215/6000000 [00:10<00:06, 355528.98it/s]
```

```text
65%|██████▍   | 3870772/6000000 [00:10<00:06, 350286.91it/s]
```

```text
65%|██████▌   | 3906604/6000000 [00:10<00:05, 352662.00it/s]
```

```text
66%|██████▌   | 3942483/6000000 [00:11<00:05, 354480.90it/s]
```

```text
66%|██████▋   | 3978443/6000000 [00:11<00:05, 356004.55it/s]
```

```text
67%|██████▋   | 4014159/6000000 [00:11<00:05, 356347.28it/s]
```

```text
68%|██████▊   | 4050714/6000000 [00:11<00:05, 359096.23it/s]
```

```text
68%|██████▊   | 4087500/6000000 [00:11<00:05, 361716.94it/s]
```

```text
69%|██████▊   | 4124046/6000000 [00:11<00:05, 362837.76it/s]
```

```text
69%|██████▉   | 4160964/6000000 [00:11<00:05, 364730.42it/s]
```

```text
70%|██████▉   | 4197551/6000000 [00:11<00:04, 365070.80it/s]
```

```text
71%|███████   | 4234080/6000000 [00:11<00:04, 365135.58it/s]
```

```text
71%|███████   | 4270677/6000000 [00:11<00:04, 365385.64it/s]
```

```text
72%|███████▏  | 4307217/6000000 [00:12<00:04, 365061.67it/s]
```

```text
72%|███████▏  | 4343724/6000000 [00:12<00:04, 360710.05it/s]
```

```text
73%|███████▎  | 4380001/6000000 [00:12<00:04, 361232.46it/s]
```

```text
74%|███████▎  | 4416134/6000000 [00:12<00:04, 360915.14it/s]
```

```text
74%|███████▍  | 4452232/6000000 [00:12<00:04, 359601.52it/s]
```

```text
75%|███████▍  | 4488197/6000000 [00:12<00:04, 359263.67it/s]
```

```text
75%|███████▌  | 4525001/6000000 [00:12<00:04, 361871.26it/s]
```

```text
76%|███████▌  | 4562048/6000000 [00:12<00:03, 364439.45it/s]
```

```text
77%|███████▋  | 4599001/6000000 [00:12<00:03, 365873.10it/s]
```

```text
77%|███████▋  | 4635897/6000000 [00:12<00:03, 366796.18it/s]
```

```text
78%|███████▊  | 4672579/6000000 [00:13<00:03, 366490.60it/s]
```

```text
78%|███████▊  | 4709230/6000000 [00:13<00:03, 362981.18it/s]
```

```text
79%|███████▉  | 4745874/6000000 [00:13<00:03, 364009.79it/s]
```

```text
80%|███████▉  | 4783001/6000000 [00:13<00:03, 366011.99it/s]
```

```text
80%|████████  | 4820305/6000000 [00:13<00:03, 368110.43it/s]
```

```text
81%|████████  | 4857499/6000000 [00:13<00:03, 369252.63it/s]
```

```text
82%|████████▏ | 4894428/6000000 [00:13<00:03, 319584.50it/s]
```

```text
82%|████████▏ | 4927571/6000000 [00:13<00:03, 305068.57it/s]
```

```text
83%|████████▎ | 4960451/6000000 [00:13<00:03, 311409.49it/s]
```

```text
83%|████████▎ | 4993752/6000000 [00:14<00:03, 317365.05it/s]
```

```text
84%|████████▍ | 5028361/6000000 [00:14<00:02, 325493.39it/s]
```

```text
84%|████████▍ | 5063102/6000000 [00:14<00:02, 331801.19it/s]
```

```text
85%|████████▍ | 5099398/6000000 [00:14<00:02, 340878.62it/s]
```

```text
86%|████████▌ | 5134443/6000000 [00:14<00:02, 343690.23it/s]
```

```text
86%|████████▌ | 5169001/6000000 [00:14<00:02, 344140.25it/s]
```

```text
87%|████████▋ | 5204877/6000000 [00:14<00:02, 348480.22it/s]
```

```text
87%|████████▋ | 5240804/6000000 [00:14<00:02, 351691.77it/s]
```

```text
88%|████████▊ | 5277399/6000000 [00:14<00:02, 355947.53it/s]
```

```text
89%|████████▊ | 5314001/6000000 [00:14<00:01, 358927.75it/s]
```

```text
89%|████████▉ | 5350988/6000000 [00:15<00:01, 362199.87it/s]
```

```text
90%|████████▉ | 5387808/6000000 [00:15<00:01, 363994.47it/s]
```

```text
90%|█████████ | 5425017/6000000 [00:15<00:01, 366417.84it/s]
```

```text
91%|█████████ | 5461673/6000000 [00:15<00:01, 366394.57it/s]
```

```text
92%|█████████▏| 5498322/6000000 [00:15<00:01, 366275.11it/s]
```

```text
92%|█████████▏| 5535474/6000000 [00:15<00:01, 367846.11it/s]
```

```text
93%|█████████▎| 5572264/6000000 [00:15<00:01, 367414.03it/s]
```

```text
93%|█████████▎| 5609387/6000000 [00:15<00:01, 368557.24it/s]
```

```text
94%|█████████▍| 5646552/6000000 [00:15<00:00, 369480.93it/s]
```

```text
95%|█████████▍| 5683896/6000000 [00:15<00:00, 370664.84it/s]
```

```text
95%|█████████▌| 5721043/6000000 [00:16<00:00, 370905.06it/s]
```

```text
96%|█████████▌| 5758432/6000000 [00:16<00:00, 371800.11it/s]
```

```text
97%|█████████▋| 5795613/6000000 [00:16<00:00, 371213.87it/s]
```

```text
97%|█████████▋| 5832736/6000000 [00:16<00:00, 370665.93it/s]
```

```text
98%|█████████▊| 5870278/6000000 [00:16<00:00, 372088.19it/s]
```

```text
98%|█████████▊| 5907488/6000000 [00:16<00:00, 371630.74it/s]
```

```text
99%|█████████▉| 5944897/6000000 [00:16<00:00, 372366.22it/s]
```

```text
100%|█████████▉| 5982135/6000000 [00:16<00:00, 371705.58it/s]
```

```text
100%|██████████| 6000000/6000000 [00:16<00:00, 356760.04it/s]
```

```text

```

```python
colors = {
    'ε = 0.1': 'tab:blue',
    'ε = 0.01': 'tab:red',
    'ε = 0': 'tab:green'
}

(
    trace_df
    .groupby(['step', 'policy'])
    ['is_action_optimal'].mean()
    .unstack()
    .plot()
)
```

## Handling drift

The environment used above is a toy situation used for introducing bandits. It is stationary, meaning that the expected reward of each arm does not change over time.

In practice, arms are dynamic, and their performance can vary over time. A simple example of this is the [Candy Cane Contest](<https://www.kaggle.com/c/santa-2020>) that was hosted on Kaggle in 2020. The expected reward of each arm diminishes each time it is pulled.

The way bandit policies in River deal with drift depends on the method. For the [`bandit.EpsilonGreedy`](<https://riverml.xyz/dev/api/bandit/EpsilonGreedy>) policy, it makes sense to use a rolling average as the reward object. What this means is that the empirical reward the policy calculates for each arm is a rolling average, rather than a global one.

```python
from river import proba, utils

policies=[
    bandit.EpsilonGreedy(
        epsilon=0.1,
        seed=42
    ),
    bandit.EpsilonGreedy(
        epsilon=0.3,
        reward_obj=utils.Rolling(stats.Mean(), window_size=50),
        seed=42
    ),
    bandit.ThompsonSampling(
        reward_obj=proba.Beta(),
        seed=42
    )
]

env = gym.make('river_bandits/CandyCaneContest-v0')

trace = bandit.evaluate(
    policies=policies,
    env=env,
    n_episodes=(n_episodes := 30),
    seed=42
)

trace_df = pd.DataFrame(tqdm(
    trace, position=0, total=(
        n_episodes *
        len(policies) *
        env._max_episode_steps
    )
))
```

```text
0%|          | 0/180000 [00:00<?, ?it/s]
```

```text
1%|▏         | 2400/180000 [00:00<00:07, 23980.53it/s]
```

```text
3%|▎         | 4854/180000 [00:00<00:07, 24301.53it/s]
```

```text
4%|▍         | 7293/180000 [00:00<00:07, 24340.46it/s]
```

```text
5%|▌         | 9735/180000 [00:00<00:06, 24366.36it/s]
```

```text
7%|▋         | 12172/180000 [00:00<00:06, 24136.40it/s]
```

```text
8%|▊         | 14658/180000 [00:00<00:06, 24375.06it/s]
```

```text
10%|▉         | 17103/180000 [00:00<00:06, 24396.75it/s]
```

```text
11%|█         | 19543/180000 [00:00<00:06, 24374.37it/s]
```

```text
12%|█▏        | 21999/180000 [00:00<00:06, 24431.23it/s]
```

```text
14%|█▎        | 24443/180000 [00:01<00:06, 24320.49it/s]
```

```text
15%|█▍        | 26876/180000 [00:01<00:06, 23952.40it/s]
```

```text
16%|█▋        | 29273/180000 [00:01<00:06, 23519.72it/s]
```

```text
18%|█▊        | 31627/180000 [00:01<00:06, 23303.66it/s]
```

```text
19%|█▉        | 33959/180000 [00:01<00:06, 22646.26it/s]
```

```text
20%|██        | 36228/180000 [00:01<00:06, 21488.75it/s]
```

```text
21%|██▏       | 38389/180000 [00:01<00:06, 20861.66it/s]
```

```text
22%|██▏       | 40485/180000 [00:01<00:06, 20556.73it/s]
```

```text
24%|██▎       | 42655/180000 [00:01<00:06, 20839.69it/s]
```

```text
25%|██▍       | 44745/180000 [00:01<00:06, 20741.14it/s]
```

```text
26%|██▌       | 46871/180000 [00:02<00:06, 20890.90it/s]
```

```text
27%|██▋       | 48964/180000 [00:02<00:06, 19599.69it/s]
```

```text
28%|██▊       | 51144/180000 [00:02<00:06, 20216.56it/s]
```

```text
30%|██▉       | 53553/180000 [00:02<00:05, 21326.54it/s]
```

```text
31%|███       | 55936/180000 [00:02<00:05, 22054.47it/s]
```

```text
32%|███▏      | 58335/180000 [00:02<00:05, 22616.59it/s]
```

```text
34%|███▎      | 60720/180000 [00:02<00:05, 22980.33it/s]
```

```text
35%|███▌      | 63147/180000 [00:02<00:05, 23360.12it/s]
```

```text
36%|███▋      | 65571/180000 [00:02<00:04, 23621.41it/s]
```

```text
38%|███▊      | 67962/180000 [00:02<00:04, 23700.88it/s]
```

```text
39%|███▉      | 70380/180000 [00:03<00:04, 23838.39it/s]
```

```text
40%|████      | 72771/180000 [00:03<00:04, 23854.19it/s]
```

```text
42%|████▏     | 75199/180000 [00:03<00:04, 23980.85it/s]
```

```text
43%|████▎     | 77599/180000 [00:03<00:04, 23921.90it/s]
```

```text
44%|████▍     | 79993/180000 [00:03<00:04, 23305.06it/s]
```

```text
46%|████▌     | 82328/180000 [00:03<00:04, 23192.30it/s]
```

```text
47%|████▋     | 84689/180000 [00:03<00:04, 23314.85it/s]
```

```text
48%|████▊     | 87023/180000 [00:03<00:04, 23051.14it/s]
```

```text
50%|████▉     | 89330/180000 [00:03<00:04, 21622.66it/s]
```

```text
51%|█████     | 91692/180000 [00:04<00:03, 22184.86it/s]
```

```text
52%|█████▏    | 94125/180000 [00:04<00:03, 22797.97it/s]
```

```text
54%|█████▎    | 96520/180000 [00:04<00:03, 23132.78it/s]
```

```text
55%|█████▍    | 98973/180000 [00:04<00:03, 23542.75it/s]
```

```text
56%|█████▋    | 101404/180000 [00:04<00:03, 23769.07it/s]
```

```text
58%|█████▊    | 103817/180000 [00:04<00:03, 23876.14it/s]
```

```text
59%|█████▉    | 106224/180000 [00:04<00:03, 23932.58it/s]
```

```text
60%|██████    | 108621/180000 [00:04<00:02, 23905.31it/s]
```

```text
62%|██████▏   | 111038/180000 [00:04<00:02, 23984.20it/s]
```

```text
63%|██████▎   | 113475/180000 [00:04<00:02, 24093.22it/s]
```

```text
64%|██████▍   | 115890/180000 [00:05<00:02, 24106.90it/s]
```

```text
66%|██████▌   | 118329/180000 [00:05<00:02, 24187.90it/s]
```

```text
67%|██████▋   | 120753/180000 [00:05<00:02, 24197.93it/s]
```

```text
68%|██████▊   | 123219/180000 [00:05<00:02, 24332.82it/s]
```

```text
70%|██████▉   | 125679/180000 [00:05<00:02, 24409.61it/s]
```

```text
71%|███████   | 128121/180000 [00:05<00:02, 24375.97it/s]
```

```text
73%|███████▎  | 130559/180000 [00:05<00:02, 24330.97it/s]
```

```text
74%|███████▍  | 132993/180000 [00:05<00:01, 24269.24it/s]
```

```text
75%|███████▌  | 135421/180000 [00:05<00:01, 24253.83it/s]
```

```text
77%|███████▋  | 137847/180000 [00:05<00:01, 24234.13it/s]
```

```text
78%|███████▊  | 140271/180000 [00:06<00:01, 24143.43it/s]
```

```text
79%|███████▉  | 142692/180000 [00:06<00:01, 24158.26it/s]
```

```text
81%|████████  | 145108/180000 [00:06<00:01, 24112.95it/s]
```

```text
82%|████████▏ | 147569/180000 [00:06<00:01, 24260.91it/s]
```

```text
83%|████████▎ | 150001/180000 [00:06<00:01, 24253.93it/s]
```

```text
85%|████████▍ | 152487/180000 [00:06<00:01, 24429.44it/s]
```

```text
86%|████████▌ | 154931/180000 [00:06<00:01, 24403.67it/s]
```

```text
87%|████████▋ | 157372/180000 [00:06<00:00, 24309.37it/s]
```

```text
89%|████████▉ | 159804/180000 [00:06<00:00, 24146.65it/s]
```

```text
90%|█████████ | 162219/180000 [00:06<00:00, 23884.52it/s]
```

```text
91%|█████████▏| 164673/180000 [00:07<00:00, 24076.32it/s]
```

```text
93%|█████████▎| 167090/180000 [00:07<00:00, 24103.52it/s]
```

```text
94%|█████████▍| 169509/180000 [00:07<00:00, 24125.77it/s]
```

```text
96%|█████████▌| 171946/180000 [00:07<00:00, 24197.76it/s]
```

```text
97%|█████████▋| 174367/180000 [00:07<00:00, 24036.41it/s]
```

```text
98%|█████████▊| 176823/180000 [00:07<00:00, 24188.34it/s]
```

```text
100%|█████████▉| 179294/180000 [00:07<00:00, 24343.01it/s]
```

```text
100%|██████████| 180000/180000 [00:07<00:00, 23455.28it/s]
```

```text

```

We can compare the performance of each policy by checking the average reward at the end of each episode.

```python
(
    trace_df
    .groupby(['policy_idx', 'episode'])
    .last()
    .groupby('policy_idx')
    .reward_stat.mean()
)
```

```text
policy_idx
0    736.1
1    817.0
2    854.0
Name: reward_stat, dtype: float64
```

We see that using a rolling average gives a boost to the epsilon greedy strategy. However, we see that the [`bandit.ThompsonSampling`](<https://riverml.xyz/dev/api/bandit/ThompsonSampling>) policy performs even better, even though no particular care was given to drift. A natural next step would thus be to see how it could be improved to handle drift. For instance, its `dist` parameter could be wrapped with a [`utils.Rolling`](<https://riverml.xyz/dev/api/utils/Rolling>):

```python
policy = bandit.ThompsonSampling(
    reward_obj=utils.Rolling(proba.Beta(), window_size=50),
    seed=42
)
```

Bandits can be used for several tasks. They can be used for content personalization, as well as online model selection (see [`model_selection.BanditRegressor`](<https://riverml.xyz/dev/api/model-selection/BanditRegressor>)). The policies in River are therefore designed to be flexible, so that they can be used in conjunction with other River modules. For instance, the `reward_obj` in [`bandit.EpsilonGreedy`](<https://riverml.xyz/dev/api/bandit/EpsilonGreedy>) can be a metric, a probability distribution, or a statistic. This works because objects in River adher to a coherent get/update interface.