Multi-armed bandits¶
River has a bandit module. It contains several multi-armed bandit policies, bandit environments, and utilities to benchmark policies on bandit problems.
Bandit environments in River implement the Gym interface. You can thus load them with gym.make. Note that Gym is intended for reinforcement learning algorithms, while bandit policies are the simplest form of reinforcement learning. Bandit policies learn by receiving a reward after each step, while reinforcement learning algorithms have to learn from feedback that may arrive at the end of a (long) sequence of steps.
import gymnasium as gym
for k in gym.envs.registry:
if k.startswith('river_bandits'):
print(k)
River's bandit module offers the bandit.evaluate function to benchmark several policies on a given environment. It takes as input a list of bandit policies, a bandit environment (the problem to solve), and a reward object.
import gymnasium as gym
from river import bandit
import pandas as pd
from tqdm import tqdm
from river import stats
policies=[
bandit.EpsilonGreedy(epsilon=0.1),
bandit.EpsilonGreedy(epsilon=0.01),
bandit.EpsilonGreedy(epsilon=0),
]
env = gym.make(
'river_bandits/KArmedTestbed-v0',
max_episode_steps=1000
)
trace = bandit.evaluate(
policies=policies,
env=env,
reward_stat=stats.Mean(),
n_episodes=(n_episodes := 2000),
)
The bandit.evaluate function returns a generator containing the results at each step of the benchmark. This can be wrapped with a pandas.DataFrame to gather all the results.
trace_df = pd.DataFrame(tqdm(
trace, position=0, total=(
n_episodes *
len(policies) *
env._max_episode_steps
)
))
trace_df.sample(5, random_state=42)
0%| | 0/6000000 [00:00<?, ?it/s]
0%| | 15313/6000000 [00:00<00:39, 153123.21it/s]
1%| | 32617/6000000 [00:00<00:36, 164833.30it/s]
1%| | 49572/6000000 [00:00<00:35, 166983.97it/s]
1%| | 66580/6000000 [00:00<00:35, 168201.84it/s]
1%|▏ | 84001/6000000 [00:00<00:34, 169957.53it/s]
2%|▏ | 101054/6000000 [00:00<00:34, 170150.67it/s]
2%|▏ | 118069/6000000 [00:00<00:34, 169977.00it/s]
2%|▏ | 135365/6000000 [00:00<00:34, 170923.26it/s]
3%|▎ | 152628/6000000 [00:00<00:34, 171453.52it/s]
3%|▎ | 169774/6000000 [00:01<00:40, 143881.43it/s]
3%|▎ | 186608/6000000 [00:01<00:38, 150483.07it/s]
3%|▎ | 203856/6000000 [00:01<00:37, 156603.76it/s]
4%|▎ | 220954/6000000 [00:01<00:35, 160692.31it/s]
4%|▍ | 238115/6000000 [00:01<00:35, 163846.30it/s]
4%|▍ | 255005/6000000 [00:01<00:34, 165320.95it/s]
5%|▍ | 272356/6000000 [00:01<00:34, 167731.66it/s]
5%|▍ | 289266/6000000 [00:01<00:34, 167614.21it/s]
5%|▌ | 306123/6000000 [00:01<00:33, 167878.30it/s]
5%|▌ | 322999/6000000 [00:01<00:33, 168140.03it/s]
6%|▌ | 340166/6000000 [00:02<00:33, 169190.37it/s]
6%|▌ | 357211/6000000 [00:02<00:33, 169564.52it/s]
6%|▌ | 374598/6000000 [00:02<00:32, 170850.44it/s]
7%|▋ | 391701/6000000 [00:02<00:32, 170572.81it/s]
7%|▋ | 408771/6000000 [00:02<00:33, 168195.21it/s]
7%|▋ | 425606/6000000 [00:02<00:33, 165049.50it/s]
7%|▋ | 442327/6000000 [00:02<00:33, 165680.90it/s]
8%|▊ | 459237/6000000 [00:02<00:33, 166689.99it/s]
8%|▊ | 476751/6000000 [00:02<00:32, 169196.78it/s]
8%|▊ | 493683/6000000 [00:02<00:32, 168711.47it/s]
9%|▊ | 510748/6000000 [00:03<00:32, 169286.23it/s]
9%|▉ | 528001/6000000 [00:03<00:32, 169756.08it/s]
9%|▉ | 545162/6000000 [00:03<00:32, 170305.86it/s]
9%|▉ | 562249/6000000 [00:03<00:31, 170470.83it/s]
10%|▉ | 579299/6000000 [00:03<00:31, 169741.47it/s]
10%|▉ | 596597/6000000 [00:03<00:31, 170706.38it/s]
10%|█ | 613670/6000000 [00:03<00:31, 170642.71it/s]
11%|█ | 630792/6000000 [00:03<00:31, 170813.27it/s]
11%|█ | 648003/6000000 [00:03<00:31, 171178.37it/s]
11%|█ | 665317/6000000 [00:03<00:31, 171762.74it/s]
11%|█▏ | 682494/6000000 [00:04<00:31, 171375.94it/s]
12%|█▏ | 699633/6000000 [00:04<00:31, 169861.98it/s]
12%|█▏ | 717001/6000000 [00:04<00:31, 170260.94it/s]
12%|█▏ | 734255/6000000 [00:04<00:30, 170934.77it/s]
13%|█▎ | 751351/6000000 [00:04<00:30, 170329.70it/s]
13%|█▎ | 768386/6000000 [00:04<00:30, 170087.83it/s]
13%|█▎ | 785734/6000000 [00:04<00:30, 171098.98it/s]
13%|█▎ | 802846/6000000 [00:04<00:30, 170709.83it/s]
14%|█▎ | 819918/6000000 [00:04<00:30, 170507.12it/s]
14%|█▍ | 837001/6000000 [00:04<00:30, 170458.16it/s]
14%|█▍ | 854090/6000000 [00:05<00:30, 170585.99it/s]
15%|█▍ | 871412/6000000 [00:05<00:29, 171372.17it/s]
15%|█▍ | 888550/6000000 [00:05<00:29, 171343.98it/s]
15%|█▌ | 905876/6000000 [00:05<00:29, 171916.74it/s]
15%|█▌ | 923068/6000000 [00:05<00:29, 171322.77it/s]
16%|█▌ | 940201/6000000 [00:05<00:29, 171047.74it/s]
16%|█▌ | 957307/6000000 [00:05<00:30, 165405.58it/s]
16%|█▌ | 974619/6000000 [00:05<00:29, 167659.07it/s]
17%|█▋ | 991511/6000000 [00:05<00:29, 168027.61it/s]
17%|█▋ | 1008554/6000000 [00:06<00:29, 168737.42it/s]
17%|█▋ | 1025495/6000000 [00:06<00:29, 168936.12it/s]
17%|█▋ | 1042402/6000000 [00:06<00:29, 168500.26it/s]
18%|█▊ | 1059381/6000000 [00:06<00:29, 168884.04it/s]
18%|█▊ | 1076714/6000000 [00:06<00:28, 170210.58it/s]
18%|█▊ | 1093741/6000000 [00:06<00:29, 168911.91it/s]
19%|█▊ | 1110677/6000000 [00:06<00:28, 169042.96it/s]
19%|█▉ | 1127586/6000000 [00:06<00:29, 167258.01it/s]
19%|█▉ | 1144318/6000000 [00:06<00:29, 167023.66it/s]
19%|█▉ | 1161025/6000000 [00:06<00:29, 166735.03it/s]
20%|█▉ | 1178335/6000000 [00:07<00:28, 168627.56it/s]
20%|█▉ | 1195245/6000000 [00:07<00:28, 168766.43it/s]
20%|██ | 1212125/6000000 [00:07<00:28, 168589.70it/s]
20%|██ | 1229255/6000000 [00:07<00:28, 169398.52it/s]
21%|██ | 1246197/6000000 [00:07<00:28, 167886.93it/s]
21%|██ | 1263001/6000000 [00:07<00:28, 167414.24it/s]
21%|██▏ | 1279746/6000000 [00:07<00:28, 164836.67it/s]
22%|██▏ | 1296239/6000000 [00:07<00:29, 159215.91it/s]
22%|██▏ | 1313291/6000000 [00:07<00:28, 162484.19it/s]
22%|██▏ | 1330175/6000000 [00:07<00:28, 164339.16it/s]
22%|██▏ | 1347051/6000000 [00:08<00:28, 165640.18it/s]
23%|██▎ | 1364343/6000000 [00:08<00:27, 167795.58it/s]
23%|██▎ | 1381144/6000000 [00:08<00:27, 167334.07it/s]
23%|██▎ | 1398001/6000000 [00:08<00:27, 167620.01it/s]
24%|██▎ | 1415139/6000000 [00:08<00:27, 168740.34it/s]
24%|██▍ | 1432101/6000000 [00:08<00:27, 169000.13it/s]
24%|██▍ | 1449084/6000000 [00:08<00:26, 169247.29it/s]
24%|██▍ | 1466013/6000000 [00:08<00:27, 166227.81it/s]
25%|██▍ | 1482776/6000000 [00:08<00:27, 166639.64it/s]
25%|██▍ | 1499944/6000000 [00:08<00:26, 168134.88it/s]
25%|██▌ | 1516904/6000000 [00:09<00:26, 168570.32it/s]
26%|██▌ | 1533768/6000000 [00:09<00:26, 167747.41it/s]
26%|██▌ | 1551003/6000000 [00:09<00:26, 169105.33it/s]
26%|██▌ | 1567930/6000000 [00:09<00:26, 169151.76it/s]
26%|██▋ | 1584849/6000000 [00:09<00:26, 168090.21it/s]
27%|██▋ | 1602001/6000000 [00:09<00:26, 168567.31it/s]
27%|██▋ | 1619318/6000000 [00:09<00:25, 169935.27it/s]
27%|██▋ | 1636315/6000000 [00:09<00:25, 168656.41it/s]
28%|██▊ | 1653200/6000000 [00:09<00:25, 168710.80it/s]
28%|██▊ | 1670520/6000000 [00:09<00:25, 170047.34it/s]
28%|██▊ | 1687528/6000000 [00:10<00:25, 169714.02it/s]
28%|██▊ | 1704502/6000000 [00:10<00:25, 169130.02it/s]
29%|██▊ | 1721580/6000000 [00:10<00:25, 169619.54it/s]
29%|██▉ | 1738544/6000000 [00:10<00:25, 169163.98it/s]
29%|██▉ | 1755462/6000000 [00:10<00:25, 168675.62it/s]
30%|██▉ | 1772647/6000000 [00:10<00:24, 169621.49it/s]
30%|██▉ | 1789732/6000000 [00:10<00:24, 169986.18it/s]
30%|███ | 1806732/6000000 [00:10<00:24, 168370.64it/s]
30%|███ | 1824001/6000000 [00:10<00:24, 169311.34it/s]
31%|███ | 1841256/6000000 [00:10<00:24, 170274.36it/s]
31%|███ | 1858287/6000000 [00:11<00:24, 170114.49it/s]
31%|███▏ | 1875301/6000000 [00:11<00:24, 169947.27it/s]
32%|███▏ | 1892342/6000000 [00:11<00:24, 170083.80it/s]
32%|███▏ | 1909352/6000000 [00:11<00:24, 169556.85it/s]
32%|███▏ | 1926309/6000000 [00:11<00:24, 168805.53it/s]
32%|███▏ | 1943285/6000000 [00:11<00:23, 169089.01it/s]
33%|███▎ | 1960200/6000000 [00:11<00:23, 169104.99it/s]
33%|███▎ | 1977112/6000000 [00:11<00:24, 167540.49it/s]
33%|███▎ | 1994101/6000000 [00:11<00:23, 168236.25it/s]
34%|███▎ | 2010928/6000000 [00:11<00:23, 167487.85it/s]
34%|███▍ | 2027979/6000000 [00:12<00:23, 168384.51it/s]
34%|███▍ | 2044820/6000000 [00:12<00:23, 167720.40it/s]
34%|███▍ | 2061594/6000000 [00:12<00:23, 167479.20it/s]
35%|███▍ | 2078720/6000000 [00:12<00:23, 168604.45it/s]
35%|███▍ | 2095582/6000000 [00:12<00:23, 167798.35it/s]
35%|███▌ | 2112364/6000000 [00:12<00:23, 167586.12it/s]
35%|███▌ | 2129124/6000000 [00:12<00:23, 167423.41it/s]
36%|███▌ | 2145868/6000000 [00:12<00:23, 162786.35it/s]
36%|███▌ | 2162776/6000000 [00:12<00:23, 164628.31it/s]
36%|███▋ | 2179645/6000000 [00:12<00:23, 165826.18it/s]
37%|███▋ | 2196414/6000000 [00:13<00:22, 166377.11it/s]
37%|███▋ | 2213535/6000000 [00:13<00:22, 167813.70it/s]
37%|███▋ | 2230328/6000000 [00:13<00:22, 167845.70it/s]
37%|███▋ | 2247120/6000000 [00:13<00:22, 167433.25it/s]
38%|███▊ | 2263901/6000000 [00:13<00:22, 167544.02it/s]
38%|███▊ | 2280660/6000000 [00:13<00:22, 167131.82it/s]
38%|███▊ | 2297679/6000000 [00:13<00:22, 168044.32it/s]
39%|███▊ | 2314486/6000000 [00:13<00:22, 161214.04it/s]
39%|███▉ | 2331535/6000000 [00:13<00:22, 163910.33it/s]
39%|███▉ | 2348395/6000000 [00:13<00:22, 165283.79it/s]
39%|███▉ | 2364962/6000000 [00:14<00:22, 165146.81it/s]
40%|███▉ | 2382005/6000000 [00:14<00:21, 166710.45it/s]
40%|███▉ | 2398809/6000000 [00:14<00:21, 167102.96it/s]
40%|████ | 2415569/6000000 [00:14<00:21, 167249.22it/s]
41%|████ | 2432592/6000000 [00:14<00:21, 168138.51it/s]
41%|████ | 2449414/6000000 [00:14<00:21, 167928.31it/s]
41%|████ | 2466213/6000000 [00:14<00:21, 167151.76it/s]
41%|████▏ | 2482933/6000000 [00:14<00:21, 163863.44it/s]
42%|████▏ | 2499902/6000000 [00:14<00:21, 165577.99it/s]
42%|████▏ | 2517001/6000000 [00:15<00:20, 166865.31it/s]
42%|████▏ | 2534275/6000000 [00:15<00:20, 168609.43it/s]
43%|████▎ | 2551146/6000000 [00:15<00:20, 168334.70it/s]
43%|████▎ | 2568001/6000000 [00:15<00:20, 167713.82it/s]
43%|████▎ | 2585390/6000000 [00:15<00:20, 169549.40it/s]
43%|████▎ | 2602396/6000000 [00:15<00:20, 169700.16it/s]
44%|████▎ | 2619419/6000000 [00:15<00:19, 169856.17it/s]
44%|████▍ | 2636517/6000000 [00:15<00:19, 170190.22it/s]
44%|████▍ | 2653538/6000000 [00:15<00:19, 169107.70it/s]
45%|████▍ | 2670452/6000000 [00:15<00:20, 166048.74it/s]
45%|████▍ | 2687781/6000000 [00:16<00:19, 168182.18it/s]
45%|████▌ | 2704945/6000000 [00:16<00:19, 169205.18it/s]
45%|████▌ | 2721876/6000000 [00:16<00:19, 169044.56it/s]
46%|████▌ | 2739007/6000000 [00:16<00:19, 169717.58it/s]
46%|████▌ | 2756634/6000000 [00:16<00:18, 171672.27it/s]
46%|████▌ | 2773806/6000000 [00:16<00:18, 170620.42it/s]
47%|████▋ | 2790873/6000000 [00:16<00:18, 170452.13it/s]
47%|████▋ | 2807922/6000000 [00:16<00:18, 170413.53it/s]
47%|████▋ | 2825055/6000000 [00:16<00:18, 170685.60it/s]
47%|████▋ | 2842126/6000000 [00:16<00:19, 162884.56it/s]
48%|████▊ | 2859001/6000000 [00:17<00:19, 164108.70it/s]
48%|████▊ | 2876150/6000000 [00:17<00:18, 166260.10it/s]
48%|████▊ | 2893447/6000000 [00:17<00:18, 168232.86it/s]
49%|████▊ | 2910336/6000000 [00:17<00:18, 168425.45it/s]
49%|████▉ | 2927636/6000000 [00:17<00:18, 169781.97it/s]
49%|████▉ | 2944633/6000000 [00:17<00:18, 169239.50it/s]
49%|████▉ | 2961643/6000000 [00:17<00:17, 169492.76it/s]
50%|████▉ | 2978856/6000000 [00:17<00:17, 170278.38it/s]
50%|████▉ | 2995891/6000000 [00:17<00:17, 170023.53it/s]
50%|█████ | 3012899/6000000 [00:17<00:18, 163033.74it/s]
51%|█████ | 3030001/6000000 [00:18<00:18, 164733.14it/s]
51%|█████ | 3046884/6000000 [00:18<00:17, 165928.78it/s]
51%|█████ | 3063840/6000000 [00:18<00:17, 166997.94it/s]
51%|█████▏ | 3081001/6000000 [00:18<00:17, 167922.94it/s]
52%|█████▏ | 3097932/6000000 [00:18<00:17, 168333.29it/s]
52%|█████▏ | 3114801/6000000 [00:18<00:17, 168438.24it/s]
52%|█████▏ | 3131969/6000000 [00:18<00:16, 169402.87it/s]
52%|█████▏ | 3149423/6000000 [00:18<00:16, 170936.95it/s]
53%|█████▎ | 3166551/6000000 [00:18<00:16, 171036.48it/s]
53%|█████▎ | 3183659/6000000 [00:18<00:17, 164760.59it/s]
53%|█████▎ | 3200953/6000000 [00:19<00:16, 167144.74it/s]
54%|█████▎ | 3218047/6000000 [00:19<00:16, 168259.07it/s]
54%|█████▍ | 3234999/6000000 [00:19<00:16, 168631.55it/s]
54%|█████▍ | 3252069/6000000 [00:19<00:16, 169244.65it/s]
54%|█████▍ | 3269414/6000000 [00:19<00:16, 170496.64it/s]
55%|█████▍ | 3286477/6000000 [00:19<00:15, 170154.10it/s]
55%|█████▌ | 3303502/6000000 [00:19<00:15, 169942.91it/s]
55%|█████▌ | 3320841/6000000 [00:19<00:15, 170969.62it/s]
56%|█████▌ | 3337943/6000000 [00:19<00:15, 170544.16it/s]
56%|█████▌ | 3355001/6000000 [00:19<00:16, 161382.49it/s]
56%|█████▌ | 3372100/6000000 [00:20<00:16, 164144.53it/s]
56%|█████▋ | 3389325/6000000 [00:20<00:15, 166504.18it/s]
57%|█████▋ | 3406351/6000000 [00:20<00:15, 167606.56it/s]
57%|█████▋ | 3423577/6000000 [00:20<00:15, 168981.62it/s]
57%|█████▋ | 3440878/6000000 [00:20<00:15, 170176.69it/s]
58%|█████▊ | 3457922/6000000 [00:20<00:14, 169765.82it/s]
58%|█████▊ | 3474917/6000000 [00:20<00:14, 169745.50it/s]
58%|█████▊ | 3492002/6000000 [00:20<00:14, 170016.74it/s]
58%|█████▊ | 3509013/6000000 [00:20<00:14, 170031.09it/s]
59%|█████▉ | 3526023/6000000 [00:21<00:15, 159832.21it/s]
59%|█████▉ | 3543001/6000000 [00:21<00:15, 162447.19it/s]
59%|█████▉ | 3560182/6000000 [00:21<00:14, 165160.33it/s]
60%|█████▉ | 3577079/6000000 [00:21<00:14, 166274.48it/s]
60%|█████▉ | 3594004/6000000 [00:21<00:14, 167149.30it/s]
60%|██████ | 3611374/6000000 [00:21<00:14, 169089.56it/s]
60%|██████ | 3628476/6000000 [00:21<00:13, 169663.57it/s]
61%|██████ | 3645466/6000000 [00:21<00:13, 169566.64it/s]
61%|██████ | 3662751/6000000 [00:21<00:13, 170544.30it/s]
61%|██████▏ | 3679818/6000000 [00:21<00:13, 169431.06it/s]
62%|██████▏ | 3696771/6000000 [00:22<00:14, 161977.62it/s]
62%|██████▏ | 3714001/6000000 [00:22<00:13, 164756.80it/s]
62%|██████▏ | 3731132/6000000 [00:22<00:13, 166668.60it/s]
62%|██████▏ | 3747847/6000000 [00:22<00:13, 166639.37it/s]
63%|██████▎ | 3764796/6000000 [00:22<00:13, 167479.33it/s]
63%|██████▎ | 3781569/6000000 [00:22<00:13, 166849.99it/s]
63%|██████▎ | 3798475/6000000 [00:22<00:13, 167505.60it/s]
64%|██████▎ | 3815557/6000000 [00:22<00:12, 168492.66it/s]
64%|██████▍ | 3832699/6000000 [00:22<00:12, 169364.02it/s]
64%|██████▍ | 3849778/6000000 [00:22<00:12, 169788.84it/s]
64%|██████▍ | 3866762/6000000 [00:23<00:12, 165153.88it/s]
65%|██████▍ | 3883557/6000000 [00:23<00:12, 165970.75it/s]
65%|██████▌ | 3900571/6000000 [00:23<00:12, 167200.83it/s]
65%|██████▌ | 3917957/6000000 [00:23<00:12, 169174.95it/s]
66%|██████▌ | 3934890/6000000 [00:23<00:12, 168924.32it/s]
66%|██████▌ | 3951813/6000000 [00:23<00:12, 169012.61it/s]
66%|██████▌ | 3969001/6000000 [00:23<00:11, 169414.84it/s]
66%|██████▋ | 3986265/6000000 [00:23<00:11, 170376.26it/s]
67%|██████▋ | 4003307/6000000 [00:23<00:11, 169889.05it/s]
67%|██████▋ | 4020299/6000000 [00:23<00:11, 169339.84it/s]
67%|██████▋ | 4037236/6000000 [00:24<00:12, 158613.74it/s]
68%|██████▊ | 4054014/6000000 [00:24<00:12, 161227.48it/s]
68%|██████▊ | 4071001/6000000 [00:24<00:11, 163417.57it/s]
68%|██████▊ | 4088335/6000000 [00:24<00:11, 166314.04it/s]
68%|██████▊ | 4105547/6000000 [00:24<00:11, 168021.16it/s]
69%|██████▊ | 4122399/6000000 [00:24<00:11, 168132.96it/s]
69%|██████▉ | 4139628/6000000 [00:24<00:10, 169367.67it/s]
69%|██████▉ | 4156622/6000000 [00:24<00:10, 169536.98it/s]
70%|██████▉ | 4173594/6000000 [00:24<00:10, 169234.42it/s]
70%|██████▉ | 4190774/6000000 [00:24<00:10, 169997.65it/s]
70%|███████ | 4207783/6000000 [00:25<00:11, 160181.87it/s]
70%|███████ | 4224788/6000000 [00:25<00:10, 163012.74it/s]
71%|███████ | 4241947/6000000 [00:25<00:10, 165505.80it/s]
71%|███████ | 4258944/6000000 [00:25<00:10, 166813.81it/s]
71%|███████▏ | 4275682/6000000 [00:25<00:10, 166848.91it/s]
72%|███████▏ | 4293001/6000000 [00:25<00:10, 168318.46it/s]
72%|███████▏ | 4310315/6000000 [00:25<00:09, 169749.89it/s]
72%|███████▏ | 4327312/6000000 [00:25<00:09, 169652.34it/s]
72%|███████▏ | 4344293/6000000 [00:25<00:09, 169401.17it/s]
73%|███████▎ | 4361515/6000000 [00:25<00:09, 170239.34it/s]
73%|███████▎ | 4378547/6000000 [00:26<00:10, 160156.94it/s]
73%|███████▎ | 4395313/6000000 [00:26<00:09, 162304.23it/s]
74%|███████▎ | 4412799/6000000 [00:26<00:09, 165949.35it/s]
74%|███████▍ | 4429830/6000000 [00:26<00:09, 167225.66it/s]
74%|███████▍ | 4446681/6000000 [00:26<00:09, 167603.85it/s]
74%|███████▍ | 4464001/6000000 [00:26<00:09, 168746.29it/s]
75%|███████▍ | 4481325/6000000 [00:26<00:08, 170080.43it/s]
75%|███████▍ | 4498356/6000000 [00:26<00:08, 169684.51it/s]
75%|███████▌ | 4515489/6000000 [00:26<00:08, 170172.79it/s]
76%|███████▌ | 4532754/6000000 [00:27<00:08, 170910.50it/s]
76%|███████▌ | 4549854/6000000 [00:27<00:08, 170447.65it/s]
76%|███████▌ | 4566905/6000000 [00:27<00:09, 158830.13it/s]
76%|███████▋ | 4583907/6000000 [00:27<00:08, 162011.99it/s]
77%|███████▋ | 4600863/6000000 [00:27<00:08, 164191.07it/s]
77%|███████▋ | 4617732/6000000 [00:27<00:08, 165502.74it/s]
77%|███████▋ | 4634947/6000000 [00:27<00:08, 167459.11it/s]
78%|███████▊ | 4652242/6000000 [00:27<00:07, 169082.46it/s]
78%|███████▊ | 4669191/6000000 [00:27<00:07, 168603.67it/s]
78%|███████▊ | 4686422/6000000 [00:27<00:07, 169703.95it/s]
78%|███████▊ | 4703607/6000000 [00:28<00:07, 170341.62it/s]
79%|███████▊ | 4720657/6000000 [00:28<00:07, 170068.03it/s]
79%|███████▉ | 4737675/6000000 [00:28<00:07, 160671.61it/s]
79%|███████▉ | 4754845/6000000 [00:28<00:07, 163837.99it/s]
80%|███████▉ | 4771896/6000000 [00:28<00:07, 165778.44it/s]
80%|███████▉ | 4789073/6000000 [00:28<00:07, 167535.13it/s]
80%|████████ | 4806286/6000000 [00:28<00:07, 168891.77it/s]
80%|████████ | 4823601/6000000 [00:28<00:06, 170154.24it/s]
81%|████████ | 4840646/6000000 [00:28<00:06, 169883.67it/s]
81%|████████ | 4857655/6000000 [00:28<00:06, 169697.44it/s]
81%|████████▏ | 4875001/6000000 [00:29<00:06, 170397.26it/s]
82%|████████▏ | 4892076/6000000 [00:29<00:06, 170500.58it/s]
82%|████████▏ | 4909133/6000000 [00:29<00:06, 162492.74it/s]
82%|████████▏ | 4926152/6000000 [00:29<00:06, 164717.48it/s]
82%|████████▏ | 4943238/6000000 [00:29<00:06, 166512.15it/s]
83%|████████▎ | 4960019/6000000 [00:29<00:06, 166893.10it/s]
83%|████████▎ | 4977147/6000000 [00:29<00:06, 168190.33it/s]
83%|████████▎ | 4994773/6000000 [00:29<00:05, 170588.76it/s]
84%|████████▎ | 5011853/6000000 [00:29<00:05, 170234.83it/s]
84%|████████▍ | 5028892/6000000 [00:29<00:05, 169950.12it/s]
84%|████████▍ | 5045910/6000000 [00:30<00:05, 170016.32it/s]
84%|████████▍ | 5062919/6000000 [00:30<00:05, 168909.05it/s]
85%|████████▍ | 5079817/6000000 [00:30<00:05, 159335.83it/s]
85%|████████▍ | 5097001/6000000 [00:30<00:05, 162650.04it/s]
85%|████████▌ | 5113785/6000000 [00:30<00:05, 164154.44it/s]
86%|████████▌ | 5130640/6000000 [00:30<00:05, 165441.28it/s]
86%|████████▌ | 5147855/6000000 [00:30<00:05, 167419.44it/s]
86%|████████▌ | 5164804/6000000 [00:30<00:04, 168029.69it/s]
86%|████████▋ | 5181805/6000000 [00:30<00:04, 168616.41it/s]
87%|████████▋ | 5198870/6000000 [00:31<00:04, 169221.67it/s]
87%|████████▋ | 5215807/6000000 [00:31<00:04, 168263.76it/s]
87%|████████▋ | 5232851/6000000 [00:31<00:04, 168909.71it/s]
87%|████████▋ | 5249751/6000000 [00:31<00:04, 160446.28it/s]
88%|████████▊ | 5266602/6000000 [00:31<00:04, 162768.63it/s]
88%|████████▊ | 5283562/6000000 [00:31<00:04, 164759.82it/s]
88%|████████▊ | 5300839/6000000 [00:31<00:04, 167114.75it/s]
89%|████████▊ | 5317596/6000000 [00:31<00:04, 167244.65it/s]
89%|████████▉ | 5334613/6000000 [00:31<00:03, 168112.72it/s]
89%|████████▉ | 5351581/6000000 [00:31<00:03, 168578.55it/s]
89%|████████▉ | 5368456/6000000 [00:32<00:03, 167992.05it/s]
90%|████████▉ | 5385267/6000000 [00:32<00:03, 167861.74it/s]
90%|█████████ | 5402267/6000000 [00:32<00:03, 168498.09it/s]
90%|█████████ | 5419123/6000000 [00:32<00:03, 160906.29it/s]
91%|█████████ | 5436053/6000000 [00:32<00:03, 163336.78it/s]
91%|█████████ | 5452926/6000000 [00:32<00:03, 164912.64it/s]
91%|█████████ | 5469848/6000000 [00:32<00:03, 166181.06it/s]
91%|█████████▏| 5487001/6000000 [00:32<00:03, 167376.81it/s]
92%|█████████▏| 5503926/6000000 [00:32<00:02, 167930.99it/s]
92%|█████████▏| 5521127/6000000 [00:32<00:02, 169144.72it/s]
92%|█████████▏| 5538343/6000000 [00:33<00:02, 170042.90it/s]
93%|█████████▎| 5555358/6000000 [00:33<00:02, 169964.97it/s]
93%|█████████▎| 5572362/6000000 [00:33<00:02, 168932.31it/s]
93%|█████████▎| 5589262/6000000 [00:33<00:02, 156859.09it/s]
93%|█████████▎| 5606235/6000000 [00:33<00:02, 160503.21it/s]
94%|█████████▎| 5622965/6000000 [00:33<00:02, 162459.85it/s]
94%|█████████▍| 5640001/6000000 [00:33<00:02, 164279.84it/s]
94%|█████████▍| 5657329/6000000 [00:33<00:02, 166919.16it/s]
95%|█████████▍| 5674505/6000000 [00:33<00:01, 168346.51it/s]
95%|█████████▍| 5691476/6000000 [00:33<00:01, 168748.19it/s]
95%|█████████▌| 5708561/6000000 [00:34<00:01, 169373.14it/s]
95%|█████████▌| 5725522/6000000 [00:34<00:01, 168805.45it/s]
96%|█████████▌| 5742630/6000000 [00:34<00:01, 169481.22it/s]
96%|█████████▌| 5759881/6000000 [00:34<00:01, 170385.10it/s]
96%|█████████▋| 5776929/6000000 [00:34<00:01, 157635.47it/s]
97%|█████████▋| 5793770/6000000 [00:34<00:01, 160684.26it/s]
97%|█████████▋| 5810658/6000000 [00:34<00:01, 163043.14it/s]
97%|█████████▋| 5827494/6000000 [00:34<00:01, 164591.24it/s]
97%|█████████▋| 5844491/6000000 [00:34<00:00, 166170.01it/s]
98%|█████████▊| 5861386/6000000 [00:34<00:00, 166989.29it/s]
98%|█████████▊| 5878131/6000000 [00:35<00:00, 166762.79it/s]
98%|█████████▊| 5895082/6000000 [00:35<00:00, 167578.38it/s]
99%|█████████▊| 5912353/6000000 [00:35<00:00, 169104.57it/s]
99%|█████████▉| 5929281/6000000 [00:35<00:00, 169078.26it/s]
99%|█████████▉| 5946201/6000000 [00:35<00:00, 155856.94it/s]
99%|█████████▉| 5963089/6000000 [00:35<00:00, 159532.83it/s]
100%|█████████▉| 5979932/6000000 [00:35<00:00, 162089.15it/s]
100%|█████████▉| 5997001/6000000 [00:35<00:00, 164407.17it/s]
100%|██████████| 6000000/6000000 [00:35<00:00, 167392.12it/s]
| episode | step | policy_idx | arm | reward | reward_stat | |
|---|---|---|---|---|---|---|
| 1324896 | 441 | 632 | 0 | 1 | 1.639248 | 1.768988 |
| 3566176 | 1188 | 725 | 1 | 0 | 0.723306 | 1.621049 |
| 1109043 | 369 | 681 | 0 | 0 | 2.522812 | 0.691467 |
| 4286042 | 1428 | 680 | 2 | 2 | -0.674206 | 0.198410 |
| 5395174 | 1798 | 391 | 1 | 3 | 3.541587 | 1.631380 |
It is then straightforward to plot the average reward each policy obtains at each step, by averaging over episodes.
policy_names = {
0: 'ε = 0.1',
1: 'ε = 0.01',
2: 'ε = 0 (greedy)'
}
(
trace_df
.assign(policy=trace_df.policy_idx.map(policy_names))
.groupby(['step', 'policy'])
['reward'].mean()
.unstack()
.plot()
)
[1m<[0m[1;95mAxes:[0m[39m [0m[33mxlabel[0m[39m=[0m[32m'step'[0m[1m>[0m
Controlling the evaluation loop¶
The bandit.evaluate function is useful for benchmarking. But in practice, you'll want to have control over your bandit policy. Indeed you'll want the freedom to pull arms (with the pull method) and update the policy (with the update method) at your discretion.
As an example, the following is a possible reimplementation of the bandit.evaluate function. Here we'll be measuring the rate at which each policy selects the optimal arm.
Note how the pull and update methods are used.
import copy
policies=[
bandit.EpsilonGreedy(epsilon=0.1),
bandit.EpsilonGreedy(epsilon=0.01),
bandit.EpsilonGreedy(epsilon=0),
]
env = gym.make(
'river_bandits/KArmedTestbed-v0',
max_episode_steps=1000
)
n_episodes = 2000
trace = []
with tqdm(total=len(policies) * n_episodes * env._max_episode_steps, position=0) as progress:
for policy in policies:
for episode in range(n_episodes):
episode_policy = policy.clone()
episode_env = copy.deepcopy(env)
episode_env.reset()
step = 0
while True:
action = episode_policy.pull(range(episode_env.action_space.n))
observation, reward, terminated, truncated, info = episode_env.step(action)
best_action = observation
episode_policy.update(action, reward)
trace.append({
"episode": episode,
"step": step,
"policy": f"ε = {policy.epsilon}",
"is_action_optimal": action == best_action
})
step += 1
progress.update()
if terminated or truncated:
break
trace_df = pd.DataFrame(trace)
0%| | 0/6000000 [00:00<?, ?it/s]
0%| | 16666/6000000 [00:00<00:35, 166651.82it/s]
1%| | 34031/6000000 [00:00<00:34, 170767.45it/s]
1%| | 51609/6000000 [00:00<00:34, 173052.56it/s]
1%| | 69174/6000000 [00:00<00:34, 174074.77it/s]
1%|▏ | 86612/6000000 [00:00<00:33, 174182.50it/s]
2%|▏ | 104311/6000000 [00:00<00:33, 175135.52it/s]
2%|▏ | 122001/6000000 [00:00<00:33, 175643.82it/s]
2%|▏ | 139716/6000000 [00:00<00:33, 176123.10it/s]
3%|▎ | 157329/6000000 [00:00<00:33, 175803.43it/s]
3%|▎ | 175074/6000000 [00:01<00:33, 176309.80it/s]
3%|▎ | 193001/6000000 [00:01<00:32, 177112.30it/s]
4%|▎ | 210713/6000000 [00:01<00:32, 176916.15it/s]
4%|▍ | 228405/6000000 [00:01<00:32, 176495.56it/s]
4%|▍ | 246060/6000000 [00:01<00:32, 176511.12it/s]
4%|▍ | 263712/6000000 [00:01<00:32, 175284.71it/s]
5%|▍ | 281431/6000000 [00:01<00:32, 175848.67it/s]
5%|▍ | 299188/6000000 [00:01<00:32, 176362.29it/s]
5%|▌ | 317001/6000000 [00:01<00:32, 176817.51it/s]
6%|▌ | 334684/6000000 [00:01<00:32, 176228.26it/s]
6%|▌ | 352308/6000000 [00:02<00:32, 176074.54it/s]
6%|▌ | 369985/6000000 [00:02<00:31, 176280.10it/s]
6%|▋ | 387697/6000000 [00:02<00:31, 176530.47it/s]
7%|▋ | 405351/6000000 [00:02<00:31, 176322.69it/s]
7%|▋ | 422984/6000000 [00:02<00:31, 175661.25it/s]
7%|▋ | 440551/6000000 [00:02<00:31, 175069.59it/s]
8%|▊ | 458059/6000000 [00:02<00:31, 174897.14it/s]
8%|▊ | 475652/6000000 [00:02<00:31, 175204.10it/s]
8%|▊ | 493222/6000000 [00:02<00:31, 175350.34it/s]
9%|▊ | 510880/6000000 [00:02<00:31, 175717.88it/s]
9%|▉ | 528568/6000000 [00:03<00:31, 176063.79it/s]
9%|▉ | 546185/6000000 [00:03<00:30, 176095.16it/s]
9%|▉ | 564001/6000000 [00:03<00:30, 176628.36it/s]
10%|▉ | 581664/6000000 [00:03<00:30, 176443.42it/s]
10%|▉ | 599309/6000000 [00:03<00:30, 176303.14it/s]
10%|█ | 616940/6000000 [00:03<00:30, 176138.69it/s]
11%|█ | 634554/6000000 [00:03<00:30, 175941.14it/s]
11%|█ | 652149/6000000 [00:03<00:30, 175892.62it/s]
11%|█ | 669739/6000000 [00:03<00:30, 175387.38it/s]
11%|█▏ | 687279/6000000 [00:03<00:30, 174922.87it/s]
12%|█▏ | 704822/6000000 [00:04<00:30, 175073.82it/s]
12%|█▏ | 722383/6000000 [00:04<00:30, 175232.39it/s]
12%|█▏ | 740062/6000000 [00:04<00:29, 175697.41it/s]
13%|█▎ | 757633/6000000 [00:04<00:29, 174928.21it/s]
13%|█▎ | 775347/6000000 [00:04<00:29, 175587.85it/s]
13%|█▎ | 793001/6000000 [00:04<00:29, 175834.48it/s]
14%|█▎ | 810586/6000000 [00:04<00:29, 175726.17it/s]
14%|█▍ | 828160/6000000 [00:04<00:29, 174345.28it/s]
14%|█▍ | 845598/6000000 [00:04<00:29, 173172.21it/s]
14%|█▍ | 862919/6000000 [00:04<00:29, 172332.71it/s]
15%|█▍ | 880155/6000000 [00:05<00:29, 172028.82it/s]
15%|█▍ | 897360/6000000 [00:05<00:30, 164932.70it/s]
15%|█▌ | 913912/6000000 [00:05<00:32, 157904.84it/s]
16%|█▌ | 930755/6000000 [00:05<00:31, 160887.61it/s]
16%|█▌ | 947787/6000000 [00:05<00:30, 163605.94it/s]
16%|█▌ | 964377/6000000 [00:05<00:30, 164273.43it/s]
16%|█▋ | 981517/6000000 [00:05<00:30, 166368.10it/s]
17%|█▋ | 998807/6000000 [00:05<00:29, 168300.08it/s]
17%|█▋ | 1016001/6000000 [00:05<00:29, 169318.74it/s]
17%|█▋ | 1033253/6000000 [00:05<00:29, 170270.90it/s]
18%|█▊ | 1050464/6000000 [00:06<00:28, 170819.68it/s]
18%|█▊ | 1067598/6000000 [00:06<00:28, 170975.16it/s]
18%|█▊ | 1084785/6000000 [00:06<00:28, 171242.03it/s]
18%|█▊ | 1101915/6000000 [00:06<00:28, 170965.24it/s]
19%|█▊ | 1119016/6000000 [00:06<00:28, 170453.87it/s]
19%|█▉ | 1136065/6000000 [00:06<00:28, 168604.22it/s]
19%|█▉ | 1152932/6000000 [00:06<00:28, 167840.62it/s]
20%|█▉ | 1170001/6000000 [00:06<00:28, 168634.71it/s]
20%|█▉ | 1187001/6000000 [00:06<00:28, 168936.91it/s]
20%|██ | 1204001/6000000 [00:06<00:28, 169109.61it/s]
20%|██ | 1221257/6000000 [00:07<00:28, 170137.93it/s]
21%|██ | 1238282/6000000 [00:07<00:27, 170168.70it/s]
21%|██ | 1255520/6000000 [00:07<00:27, 170828.42it/s]
21%|██ | 1272604/6000000 [00:07<00:27, 170544.40it/s]
21%|██▏ | 1289660/6000000 [00:07<00:27, 170250.98it/s]
22%|██▏ | 1306686/6000000 [00:07<00:27, 169874.98it/s]
22%|██▏ | 1323734/6000000 [00:07<00:27, 170055.53it/s]
22%|██▏ | 1341001/6000000 [00:07<00:27, 170715.92it/s]
23%|██▎ | 1358239/6000000 [00:07<00:27, 171211.88it/s]
23%|██▎ | 1375366/6000000 [00:07<00:27, 171229.21it/s]
23%|██▎ | 1392608/6000000 [00:08<00:26, 171584.14it/s]
23%|██▎ | 1409856/6000000 [00:08<00:26, 171851.88it/s]
24%|██▍ | 1427078/6000000 [00:08<00:26, 171962.18it/s]
24%|██▍ | 1444275/6000000 [00:08<00:26, 171679.68it/s]
24%|██▍ | 1461470/6000000 [00:08<00:26, 171759.48it/s]
25%|██▍ | 1478647/6000000 [00:08<00:26, 171440.26it/s]
25%|██▍ | 1495792/6000000 [00:08<00:26, 168391.83it/s]
25%|██▌ | 1512643/6000000 [00:08<00:26, 168341.58it/s]
25%|██▌ | 1529783/6000000 [00:08<00:26, 169248.17it/s]
26%|██▌ | 1546715/6000000 [00:08<00:26, 168599.83it/s]
26%|██▌ | 1563945/6000000 [00:09<00:26, 169699.62it/s]
26%|██▋ | 1581035/6000000 [00:09<00:25, 170057.34it/s]
27%|██▋ | 1598252/6000000 [00:09<00:25, 170686.81it/s]
27%|██▋ | 1615417/6000000 [00:09<00:25, 170974.92it/s]
27%|██▋ | 1632634/6000000 [00:09<00:25, 171331.59it/s]
27%|██▋ | 1649825/6000000 [00:09<00:25, 171502.73it/s]
28%|██▊ | 1666977/6000000 [00:09<00:25, 167184.42it/s]
28%|██▊ | 1684001/6000000 [00:09<00:25, 167907.89it/s]
28%|██▊ | 1701074/6000000 [00:09<00:25, 168742.73it/s]
29%|██▊ | 1718087/6000000 [00:09<00:25, 169154.47it/s]
29%|██▉ | 1735342/6000000 [00:10<00:25, 170165.44it/s]
29%|██▉ | 1752576/6000000 [00:10<00:24, 170812.70it/s]
29%|██▉ | 1769664/6000000 [00:10<00:24, 170443.81it/s]
30%|██▉ | 1786867/6000000 [00:10<00:24, 170916.99it/s]
30%|███ | 1803962/6000000 [00:10<00:24, 170828.95it/s]
30%|███ | 1821064/6000000 [00:10<00:24, 170884.23it/s]
31%|███ | 1838154/6000000 [00:10<00:24, 170853.62it/s]
31%|███ | 1855271/6000000 [00:10<00:24, 170946.75it/s]
31%|███ | 1872411/6000000 [00:10<00:24, 171081.79it/s]
31%|███▏ | 1889557/6000000 [00:10<00:24, 171194.01it/s]
32%|███▏ | 1906779/6000000 [00:11<00:23, 171500.73it/s]
32%|███▏ | 1924001/6000000 [00:11<00:23, 171681.39it/s]
32%|███▏ | 1941170/6000000 [00:11<00:23, 171128.77it/s]
33%|███▎ | 1958380/6000000 [00:11<00:23, 171417.25it/s]
33%|███▎ | 1975697/6000000 [00:11<00:23, 171942.07it/s]
33%|███▎ | 1992892/6000000 [00:11<00:23, 171724.29it/s]
34%|███▎ | 2010065/6000000 [00:11<00:23, 167401.81it/s]
34%|███▍ | 2026829/6000000 [00:11<00:23, 166265.88it/s]
34%|███▍ | 2043472/6000000 [00:11<00:23, 165625.37it/s]
34%|███▍ | 2060046/6000000 [00:11<00:23, 164199.40it/s]
35%|███▍ | 2076475/6000000 [00:12<00:23, 163839.09it/s]
35%|███▍ | 2092865/6000000 [00:12<00:23, 163322.83it/s]
35%|███▌ | 2109201/6000000 [00:12<00:23, 162258.54it/s]
35%|███▌ | 2125632/6000000 [00:12<00:23, 162864.64it/s]
36%|███▌ | 2142053/6000000 [00:12<00:23, 163263.80it/s]
36%|███▌ | 2158509/6000000 [00:12<00:23, 163650.09it/s]
36%|███▌ | 2174876/6000000 [00:12<00:23, 162168.15it/s]
37%|███▋ | 2191231/6000000 [00:12<00:23, 162577.99it/s]
37%|███▋ | 2207492/6000000 [00:12<00:23, 162297.58it/s]
37%|███▋ | 2223724/6000000 [00:13<00:23, 161867.09it/s]
37%|███▋ | 2240001/6000000 [00:13<00:23, 162134.53it/s]
38%|███▊ | 2256383/6000000 [00:13<00:23, 162637.14it/s]
38%|███▊ | 2272755/6000000 [00:13<00:22, 162959.33it/s]
38%|███▊ | 2289052/6000000 [00:13<00:22, 162668.13it/s]
38%|███▊ | 2305460/6000000 [00:13<00:22, 163090.46it/s]
39%|███▊ | 2321793/6000000 [00:13<00:22, 163160.95it/s]
39%|███▉ | 2338110/6000000 [00:13<00:22, 160454.07it/s]
39%|███▉ | 2354452/6000000 [00:13<00:22, 161331.76it/s]
40%|███▉ | 2370831/6000000 [00:13<00:22, 162061.01it/s]
40%|███▉ | 2387044/6000000 [00:14<00:22, 162044.77it/s]
40%|████ | 2403453/6000000 [00:14<00:22, 162653.63it/s]
40%|████ | 2419907/6000000 [00:14<00:21, 163216.28it/s]
41%|████ | 2436232/6000000 [00:14<00:21, 162673.70it/s]
41%|████ | 2452655/6000000 [00:14<00:21, 163136.24it/s]
41%|████ | 2469146/6000000 [00:14<00:21, 163664.26it/s]
41%|████▏ | 2485514/6000000 [00:14<00:21, 163619.26it/s]
42%|████▏ | 2501939/6000000 [00:14<00:21, 163807.76it/s]
42%|████▏ | 2518321/6000000 [00:14<00:21, 160232.48it/s]
42%|████▏ | 2534410/6000000 [00:14<00:21, 160425.69it/s]
43%|████▎ | 2550886/6000000 [00:15<00:21, 161709.41it/s]
43%|████▎ | 2567068/6000000 [00:15<00:21, 161641.90it/s]
43%|████▎ | 2583606/6000000 [00:15<00:20, 162756.13it/s]
43%|████▎ | 2599888/6000000 [00:15<00:20, 162733.19it/s]
44%|████▎ | 2616281/6000000 [00:15<00:20, 163090.22it/s]
44%|████▍ | 2632593/6000000 [00:15<00:20, 163092.31it/s]
44%|████▍ | 2649063/6000000 [00:15<00:20, 163572.50it/s]
44%|████▍ | 2665632/6000000 [00:15<00:20, 164206.74it/s]
45%|████▍ | 2682054/6000000 [00:15<00:20, 159530.29it/s]
45%|████▍ | 2698037/6000000 [00:15<00:20, 158389.53it/s]
45%|████▌ | 2713897/6000000 [00:16<00:21, 152255.99it/s]
45%|████▌ | 2729599/6000000 [00:16<00:21, 153613.94it/s]
46%|████▌ | 2745373/6000000 [00:16<00:21, 154813.94it/s]
46%|████▌ | 2761001/6000000 [00:16<00:20, 155184.54it/s]
46%|████▋ | 2776781/6000000 [00:16<00:20, 155955.31it/s]
47%|████▋ | 2792396/6000000 [00:16<00:20, 155874.04it/s]
47%|████▋ | 2807997/6000000 [00:16<00:20, 155825.65it/s]
47%|████▋ | 2823685/6000000 [00:16<00:20, 156138.10it/s]
47%|████▋ | 2839306/6000000 [00:16<00:20, 153208.03it/s]
48%|████▊ | 2855316/6000000 [00:16<00:20, 155239.68it/s]
48%|████▊ | 2871369/6000000 [00:17<00:19, 156807.67it/s]
48%|████▊ | 2887396/6000000 [00:17<00:19, 157837.51it/s]
48%|████▊ | 2903694/6000000 [00:17<00:19, 159371.14it/s]
49%|████▊ | 2919930/6000000 [00:17<00:19, 160263.50it/s]
49%|████▉ | 2936216/6000000 [00:17<00:19, 161037.92it/s]
49%|████▉ | 2952422/6000000 [00:17<00:18, 161343.07it/s]
49%|████▉ | 2968560/6000000 [00:17<00:18, 161205.76it/s]
50%|████▉ | 2984864/6000000 [00:17<00:18, 161753.57it/s]
50%|█████ | 3001041/6000000 [00:17<00:19, 156507.43it/s]
50%|█████ | 3017144/6000000 [00:17<00:18, 157830.73it/s]
51%|█████ | 3033338/6000000 [00:18<00:18, 159042.81it/s]
51%|█████ | 3049265/6000000 [00:18<00:19, 154694.38it/s]
51%|█████ | 3065189/6000000 [00:18<00:18, 156016.88it/s]
51%|█████▏ | 3081509/6000000 [00:18<00:18, 158127.38it/s]
52%|█████▏ | 3097606/6000000 [00:18<00:18, 158965.95it/s]
52%|█████▏ | 3113934/6000000 [00:18<00:18, 160245.13it/s]
52%|█████▏ | 3129974/6000000 [00:18<00:17, 160117.29it/s]
52%|█████▏ | 3146001/6000000 [00:18<00:17, 160150.25it/s]
53%|█████▎ | 3162024/6000000 [00:18<00:17, 157730.26it/s]
53%|█████▎ | 3178301/6000000 [00:18<00:17, 159217.35it/s]
53%|█████▎ | 3194464/6000000 [00:19<00:17, 159931.72it/s]
54%|█████▎ | 3210752/6000000 [00:19<00:17, 160809.43it/s]
54%|█████▍ | 3226840/6000000 [00:19<00:17, 160370.79it/s]
54%|█████▍ | 3243160/6000000 [00:19<00:17, 161213.28it/s]
54%|█████▍ | 3259285/6000000 [00:19<00:17, 161106.05it/s]
55%|█████▍ | 3275619/6000000 [00:19<00:16, 161772.00it/s]
55%|█████▍ | 3291892/6000000 [00:19<00:16, 162058.24it/s]
55%|█████▌ | 3308100/6000000 [00:19<00:16, 161756.94it/s]
55%|█████▌ | 3324277/6000000 [00:19<00:17, 155689.82it/s]
56%|█████▌ | 3340472/6000000 [00:19<00:16, 157513.15it/s]
56%|█████▌ | 3356654/6000000 [00:20<00:16, 158779.27it/s]
56%|█████▌ | 3373001/6000000 [00:20<00:16, 160085.04it/s]
56%|█████▋ | 3389158/6000000 [00:20<00:16, 160524.54it/s]
57%|█████▋ | 3405227/6000000 [00:20<00:16, 160354.44it/s]
57%|█████▋ | 3421442/6000000 [00:20<00:16, 160887.41it/s]
57%|█████▋ | 3437605/6000000 [00:20<00:15, 161106.65it/s]
58%|█████▊ | 3453722/6000000 [00:20<00:15, 160503.44it/s]
58%|█████▊ | 3469970/6000000 [00:20<00:15, 161093.04it/s]
58%|█████▊ | 3486083/6000000 [00:20<00:15, 160463.00it/s]
58%|█████▊ | 3502620/6000000 [00:20<00:15, 161924.94it/s]
59%|█████▊ | 3518957/6000000 [00:21<00:15, 162354.85it/s]
59%|█████▉ | 3535195/6000000 [00:21<00:15, 161637.42it/s]
59%|█████▉ | 3551411/6000000 [00:21<00:15, 161791.17it/s]
59%|█████▉ | 3567693/6000000 [00:21<00:15, 162097.53it/s]
60%|█████▉ | 3583921/6000000 [00:21<00:14, 162151.49it/s]
60%|██████ | 3600154/6000000 [00:21<00:14, 162202.45it/s]
60%|██████ | 3616669/6000000 [00:21<00:14, 163084.18it/s]
61%|██████ | 3632978/6000000 [00:21<00:14, 162586.12it/s]
61%|██████ | 3649238/6000000 [00:21<00:14, 158519.30it/s]
61%|██████ | 3665575/6000000 [00:22<00:14, 159943.74it/s]
61%|██████▏ | 3682001/6000000 [00:22<00:14, 161048.71it/s]
62%|██████▏ | 3698345/6000000 [00:22<00:14, 161755.91it/s]
62%|██████▏ | 3714641/6000000 [00:22<00:14, 162112.46it/s]
62%|██████▏ | 3730870/6000000 [00:22<00:13, 162164.11it/s]
62%|██████▏ | 3747092/6000000 [00:22<00:13, 162149.43it/s]
63%|██████▎ | 3763547/6000000 [00:22<00:13, 162867.49it/s]
63%|██████▎ | 3780000/6000000 [00:22<00:13, 163364.89it/s]
63%|██████▎ | 3796339/6000000 [00:22<00:13, 163314.36it/s]
64%|██████▎ | 3812672/6000000 [00:22<00:13, 163088.05it/s]
64%|██████▍ | 3828982/6000000 [00:23<00:13, 157650.90it/s]
64%|██████▍ | 3845150/6000000 [00:23<00:13, 158829.32it/s]
64%|██████▍ | 3861501/6000000 [00:23<00:13, 160207.85it/s]
65%|██████▍ | 3877811/6000000 [00:23<00:13, 161063.81it/s]
65%|██████▍ | 3894141/6000000 [00:23<00:13, 161728.44it/s]
65%|██████▌ | 3910327/6000000 [00:23<00:12, 161688.06it/s]
65%|██████▌ | 3926565/6000000 [00:23<00:12, 161894.27it/s]
66%|██████▌ | 3942772/6000000 [00:23<00:12, 161946.40it/s]
66%|██████▌ | 3959012/6000000 [00:23<00:12, 162081.64it/s]
66%|██████▋ | 3975342/6000000 [00:23<00:12, 162444.18it/s]
67%|██████▋ | 3991589/6000000 [00:24<00:12, 155838.53it/s]
67%|██████▋ | 4007963/6000000 [00:24<00:12, 158137.81it/s]
67%|██████▋ | 4024493/6000000 [00:24<00:12, 160240.54it/s]
67%|██████▋ | 4040914/6000000 [00:24<00:12, 161412.84it/s]
68%|██████▊ | 4057279/6000000 [00:24<00:11, 162076.61it/s]
68%|██████▊ | 4073508/6000000 [00:24<00:12, 157422.65it/s]
68%|██████▊ | 4089662/6000000 [00:24<00:12, 158624.85it/s]
68%|██████▊ | 4106180/6000000 [00:24<00:11, 160555.81it/s]
69%|██████▊ | 4122631/6000000 [00:24<00:11, 161726.57it/s]
69%|██████▉ | 4139000/6000000 [00:24<00:11, 162309.92it/s]
69%|██████▉ | 4155246/6000000 [00:25<00:11, 161661.95it/s]
70%|██████▉ | 4171900/6000000 [00:25<00:11, 163114.15it/s]
70%|██████▉ | 4188598/6000000 [00:25<00:11, 164265.95it/s]
70%|███████ | 4205250/6000000 [00:25<00:10, 164938.93it/s]
70%|███████ | 4222013/6000000 [00:25<00:10, 165741.75it/s]
71%|███████ | 4238809/6000000 [00:25<00:10, 166405.96it/s]
71%|███████ | 4255453/6000000 [00:25<00:10, 166251.48it/s]
71%|███████ | 4272080/6000000 [00:25<00:10, 165787.64it/s]
71%|███████▏ | 4288661/6000000 [00:25<00:10, 165487.10it/s]
72%|███████▏ | 4305438/6000000 [00:25<00:10, 166169.30it/s]
72%|███████▏ | 4322056/6000000 [00:26<00:10, 160171.61it/s]
72%|███████▏ | 4338911/6000000 [00:26<00:10, 162615.04it/s]
73%|███████▎ | 4355513/6000000 [00:26<00:10, 163615.33it/s]
73%|███████▎ | 4371943/6000000 [00:26<00:09, 163817.76it/s]
73%|███████▎ | 4388484/6000000 [00:26<00:09, 164288.61it/s]
73%|███████▎ | 4405043/6000000 [00:26<00:09, 164675.06it/s]
74%|███████▎ | 4421783/6000000 [00:26<00:09, 165487.92it/s]
74%|███████▍ | 4438415/6000000 [00:26<00:09, 165732.87it/s]
74%|███████▍ | 4455054/6000000 [00:26<00:09, 165929.01it/s]
75%|███████▍ | 4471651/6000000 [00:26<00:09, 165820.43it/s]
75%|███████▍ | 4488236/6000000 [00:27<00:09, 164643.55it/s]
75%|███████▌ | 4504986/6000000 [00:27<00:09, 165493.01it/s]
75%|███████▌ | 4521863/6000000 [00:27<00:08, 166470.35it/s]
76%|███████▌ | 4538513/6000000 [00:27<00:08, 166013.32it/s]
76%|███████▌ | 4555313/6000000 [00:27<00:08, 166605.39it/s]
76%|███████▌ | 4572050/6000000 [00:27<00:08, 166834.04it/s]
76%|███████▋ | 4588770/6000000 [00:27<00:08, 166941.75it/s]
77%|███████▋ | 4605466/6000000 [00:27<00:08, 166765.74it/s]
77%|███████▋ | 4622144/6000000 [00:27<00:08, 166445.14it/s]
77%|███████▋ | 4638790/6000000 [00:27<00:08, 166417.46it/s]
78%|███████▊ | 4655507/6000000 [00:28<00:08, 166641.64it/s]
78%|███████▊ | 4672172/6000000 [00:28<00:08, 160830.26it/s]
78%|███████▊ | 4688915/6000000 [00:28<00:08, 162757.90it/s]
78%|███████▊ | 4705501/6000000 [00:28<00:07, 163670.26it/s]
79%|███████▊ | 4722135/6000000 [00:28<00:07, 164458.70it/s]
79%|███████▉ | 4739001/6000000 [00:28<00:07, 165513.44it/s]
79%|███████▉ | 4755671/6000000 [00:28<00:07, 165866.69it/s]
80%|███████▉ | 4772268/6000000 [00:28<00:07, 165444.49it/s]
80%|███████▉ | 4789000/6000000 [00:28<00:07, 166002.50it/s]
80%|████████ | 4805828/6000000 [00:28<00:07, 166681.90it/s]
80%|████████ | 4822666/6000000 [00:29<00:07, 167189.42it/s]
81%|████████ | 4839388/6000000 [00:29<00:07, 162684.59it/s]
81%|████████ | 4856001/6000000 [00:29<00:06, 163587.53it/s]
81%|████████ | 4872561/6000000 [00:29<00:06, 164181.69it/s]
81%|████████▏ | 4889001/6000000 [00:29<00:06, 164190.30it/s]
82%|████████▏ | 4905816/6000000 [00:29<00:06, 165368.71it/s]
82%|████████▏ | 4922590/6000000 [00:29<00:06, 166075.07it/s]
82%|████████▏ | 4939376/6000000 [00:29<00:06, 166607.18it/s]
83%|████████▎ | 4956042/6000000 [00:29<00:06, 166538.33it/s]
83%|████████▎ | 4972700/6000000 [00:29<00:06, 166380.81it/s]
83%|████████▎ | 4989597/6000000 [00:30<00:06, 167153.84it/s]
83%|████████▎ | 5006315/6000000 [00:30<00:06, 161864.28it/s]
84%|████████▎ | 5022913/6000000 [00:30<00:05, 163067.65it/s]
84%|████████▍ | 5039696/6000000 [00:30<00:05, 164471.23it/s]
84%|████████▍ | 5056310/6000000 [00:30<00:05, 164964.55it/s]
85%|████████▍ | 5073040/6000000 [00:30<00:05, 165657.14it/s]
85%|████████▍ | 5089618/6000000 [00:30<00:05, 165160.25it/s]
85%|████████▌ | 5106308/6000000 [00:30<00:05, 165675.52it/s]
85%|████████▌ | 5122882/6000000 [00:30<00:05, 165692.94it/s]
86%|████████▌ | 5139527/6000000 [00:31<00:05, 165918.69it/s]
86%|████████▌ | 5156443/6000000 [00:31<00:05, 166888.29it/s]
86%|████████▌ | 5173135/6000000 [00:31<00:05, 162787.82it/s]
86%|████████▋ | 5189457/6000000 [00:31<00:04, 162914.10it/s]
87%|████████▋ | 5206126/6000000 [00:31<00:04, 164029.55it/s]
87%|████████▋ | 5222988/6000000 [00:31<00:04, 165393.58it/s]
87%|████████▋ | 5239675/6000000 [00:31<00:04, 165831.59it/s]
88%|████████▊ | 5256589/6000000 [00:31<00:04, 166818.38it/s]
88%|████████▊ | 5273277/6000000 [00:31<00:04, 166237.59it/s]
88%|████████▊ | 5290171/6000000 [00:31<00:04, 167042.38it/s]
88%|████████▊ | 5306879/6000000 [00:32<00:04, 166975.36it/s]
89%|████████▊ | 5323579/6000000 [00:32<00:04, 166969.44it/s]
89%|████████▉ | 5340278/6000000 [00:32<00:04, 160268.40it/s]
89%|████████▉ | 5356998/6000000 [00:32<00:03, 162285.32it/s]
90%|████████▉ | 5373691/6000000 [00:32<00:03, 163648.31it/s]
90%|████████▉ | 5390406/6000000 [00:32<00:03, 164681.08it/s]
90%|█████████ | 5407325/6000000 [00:32<00:03, 166017.11it/s]
90%|█████████ | 5424001/6000000 [00:32<00:03, 166235.44it/s]
91%|█████████ | 5440639/6000000 [00:32<00:03, 166194.67it/s]
91%|█████████ | 5457301/6000000 [00:32<00:03, 166321.45it/s]
91%|█████████ | 5473941/6000000 [00:33<00:03, 165580.40it/s]
92%|█████████▏| 5490505/6000000 [00:33<00:03, 165221.21it/s]
92%|█████████▏| 5507213/6000000 [00:33<00:02, 165775.39it/s]
92%|█████████▏| 5523794/6000000 [00:33<00:02, 163081.16it/s]
92%|█████████▏| 5540413/6000000 [00:33<00:02, 163999.28it/s]
93%|█████████▎| 5557001/6000000 [00:33<00:02, 164532.11it/s]
93%|█████████▎| 5573606/6000000 [00:33<00:02, 164981.74it/s]
93%|█████████▎| 5590110/6000000 [00:33<00:02, 164842.09it/s]
93%|█████████▎| 5606856/6000000 [00:33<00:02, 165623.32it/s]
94%|█████████▎| 5623422/6000000 [00:33<00:02, 165399.01it/s]
94%|█████████▍| 5640001/6000000 [00:34<00:02, 165501.25it/s]
94%|█████████▍| 5656915/6000000 [00:34<00:02, 166590.42it/s]
95%|█████████▍| 5673609/6000000 [00:34<00:01, 166692.80it/s]
95%|█████████▍| 5690280/6000000 [00:34<00:01, 159530.48it/s]
95%|█████████▌| 5706694/6000000 [00:34<00:01, 160869.30it/s]
95%|█████████▌| 5723186/6000000 [00:34<00:01, 162056.03it/s]
96%|█████████▌| 5739843/6000000 [00:34<00:01, 163389.08it/s]
96%|█████████▌| 5756550/6000000 [00:34<00:01, 164478.75it/s]
96%|█████████▌| 5773214/6000000 [00:34<00:01, 165120.90it/s]
97%|█████████▋| 5790001/6000000 [00:34<00:01, 165883.16it/s]
97%|█████████▋| 5806751/6000000 [00:35<00:01, 166364.40it/s]
97%|█████████▋| 5823423/6000000 [00:35<00:01, 166468.81it/s]
97%|█████████▋| 5840197/6000000 [00:35<00:00, 166849.10it/s]
98%|█████████▊| 5856886/6000000 [00:35<00:00, 163288.45it/s]
98%|█████████▊| 5873510/6000000 [00:35<00:00, 164157.32it/s]
98%|█████████▊| 5890250/6000000 [00:35<00:00, 165118.12it/s]
98%|█████████▊| 5906913/6000000 [00:35<00:00, 165566.25it/s]
99%|█████████▊| 5923614/6000000 [00:35<00:00, 165996.77it/s]
99%|█████████▉| 5940220/6000000 [00:35<00:00, 165978.34it/s]
99%|█████████▉| 5956883/6000000 [00:35<00:00, 166172.37it/s]
100%|█████████▉| 5973607/6000000 [00:36<00:00, 166489.44it/s]
100%|█████████▉| 5990336/6000000 [00:36<00:00, 166728.24it/s]
100%|██████████| 6000000/6000000 [00:36<00:00, 165662.02it/s]
colors = {
'ε = 0.1': 'tab:blue',
'ε = 0.01': 'tab:red',
'ε = 0': 'tab:green'
}
(
trace_df
.groupby(['step', 'policy'])
['is_action_optimal'].mean()
.unstack()
.plot()
)
[1m<[0m[1;95mAxes:[0m[39m [0m[33mxlabel[0m[39m=[0m[32m'step'[0m[1m>[0m
Handling drift¶
The environment used above is a toy situation used for introducing bandits. It is stationary, meaning that the expected reward of each arm does not change over time.
In practice, arms are dynamic, and their performance can vary over time. A simple example of this is the Candy Cane Contest that was hosted on Kaggle in 2020. The expected reward of each arm diminishes each time it is pulled.
The way bandit policies in River deal with drift depends on the method. For the bandit.EpsilonGreedy policy, it makes sense to use a rolling average as the reward object. What this means is that the empirical reward the policy calculates for each arm is a rolling average, rather than a global one.
from river import proba, utils
policies=[
bandit.EpsilonGreedy(
epsilon=0.1,
seed=42
),
bandit.EpsilonGreedy(
epsilon=0.3,
reward_obj=utils.Rolling(stats.Mean(), window_size=50),
seed=42
),
bandit.ThompsonSampling(
reward_obj=proba.Beta(),
seed=42
)
]
env = gym.make('river_bandits/CandyCaneContest-v0')
trace = bandit.evaluate(
policies=policies,
env=env,
n_episodes=(n_episodes := 30),
seed=42
)
trace_df = pd.DataFrame(tqdm(
trace, position=0, total=(
n_episodes *
len(policies) *
env._max_episode_steps
)
))
/tmp/ipykernel_3807/3267528588.py:10: DeprecationWarning: Passing an instance to utils.Rolling is deprecated and will be removed in a future release. Pass the class and forward constructor kwargs instead, e.g. utils.Rolling(stats.Mean, window_size=...) or utils.Rolling(stats.Var, window_size=..., ddof=0).
reward_obj=utils.Rolling(stats.Mean(), window_size=50),
0%| | 0/180000 [00:00<?, ?it/s]
1%| | 1243/180000 [00:00<00:14, 12426.28it/s]
1%|▏ | 2486/180000 [00:00<00:14, 12215.03it/s]
2%|▏ | 3711/180000 [00:00<00:14, 12226.09it/s]
3%|▎ | 4935/180000 [00:00<00:14, 12226.81it/s]
3%|▎ | 6158/180000 [00:00<00:14, 11956.43it/s]
4%|▍ | 7436/180000 [00:00<00:14, 12231.35it/s]
5%|▍ | 8667/180000 [00:00<00:13, 12251.15it/s]
6%|▌ | 9966/180000 [00:00<00:13, 12483.64it/s]
6%|▋ | 11256/180000 [00:00<00:13, 12612.81it/s]
7%|▋ | 12518/180000 [00:01<00:13, 12340.96it/s]
8%|▊ | 13796/180000 [00:01<00:13, 12472.57it/s]
8%|▊ | 15078/180000 [00:01<00:13, 12573.99it/s]
9%|▉ | 16337/180000 [00:01<00:13, 12496.76it/s]
10%|▉ | 17588/180000 [00:01<00:13, 12355.69it/s]
10%|█ | 18825/180000 [00:01<00:13, 12196.30it/s]
11%|█ | 20048/180000 [00:01<00:13, 12204.96it/s]
12%|█▏ | 21342/180000 [00:01<00:12, 12416.01it/s]
13%|█▎ | 22645/180000 [00:01<00:12, 12598.44it/s]
13%|█▎ | 23940/180000 [00:01<00:12, 12698.56it/s]
14%|█▍ | 25211/180000 [00:02<00:12, 12452.74it/s]
15%|█▍ | 26458/180000 [00:02<00:12, 12321.71it/s]
15%|█▌ | 27692/180000 [00:02<00:12, 12215.66it/s]
16%|█▌ | 28915/180000 [00:02<00:12, 12058.81it/s]
17%|█▋ | 30122/180000 [00:02<00:12, 11821.48it/s]
17%|█▋ | 31481/180000 [00:02<00:12, 12334.90it/s]
18%|█▊ | 32717/180000 [00:02<00:11, 12285.32it/s]
19%|█▉ | 33948/180000 [00:02<00:11, 12244.67it/s]
20%|█▉ | 35174/180000 [00:02<00:11, 12193.19it/s]
20%|██ | 36395/180000 [00:02<00:11, 12072.02it/s]
21%|██ | 37635/180000 [00:03<00:11, 12167.20it/s]
22%|██▏ | 38929/180000 [00:03<00:11, 12395.09it/s]
22%|██▏ | 40170/180000 [00:03<00:11, 12290.74it/s]
23%|██▎ | 41407/180000 [00:03<00:11, 12313.62it/s]
24%|██▎ | 42639/180000 [00:03<00:11, 12226.09it/s]
24%|██▍ | 43870/180000 [00:03<00:11, 12249.97it/s]
25%|██▌ | 45096/180000 [00:03<00:11, 12231.94it/s]
26%|██▌ | 46320/180000 [00:03<00:10, 12231.41it/s]
26%|██▋ | 47544/180000 [00:03<00:10, 12172.66it/s]
27%|██▋ | 48762/180000 [00:03<00:10, 12062.70it/s]
28%|██▊ | 49969/180000 [00:04<00:10, 12050.26it/s]
28%|██▊ | 51175/180000 [00:04<00:10, 12045.06it/s]
29%|██▉ | 52380/180000 [00:04<00:10, 12028.23it/s]
30%|██▉ | 53646/180000 [00:04<00:10, 12215.44it/s]
30%|███ | 54868/180000 [00:04<00:10, 12100.52it/s]
31%|███ | 56079/180000 [00:04<00:10, 12070.18it/s]
32%|███▏ | 57287/180000 [00:04<00:10, 12053.20it/s]
32%|███▏ | 58493/180000 [00:04<00:10, 12040.87it/s]
33%|███▎ | 59698/180000 [00:04<00:10, 12014.23it/s]
34%|███▍ | 60900/180000 [00:04<00:09, 11939.51it/s]
35%|███▍ | 62109/180000 [00:05<00:09, 11981.32it/s]
35%|███▌ | 63327/180000 [00:05<00:09, 12034.54it/s]
36%|███▌ | 64540/180000 [00:05<00:09, 12061.79it/s]
37%|███▋ | 65747/180000 [00:05<00:09, 12012.12it/s]
37%|███▋ | 66949/180000 [00:05<00:09, 11939.30it/s]
38%|███▊ | 68157/180000 [00:05<00:09, 11978.97it/s]
39%|███▊ | 69369/180000 [00:05<00:09, 12018.47it/s]
39%|███▉ | 70574/180000 [00:05<00:09, 12027.37it/s]
40%|███▉ | 71777/180000 [00:05<00:09, 12002.98it/s]
41%|████ | 72978/180000 [00:05<00:08, 11911.46it/s]
41%|████ | 74178/180000 [00:06<00:08, 11935.71it/s]
42%|████▏ | 75378/180000 [00:06<00:08, 11949.00it/s]
43%|████▎ | 76575/180000 [00:06<00:08, 11952.08it/s]
43%|████▎ | 77771/180000 [00:06<00:08, 11882.25it/s]
44%|████▍ | 78977/180000 [00:06<00:08, 11935.14it/s]
45%|████▍ | 80182/180000 [00:06<00:08, 11968.04it/s]
45%|████▌ | 81453/180000 [00:06<00:08, 12184.36it/s]
46%|████▌ | 82734/180000 [00:06<00:07, 12367.73it/s]
47%|████▋ | 83971/180000 [00:06<00:07, 12267.96it/s]
47%|████▋ | 85199/180000 [00:06<00:07, 12130.79it/s]
48%|████▊ | 86413/180000 [00:07<00:07, 12096.04it/s]
49%|████▊ | 87689/180000 [00:07<00:07, 12292.17it/s]
49%|████▉ | 88972/180000 [00:07<00:07, 12451.62it/s]
50%|█████ | 90218/180000 [00:07<00:07, 12325.39it/s]
51%|█████ | 91543/180000 [00:07<00:07, 12598.83it/s]
52%|█████▏ | 92804/180000 [00:07<00:06, 12527.69it/s]
52%|█████▏ | 94095/180000 [00:07<00:06, 12636.91it/s]
53%|█████▎ | 95388/180000 [00:07<00:06, 12717.94it/s]
54%|█████▎ | 96661/180000 [00:07<00:06, 12463.47it/s]
54%|█████▍ | 97909/180000 [00:08<00:06, 12355.60it/s]
55%|█████▌ | 99192/180000 [00:08<00:06, 12494.06it/s]
56%|█████▌ | 100479/180000 [00:08<00:06, 12603.10it/s]
57%|█████▋ | 101754/180000 [00:08<00:06, 12643.48it/s]
57%|█████▋ | 103019/180000 [00:08<00:06, 12301.44it/s]
58%|█████▊ | 104296/180000 [00:08<00:06, 12437.50it/s]
59%|█████▊ | 105570/180000 [00:08<00:05, 12517.33it/s]
59%|█████▉ | 106842/180000 [00:08<00:05, 12572.01it/s]
60%|██████ | 108101/180000 [00:08<00:05, 12342.69it/s]
61%|██████ | 109443/180000 [00:08<00:05, 12657.49it/s]
62%|██████▏ | 110719/180000 [00:09<00:05, 12687.30it/s]
62%|██████▏ | 112001/180000 [00:09<00:05, 12726.22it/s]
63%|██████▎ | 113301/180000 [00:09<00:05, 12801.86it/s]
64%|██████▎ | 114582/180000 [00:09<00:05, 12695.48it/s]
64%|██████▍ | 115853/180000 [00:09<00:05, 12618.59it/s]
65%|██████▌ | 117144/180000 [00:09<00:04, 12701.71it/s]
66%|██████▌ | 118434/180000 [00:09<00:04, 12758.59it/s]
67%|██████▋ | 119724/180000 [00:09<00:04, 12800.01it/s]
67%|██████▋ | 121005/180000 [00:09<00:04, 12746.01it/s]
68%|██████▊ | 122313/180000 [00:09<00:04, 12845.38it/s]
69%|██████▊ | 123606/180000 [00:10<00:04, 12870.49it/s]
69%|██████▉ | 124905/180000 [00:10<00:04, 12900.63it/s]
70%|███████ | 126196/180000 [00:10<00:04, 12689.85it/s]
71%|███████ | 127539/180000 [00:10<00:04, 12905.03it/s]
72%|███████▏ | 128831/180000 [00:10<00:04, 12659.89it/s]
72%|███████▏ | 130099/180000 [00:10<00:03, 12504.94it/s]
73%|███████▎ | 131351/180000 [00:10<00:03, 12422.76it/s]
74%|███████▎ | 132595/180000 [00:10<00:03, 12243.88it/s]
74%|███████▍ | 133821/180000 [00:10<00:03, 12222.11it/s]
75%|███████▌ | 135045/180000 [00:10<00:03, 12221.33it/s]
76%|███████▌ | 136268/180000 [00:11<00:03, 12146.81it/s]
76%|███████▋ | 137483/180000 [00:11<00:03, 12132.98it/s]
77%|███████▋ | 138697/180000 [00:11<00:03, 12031.72it/s]
78%|███████▊ | 139920/180000 [00:11<00:03, 12084.76it/s]
78%|███████▊ | 141129/180000 [00:11<00:03, 11821.01it/s]
79%|███████▉ | 142332/180000 [00:11<00:03, 11881.35it/s]
80%|███████▉ | 143541/180000 [00:11<00:03, 11938.91it/s]
80%|████████ | 144736/180000 [00:11<00:02, 11900.81it/s]
81%|████████ | 145962/180000 [00:11<00:02, 12004.90it/s]
82%|████████▏ | 147186/180000 [00:11<00:02, 12074.19it/s]
82%|████████▏ | 148410/180000 [00:12<00:02, 12122.56it/s]
83%|████████▎ | 149626/180000 [00:12<00:02, 12132.32it/s]
84%|████████▍ | 150840/180000 [00:12<00:02, 12033.26it/s]
84%|████████▍ | 152070/180000 [00:12<00:02, 12108.36it/s]
85%|████████▌ | 153296/180000 [00:12<00:02, 12152.81it/s]
86%|████████▌ | 154512/180000 [00:12<00:02, 12122.02it/s]
87%|████████▋ | 155737/180000 [00:12<00:01, 12159.84it/s]
87%|████████▋ | 156954/180000 [00:12<00:01, 12122.30it/s]
88%|████████▊ | 158167/180000 [00:12<00:01, 12052.19it/s]
89%|████████▊ | 159373/180000 [00:12<00:01, 12012.90it/s]
89%|████████▉ | 160575/180000 [00:13<00:01, 11945.60it/s]
90%|████████▉ | 161770/180000 [00:13<00:01, 11926.20it/s]
91%|█████████ | 162963/180000 [00:13<00:01, 11874.50it/s]
91%|█████████ | 164178/180000 [00:13<00:01, 11950.83it/s]
92%|█████████▏| 165388/180000 [00:13<00:01, 11994.33it/s]
93%|█████████▎| 166588/180000 [00:13<00:01, 11995.28it/s]
93%|█████████▎| 167788/180000 [00:13<00:01, 11991.39it/s]
94%|█████████▍| 168988/180000 [00:13<00:00, 11944.37it/s]
95%|█████████▍| 170202/180000 [00:13<00:00, 12000.30it/s]
95%|█████████▌| 171419/180000 [00:13<00:00, 12050.06it/s]
96%|█████████▌| 172625/180000 [00:14<00:00, 12026.09it/s]
97%|█████████▋| 173831/180000 [00:14<00:00, 12035.41it/s]
97%|█████████▋| 175035/180000 [00:14<00:00, 11933.24it/s]
98%|█████████▊| 176241/180000 [00:14<00:00, 11969.98it/s]
99%|█████████▊| 177469/180000 [00:14<00:00, 12062.38it/s]
99%|█████████▉| 178686/180000 [00:14<00:00, 12091.14it/s]
100%|█████████▉| 179907/180000 [00:14<00:00, 12124.08it/s]
100%|██████████| 180000/180000 [00:14<00:00, 12241.02it/s]
We can compare the performance of each policy by checking the average reward at the end of each episode.
(
trace_df
.groupby(['policy_idx', 'episode'])
.last()
.groupby('policy_idx')
.reward_stat.mean()
)
policy_idx
[1;36m0[0m [1;36m736.1[0m
[1;36m1[0m [1;36m817.0[0m
[1;36m2[0m [1;36m854.0[0m
Name: reward_stat, dtype: float64
We see that using a rolling average gives a boost to the epsilon greedy strategy. However, we see that the bandit.ThompsonSampling policy performs even better, even though no particular care was given to drift. A natural next step would thus be to see how it could be improved to handle drift. For instance, its dist parameter could be wrapped with a utils.Rolling:
policy = bandit.ThompsonSampling(
reward_obj=utils.Rolling(proba.Beta(), window_size=50),
seed=42
)
/tmp/ipykernel_3807/502223139.py:2: DeprecationWarning: Passing an instance to utils.Rolling is deprecated and will be removed in a future release. Pass the class and forward constructor kwargs instead, e.g. utils.Rolling(stats.Mean, window_size=...) or utils.Rolling(stats.Var, window_size=..., ddof=0).
reward_obj=utils.Rolling(proba.Beta(), window_size=50),
Bandits can be used for several tasks. They can be used for content personalization, as well as online model selection (see model_selection.BanditRegressor). The policies in River are therefore designed to be flexible, so that they can be used in conjunction with other River modules. For instance, the reward_obj in bandit.EpsilonGreedy can be a metric, a probability distribution, or a statistic. This works because objects in River adher to a coherent get/update interface.