Bike-sharing forecasting¶

%load_ext autoreload
%autoreload 2

In this tutorial we're going to forecast the number of bikes in 5 bike stations from the city of Toulouse. We'll do so by building a simple model step by step. The dataset contains 182,470 observations. Let's first take a peak at the data.

from pprint import pprint
from river import datasets

dataset = datasets.Bikes()

for x, y in dataset:
    pprint(x)
    print(f'Number of available bikes: {y}')
    break

Downloading https://maxhalford.github.io/files/datasets/toulouse_bikes.zip (1.12 MB)
Uncompressing into /home/runner/river_data/Bikes
{'clouds': 75,
 'description': 'light rain',
 'humidity': 81,
 'moment': datetime.datetime(2016, 4, 1, 0, 0, 7),
 'pressure': 1017.0,
 'station': 'metro-canal-du-midi',
 'temperature': 6.54,
 'wind': 9.3}
Number of available bikes: 1

Let's start by using a simple linear regression on the numeric features. We can select the numeric features and discard the rest of the features using a Select. Linear regression is very likely to go haywire if we don't scale the data, so we'll use a StandardScaler to do just that. We'll evaluate the model by measuring the mean absolute error. Finally we'll print the score every 20,000 observations.

from river import compose
from river import linear_model
from river import metrics
from river import evaluate
from river import preprocessing
from river import optim

model = compose.Select('clouds', 'humidity', 'pressure', 'temperature', 'wind')
model |= preprocessing.StandardScaler()
model |= linear_model.LinearRegression(optimizer=optim.SGD(0.001))

metric = metrics.MAE()

evaluate.progressive_val_score(dataset, model, metric, print_every=20_000)

[20,000] MAE: 4.912727
[40,000] MAE: 5.333554
[60,000] MAE: 5.330948
[80,000] MAE: 5.392313
[100,000] MAE: 5.423059
[120,000] MAE: 5.541223
[140,000] MAE: 5.613023
[160,000] MAE: 5.622428
[180,000] MAE: 5.567824





MAE: 5.563893

The model doesn't seem to be doing that well, but then again we didn't provide a lot of features. Generally, a good idea for this kind of problem is to look at an average of the previous values. For example, for each station we can look at the average number of bikes per hour. To do so we first have to extract the hour from the moment field. We can then use a TargetAgg to aggregate the values of the target.

from river import feature_extraction
from river import stats

def get_hour(x):
    x['hour'] = x['moment'].hour
    return x

model = compose.Select('clouds', 'humidity', 'pressure', 'temperature', 'wind')
model += (
    get_hour |
    feature_extraction.TargetAgg(by=['station', 'hour'], how=stats.Mean())
)
model |= preprocessing.StandardScaler()
model |= linear_model.LinearRegression(optimizer=optim.SGD(0.001))

metric = metrics.MAE()

evaluate.progressive_val_score(dataset, model, metric, print_every=20_000)

[20,000] MAE: 3.721246
[40,000] MAE: 3.829972
[60,000] MAE: 3.845068
[80,000] MAE: 3.910259
[100,000] MAE: 3.888652
[120,000] MAE: 3.923727
[140,000] MAE: 3.980953
[160,000] MAE: 3.950034
[180,000] MAE: 3.934545





MAE: 3.933498

By adding a single feature, we've managed to significantly reduce the mean absolute error. At this point you might think that the model is getting slightly complex, and is difficult to understand and test. Pipelines have the advantage of being terse, but they aren't always to debug. Thankfully river has some ways to relieve the pain.

The first thing we can do it to visualize the pipeline, to get an idea of how the data flows through it.

model

['clouds', 'humidity', 'pressure', 'temperature', 'wind']


{'keys': {'temperature', 'pressure', 'wind', 'clouds', 'humidity'}}

get_hour


def get_hour(x):
    x['hour'] = x['moment'].hour
    return x

y_mean_by_station_and_hour


{'_feature_name': 'y_mean_by_station_and_hour',
 '_groups': defaultdict(functools.partial(<function deepcopy at 0x7fdd64ba4af0>, Mean: 0.),
                        {('metro-canal-du-midi', 0): Mean: 7.93981,
                         ('metro-canal-du-midi', 1): Mean: 8.179704,
                         ('metro-canal-du-midi', 2): Mean: 8.35824,
                         ('metro-canal-du-midi', 3): Mean: 8.656051,
                         ('metro-canal-du-midi', 4): Mean: 8.868445,
                         ('metro-canal-du-midi', 5): Mean: 8.99656,
                         ('metro-canal-du-midi', 6): Mean: 9.09966,
                         ('metro-canal-du-midi', 7): Mean: 8.852642,
                         ('metro-canal-du-midi', 8): Mean: 12.66712,
                         ('metro-canal-du-midi', 9): Mean: 13.412186,
                         ('metro-canal-du-midi', 10): Mean: 12.486815,
                         ('metro-canal-du-midi', 11): Mean: 11.675479,
                         ('metro-canal-du-midi', 12): Mean: 10.197409,
                         ('metro-canal-du-midi', 13): Mean: 10.650855,
                         ('metro-canal-du-midi', 14): Mean: 11.109123,
                         ('metro-canal-du-midi', 15): Mean: 11.068934,
                         ('metro-canal-du-midi', 16): Mean: 11.274958,
                         ('metro-canal-du-midi', 17): Mean: 8.459136,
                         ('metro-canal-du-midi', 18): Mean: 7.587469,
                         ('metro-canal-du-midi', 19): Mean: 7.734677,
                         ('metro-canal-du-midi', 20): Mean: 7.582465,
                         ('metro-canal-du-midi', 21): Mean: 7.190665,
                         ('metro-canal-du-midi', 22): Mean: 7.486895,
                         ('metro-canal-du-midi', 23): Mean: 7.840791,
                         ('place-des-carmes', 0): Mean: 4.720696,
                         ('place-des-carmes', 1): Mean: 3.390295,
                         ('place-des-carmes', 2): Mean: 2.232181,
                         ('place-des-carmes', 3): Mean: 1.371981,
                         ('place-des-carmes', 4): Mean: 1.051665,
                         ('place-des-carmes', 5): Mean: 0.984993,
                         ('place-des-carmes', 6): Mean: 2.039947,
                         ('place-des-carmes', 7): Mean: 3.850369,
                         ('place-des-carmes', 8): Mean: 3.792624,
                         ('place-des-carmes', 9): Mean: 5.957182,
                         ('place-des-carmes', 10): Mean: 8.575303,
                         ('place-des-carmes', 11): Mean: 9.321546,
                         ('place-des-carmes', 12): Mean: 10.511931,
                         ('place-des-carmes', 13): Mean: 11.392745,
                         ('place-des-carmes', 14): Mean: 10.735003,
                         ('place-des-carmes', 15): Mean: 10.198787,
                         ('place-des-carmes', 16): Mean: 9.941479,
                         ('place-des-carmes', 17): Mean: 9.125579,
                         ('place-des-carmes', 18): Mean: 7.660775,
                         ('place-des-carmes', 19): Mean: 6.847649,
                         ('place-des-carmes', 20): Mean: 9.626876,
                         ('place-des-carmes', 21): Mean: 11.602929,
                         ('place-des-carmes', 22): Mean: 10.405537,
                         ('place-des-carmes', 23): Mean: 7.700904,
                         ('place-esquirol', 0): Mean: 7.415789,
                         ('place-esquirol', 1): Mean: 5.244396,
                         ('place-esquirol', 2): Mean: 2.858635,
                         ('place-esquirol', 3): Mean: 1.155929,
                         ('place-esquirol', 4): Mean: 0.73306,
                         ('place-esquirol', 5): Mean: 0.668546,
                         ('place-esquirol', 6): Mean: 1.21265,
                         ('place-esquirol', 7): Mean: 3.107535,
                         ('place-esquirol', 8): Mean: 8.518696,
                         ('place-esquirol', 9): Mean: 15.470588,
                         ('place-esquirol', 10): Mean: 19.465005,
                         ('place-esquirol', 11): Mean: 22.976512,
                         ('place-esquirol', 12): Mean: 25.324159,
                         ('place-esquirol', 13): Mean: 25.428847,
                         ('place-esquirol', 14): Mean: 24.57762,
                         ('place-esquirol', 15): Mean: 24.416851,
                         ('place-esquirol', 16): Mean: 23.555125,
                         ('place-esquirol', 17): Mean: 22.062564,
                         ('place-esquirol', 18): Mean: 18.10623,
                         ('place-esquirol', 19): Mean: 11.916638,
                         ('place-esquirol', 20): Mean: 13.346362,
                         ('place-esquirol', 21): Mean: 16.743318,
                         ('place-esquirol', 22): Mean: 15.562088,
                         ('place-esquirol', 23): Mean: 10.911134,
                         ('place-jeanne-darc', 0): Mean: 6.541667,
                         ('place-jeanne-darc', 1): Mean: 5.99892,
                         ('place-jeanne-darc', 2): Mean: 5.598169,
                         ('place-jeanne-darc', 3): Mean: 5.180556,
                         ('place-jeanne-darc', 4): Mean: 4.779626,
                         ('place-jeanne-darc', 5): Mean: 4.67063,
                         ('place-jeanne-darc', 6): Mean: 4.611995,
                         ('place-jeanne-darc', 7): Mean: 4.960718,
                         ('place-jeanne-darc', 8): Mean: 5.552273,
                         ('place-jeanne-darc', 9): Mean: 6.249573,
                         ('place-jeanne-darc', 10): Mean: 5.735553,
                         ('place-jeanne-darc', 11): Mean: 5.616142,
                         ('place-jeanne-darc', 12): Mean: 5.787478,
                         ('place-jeanne-darc', 13): Mean: 5.817699,
                         ('place-jeanne-darc', 14): Mean: 5.657546,
                         ('place-jeanne-darc', 15): Mean: 6.224604,
                         ('place-jeanne-darc', 16): Mean: 5.796141,
                         ('place-jeanne-darc', 17): Mean: 5.743089,
                         ('place-jeanne-darc', 18): Mean: 5.674784,
                         ('place-jeanne-darc', 19): Mean: 5.833068,
                         ('place-jeanne-darc', 20): Mean: 6.015755,
                         ('place-jeanne-darc', 21): Mean: 6.242541,
                         ('place-jeanne-darc', 22): Mean: 6.141509,
                         ('place-jeanne-darc', 23): Mean: 6.493028,
                         ('pomme', 0): Mean: 3.301532,
                         ('pomme', 1): Mean: 2.312914,
                         ('pomme', 2): Mean: 2.144453,
                         ('pomme', 3): Mean: 1.563622,
                         ('pomme', 4): Mean: 0.947328,
                         ('pomme', 5): Mean: 0.924175,
                         ('pomme', 6): Mean: 1.287805,
                         ('pomme', 7): Mean: 1.299456,
                         ('pomme', 8): Mean: 2.94988,
                         ('pomme', 9): Mean: 7.89396,
                         ('pomme', 10): Mean: 11.791436,
                         ('pomme', 11): Mean: 12.976854,
                         ('pomme', 12): Mean: 13.962654,
                         ('pomme', 13): Mean: 11.692257,
                         ('pomme', 14): Mean: 11.180851,
                         ('pomme', 15): Mean: 11.939586,
                         ('pomme', 16): Mean: 12.267051,
                         ('pomme', 17): Mean: 12.132993,
                         ('pomme', 18): Mean: 11.399108,
                         ('pomme', 19): Mean: 6.37021,
                         ('pomme', 20): Mean: 5.279234,
                         ('pomme', 21): Mean: 6.254257,
                         ('pomme', 22): Mean: 6.568678,
                         ('pomme', 23): Mean: 5.235756}),
 'by': ['station', 'hour'],
 'how': Mean: 0.,
 'on': 'y'}

StandardScaler


{'counts': Counter({'y_mean_by_station_and_hour': 182470,
                    'temperature': 182470,
                    'pressure': 182470,
                    'wind': 182470,
                    'clouds': 182470,
                    'humidity': 182470}),
 'means': defaultdict(<class 'float'>,
                      {'clouds': 30.315131254453505,
                       'humidity': 62.24244533347998,
                       'pressure': 1017.0563060996391,
                       'temperature': 20.50980692716619,
                       'wind': 3.4184331122924543,
                       'y_mean_by_station_and_hour': 9.468200635816528}),
 'vars': defaultdict(<class 'float'>,
                     {'clouds': 1389.0025610928221,
                      'humidity': 349.59967918503554,
                      'pressure': 33.298307526514115,
                      'temperature': 34.70701720774977,
                      'wind': 4.473627075744674,
                      'y_mean_by_station_and_hour': 33.720872727055365}),
 'with_std': True}

LinearRegression


{'_weights': {'y_mean_by_station_and_hour': 3.8719218238428725, 'temperature': -2.7950777648287275, 'pressure': 2.148194846809031, 'wind': -0.26121936045271593, 'clouds': -0.6106470619450337, 'humidity': 3.881792987494523},
 '_y_name': None,
 'clip_gradient': 1000000000000.0,
 'initializer': Zeros (),
 'intercept': 6.096564954881427,
 'intercept_init': 0.0,
 'intercept_lr': Constant({'learning_rate': 0.01}),
 'l1': 0.0,
 'l2': 0.0,
 'loss': Squared({}),
 'optimizer': SGD({'lr': Constant({'learning_rate': 0.001}), 'n_iterations': 182470})}

We can also use the debug_one method to see what happens to one particular instance. Let's train the model on the first 10,000 observations and then call debug_one on the next one. To do this, we will turn the Bike object into a Python generator with iter() function. The Pythonic way to read the first 10,000 elements of a generator is to use itertools.islice.

import itertools

model = compose.Select('clouds', 'humidity', 'pressure', 'temperature', 'wind')
model += (
    get_hour |
    feature_extraction.TargetAgg(by=['station', 'hour'], how=stats.Mean())
)
model |= preprocessing.StandardScaler()
model |= linear_model.LinearRegression()

for x, y in itertools.islice(dataset, 10000):
    y_pred = model.predict_one(x)
    model.learn_one(x, y)

x, y = next(iter(dataset))
print(model.debug_one(x))

0. Input
--------
clouds: 75 (int)
description: light rain (str)
humidity: 81 (int)
moment: 2016-04-01 00:00:07 (datetime)
pressure: 1,017.00000 (float)
station: metro-canal-du-midi (str)
temperature: 6.54000 (float)
wind: 9.30000 (float)

1. Transformer union
--------------------
    1.0 Select
    ----------
    clouds: 75 (int)
    humidity: 81 (int)
    pressure: 1,017.00000 (float)
    temperature: 6.54000 (float)
    wind: 9.30000 (float)

    1.1 get_hour | y_mean_by_station_and_hour
    -----------------------------------------
    y_mean_by_station_and_hour: 4.43243 (float)

clouds: 75 (int)
humidity: 81 (int)
pressure: 1,017.00000 (float)
temperature: 6.54000 (float)
wind: 9.30000 (float)
y_mean_by_station_and_hour: 4.43243 (float)

2. StandardScaler
-----------------
clouds: 0.47566 (float)
humidity: 0.42247 (float)
pressure: 1.05314 (float)
temperature: -1.22098 (float)
wind: 2.21104 (float)
y_mean_by_station_and_hour: -0.59098 (float)

3. LinearRegression
-------------------
Name                         Value      Weight     Contribution  
                 Intercept    1.00000    6.58252        6.58252  
                  pressure    1.05314    3.78529        3.98646  
                  humidity    0.42247    1.44921        0.61225  
y_mean_by_station_and_hour   -0.59098    0.54167       -0.32011  
                    clouds    0.47566   -1.92255       -0.91448  
                      wind    2.21104   -0.77720       -1.71843  
               temperature   -1.22098    2.47030       -3.01619

Prediction: 5.21201

The debug_one method shows what happens to an input set of features, step by step.

And now comes the catch. Up until now we've been using the progressive_val_score method from the evaluate module. What this does it that it sequentially predicts the output of an observation and updates the model immediately afterwards. This way of proceeding is often used for evaluating online learning models. But in some cases it is the wrong approach.

When evaluating a machine learning model, the goal is to simulate production conditions in order to get a trust-worthy assessment of the performance of the model. In our case, we typically want to forecast the number of bikes available in a station, say, 30 minutes ahead. Then, once the 30 minutes have passed, the true number of available bikes will be available and we will be able to update the model using the features available 30 minutes ago.

What we really want is to evaluate the model by forecasting 30 minutes ahead and only updating the model once the true values are available. This can be done using the moment and delay parameters in the progressive_val_score method. The idea is that each observation in the stream of the data is shown twice to the model: once for making a prediction, and once for updating the model when the true value is revealed. The moment parameter determines which variable should be used as a timestamp, while the delay parameter controls the duration to wait before revealing the true values to the model.

import datetime as dt

evaluate.progressive_val_score(
    dataset=dataset,
    model=model.clone(),
    metric=metrics.MAE(),
    moment='moment',
    delay=dt.timedelta(minutes=30),
    print_every=20_000
)

[20,000] MAE: 4.203433
[40,000] MAE: 4.195404
[60,000] MAE: 4.130316
[80,000] MAE: 4.122796
[100,000] MAE: 4.069826
[120,000] MAE: 4.066034
[140,000] MAE: 4.088604
[160,000] MAE: 4.059282
[180,000] MAE: 4.026821





MAE: 4.024939

The performance is a bit worse, which is to be expected. Indeed, the task is more difficult: the model is only shown the ground truth 30 minutes after making a prediction.

The takeaway of this notebook is that the progressive_val_score method can be used to simulate a production scenario, and is thus extremely valuable.