Debugging a pipeline¶
river encourages users to make use of pipelines. The biggest pain point of pipelines is that it can be hard to understand what's happening to the data, especially when the pipeline is complex. Fortunately the Pipeline class has a debug_one method that can help out.
Let's look at a fairly complex pipeline for predicting the number of bikes in 5 bike stations from the city of Toulouse. It doesn't matter if you understand the pipeline or not; the point of this notebook is to learn how to introspect a pipeline.
import datetime as dt
from river import compose
from river import datasets
from river import feature_extraction
from river import linear_model
from river import metrics
from river import preprocessing
from river import stats
from river import stream
X_y = datasets.Bikes()
X_y = stream.simulate_qa(X_y, moment='moment', delay=dt.timedelta(minutes=30))
def add_time_features(x):
return {
**x,
'hour': x['moment'].hour,
'day': x['moment'].weekday()
}
model = add_time_features
model |= (
compose.Select('clouds', 'humidity', 'pressure', 'temperature', 'wind') +
feature_extraction.TargetAgg(by=['station', 'hour'], how=stats.Mean()) +
feature_extraction.TargetAgg(by='station', how=stats.EWMean())
)
model |= preprocessing.StandardScaler()
model |= linear_model.LinearRegression()
metric = metrics.MAE()
questions = {}
for i, x, y in X_y:
# Question
is_question = y is None
if is_question:
y_pred = model.predict_one(x)
questions[i] = y_pred
# Answer
else:
metric.update(y, questions[i])
model = model.learn_one(x, y)
if i >= 30000 and i % 30000 == 0:
print(i, metric)
30000 MAE: 2.220942
60000 MAE: 2.270271
90000 MAE: 2.301302
120000 MAE: 2.275876
150000 MAE: 2.275224
180000 MAE: 2.289347
Let's start by looking at the pipeline. You can click each cell to display the current state for each step of the pipeline.
model
add_time_features
def add_time_features(x):
return {
**x,
'hour': x['moment'].hour,
'day': x['moment'].weekday()
}
['clouds', 'humidity', 'pressure', 'temperature', 'wind']
{'whitelist': {'pressure', 'temperature', 'wind', 'clouds', 'humidity'}}
target_mean_by_station_and_hour
{'by': ['station', 'hour'],
'feature_name': 'target_mean_by_station_and_hour',
'groups': defaultdict(functools.partial(<function deepcopy at 0x7efc22241310>, Mean: 0.),
{'metro-canal-du-midi_0': Mean: 7.93981,
'metro-canal-du-midi_1': Mean: 8.179704,
'metro-canal-du-midi_10': Mean: 12.486815,
'metro-canal-du-midi_11': Mean: 11.675479,
'metro-canal-du-midi_12': Mean: 10.197409,
'metro-canal-du-midi_13': Mean: 10.650855,
'metro-canal-du-midi_14': Mean: 11.109123,
'metro-canal-du-midi_15': Mean: 11.068934,
'metro-canal-du-midi_16': Mean: 11.274958,
'metro-canal-du-midi_17': Mean: 8.459136,
'metro-canal-du-midi_18': Mean: 7.587469,
'metro-canal-du-midi_19': Mean: 7.734677,
'metro-canal-du-midi_2': Mean: 8.35824,
'metro-canal-du-midi_20': Mean: 7.582465,
'metro-canal-du-midi_21': Mean: 7.190665,
'metro-canal-du-midi_22': Mean: 7.486895,
'metro-canal-du-midi_23': Mean: 7.840791,
'metro-canal-du-midi_3': Mean: 8.656051,
'metro-canal-du-midi_4': Mean: 8.868445,
'metro-canal-du-midi_5': Mean: 8.99656,
'metro-canal-du-midi_6': Mean: 9.09966,
'metro-canal-du-midi_7': Mean: 8.852642,
'metro-canal-du-midi_8': Mean: 12.66712,
'metro-canal-du-midi_9': Mean: 13.412186,
'place-des-carmes_0': Mean: 4.720696,
'place-des-carmes_1': Mean: 3.390295,
'place-des-carmes_10': Mean: 8.575303,
'place-des-carmes_11': Mean: 9.321546,
'place-des-carmes_12': Mean: 10.511931,
'place-des-carmes_13': Mean: 11.392745,
'place-des-carmes_14': Mean: 10.735003,
'place-des-carmes_15': Mean: 10.198787,
'place-des-carmes_16': Mean: 9.941479,
'place-des-carmes_17': Mean: 9.125579,
'place-des-carmes_18': Mean: 7.660775,
'place-des-carmes_19': Mean: 6.847649,
'place-des-carmes_2': Mean: 2.232181,
'place-des-carmes_20': Mean: 9.626876,
'place-des-carmes_21': Mean: 11.602929,
'place-des-carmes_22': Mean: 10.405537,
'place-des-carmes_23': Mean: 7.700904,
'place-des-carmes_3': Mean: 1.371981,
'place-des-carmes_4': Mean: 1.051665,
'place-des-carmes_5': Mean: 0.984993,
'place-des-carmes_6': Mean: 2.039947,
'place-des-carmes_7': Mean: 3.850369,
'place-des-carmes_8': Mean: 3.792624,
'place-des-carmes_9': Mean: 5.957182,
'place-esquirol_0': Mean: 7.415789,
'place-esquirol_1': Mean: 5.244396,
'place-esquirol_10': Mean: 19.465005,
'place-esquirol_11': Mean: 22.976512,
'place-esquirol_12': Mean: 25.324159,
'place-esquirol_13': Mean: 25.428847,
'place-esquirol_14': Mean: 24.57762,
'place-esquirol_15': Mean: 24.416851,
'place-esquirol_16': Mean: 23.555125,
'place-esquirol_17': Mean: 22.062564,
'place-esquirol_18': Mean: 18.10623,
'place-esquirol_19': Mean: 11.916638,
'place-esquirol_2': Mean: 2.858635,
'place-esquirol_20': Mean: 13.346362,
'place-esquirol_21': Mean: 16.743318,
'place-esquirol_22': Mean: 15.562088,
'place-esquirol_23': Mean: 10.911134,
'place-esquirol_3': Mean: 1.155929,
'place-esquirol_4': Mean: 0.73306,
'place-esquirol_5': Mean: 0.668546,
'place-esquirol_6': Mean: 1.21265,
'place-esquirol_7': Mean: 3.107535,
'place-esquirol_8': Mean: 8.518696,
'place-esquirol_9': Mean: 15.470588,
'place-jeanne-darc_0': Mean: 6.541667,
'place-jeanne-darc_1': Mean: 5.99892,
'place-jeanne-darc_10': Mean: 5.735553,
'place-jeanne-darc_11': Mean: 5.616142,
'place-jeanne-darc_12': Mean: 5.787478,
'place-jeanne-darc_13': Mean: 5.817699,
'place-jeanne-darc_14': Mean: 5.657546,
'place-jeanne-darc_15': Mean: 6.224604,
'place-jeanne-darc_16': Mean: 5.796141,
'place-jeanne-darc_17': Mean: 5.743089,
'place-jeanne-darc_18': Mean: 5.674784,
'place-jeanne-darc_19': Mean: 5.833068,
'place-jeanne-darc_2': Mean: 5.598169,
'place-jeanne-darc_20': Mean: 6.015755,
'place-jeanne-darc_21': Mean: 6.242541,
'place-jeanne-darc_22': Mean: 6.141509,
'place-jeanne-darc_23': Mean: 6.493028,
'place-jeanne-darc_3': Mean: 5.180556,
'place-jeanne-darc_4': Mean: 4.779626,
'place-jeanne-darc_5': Mean: 4.67063,
'place-jeanne-darc_6': Mean: 4.611995,
'place-jeanne-darc_7': Mean: 4.960718,
'place-jeanne-darc_8': Mean: 5.552273,
'place-jeanne-darc_9': Mean: 6.249573,
'pomme_0': Mean: 3.301532,
'pomme_1': Mean: 2.312914,
'pomme_10': Mean: 11.791436,
'pomme_11': Mean: 12.976854,
'pomme_12': Mean: 13.962654,
'pomme_13': Mean: 11.692257,
'pomme_14': Mean: 11.180851,
'pomme_15': Mean: 11.939586,
'pomme_16': Mean: 12.267051,
'pomme_17': Mean: 12.132993,
'pomme_18': Mean: 11.399108,
'pomme_19': Mean: 6.37021,
'pomme_2': Mean: 2.144453,
'pomme_20': Mean: 5.279234,
'pomme_21': Mean: 6.254257,
'pomme_22': Mean: 6.568678,
'pomme_23': Mean: 5.235756,
'pomme_3': Mean: 1.563622,
'pomme_4': Mean: 0.947328,
'pomme_5': Mean: 0.924175,
'pomme_6': Mean: 1.287805,
'pomme_7': Mean: 1.299456,
'pomme_8': Mean: 2.94988,
'pomme_9': Mean: 7.89396}),
'how': Mean: 0.,
'target_name': 'target'}
target_ewm_0.5_by_station
{'by': ['station'],
'feature_name': 'target_ewm_0.5_by_station',
'groups': defaultdict(functools.partial(<function deepcopy at 0x7efc22241310>, EWMean: 0.),
{'metro-canal-du-midi': EWMean: 4.690531,
'place-des-carmes': EWMean: 3.295317,
'place-esquirol': EWMean: 31.539759,
'place-jeanne-darc': EWMean: 22.449934,
'pomme': EWMean: 11.803716}),
'how': EWMean: 0.,
'target_name': 'target'}
StandardScaler
{'counts': Counter({'target_ewm_0.5_by_station': 182470,
'target_mean_by_station_and_hour': 182470,
'pressure': 182470,
'temperature': 182470,
'wind': 182470,
'clouds': 182470,
'humidity': 182470}),
'means': defaultdict(<class 'float'>,
{'clouds': 30.315131254453505,
'humidity': 62.24244533347998,
'pressure': 1017.0563060996391,
'target_ewm_0.5_by_station': 10.08331958752748,
'target_mean_by_station_and_hour': 9.410348580619415,
'temperature': 20.50980692716619,
'wind': 3.4184331122924543}),
'vars': defaultdict(<class 'float'>,
{'clouds': 1389.0025610928221,
'humidity': 349.59967918503554,
'pressure': 33.298307526514115,
'target_ewm_0.5_by_station': 80.17355266024735,
'target_mean_by_station_and_hour': 33.98249801051089,
'temperature': 34.70701720774977,
'wind': 4.473627075744674})}
LinearRegression
{'_weights': {'target_ewm_0.5_by_station': 9.264175276315456, 'target_mean_by_station_and_hour': 0.1980140007049765, 'pressure': 0.18137498909137667, 'temperature': -0.42112178062191846, 'wind': -0.04087954775179744, 'clouds': -0.32696947944582777, 'humidity': 1.0125248437612913},
'_y_name': None,
'clip_gradient': 1000000000000.0,
'initializer': Zeros (),
'intercept': 9.223158690689173,
'intercept_init': 0.0,
'intercept_lr': Constant({'learning_rate': 0.01}),
'l2': 0.0,
'loss': Squared({}),
'optimizer': SGD({'lr': Constant({'learning_rate': 0.01}), 'n_iterations': 182470})}
As mentioned above the Pipeline class has a debug_one method. You can use this at any point you want to visualize what happen to an input x. For example, let's see what happens to the last seen x.
print(model.debug_one(x))
0. Input
--------
clouds: 88 (int)
description: overcast clouds (str)
humidity: 84 (int)
moment: 2016-10-05 09:57:18 (datetime)
pressure: 1,017.34000 (float)
station: pomme (str)
temperature: 17.45000 (float)
wind: 1.95000 (float)
1. add_time_features
--------------------
clouds: 88 (int)
day: 2 (int)
description: overcast clouds (str)
hour: 9 (int)
humidity: 84 (int)
moment: 2016-10-05 09:57:18 (datetime)
pressure: 1,017.34000 (float)
station: pomme (str)
temperature: 17.45000 (float)
wind: 1.95000 (float)
2. Transformer union
--------------------
2.0 Select
----------
clouds: 88 (int)
humidity: 84 (int)
pressure: 1,017.34000 (float)
temperature: 17.45000 (float)
wind: 1.95000 (float)
2.1 TargetAgg
-------------
target_mean_by_station_and_hour: 7.89396 (float)
2.2 TargetAgg1
--------------
target_ewm_0.5_by_station: 11.80372 (float)
clouds: 88 (int)
humidity: 84 (int)
pressure: 1,017.34000 (float)
target_ewm_0.5_by_station: 11.80372 (float)
target_mean_by_station_and_hour: 7.89396 (float)
temperature: 17.45000 (float)
wind: 1.95000 (float)
3. StandardScaler
-----------------
clouds: 1.54778 (float)
humidity: 1.16366 (float)
pressure: 0.04916 (float)
target_ewm_0.5_by_station: 0.19214 (float)
target_mean_by_station_and_hour: -0.26013 (float)
temperature: -0.51938 (float)
wind: -0.69426 (float)
4. LinearRegression
-------------------
Name Value Weight Contribution
Intercept 1.00000 9.22316 9.22316
target_ewm_0.5_by_station 0.19214 9.26418 1.78000
humidity 1.16366 1.01252 1.17823
temperature -0.51938 -0.42112 0.21872
wind -0.69426 -0.04088 0.02838
pressure 0.04916 0.18137 0.00892
target_mean_by_station_and_hour -0.26013 0.19801 -0.05151
clouds 1.54778 -0.32697 -0.50608
Prediction: 11.87982
The pipeline does quite a few things, but using debug_one shows what happens step by step. This is really useful for checking that the pipeline is behaving as you're expecting it too. Remember that you can debug_one whenever you wish, be it before, during, or after training a model.