fetch
def fetch(
url:str, name:str
)->pathlib.Path:Download url to the cache as name unless it is already there.
Benedict Thekkel
Most forecasting tutorials change dataset on every page, so their results cannot be compared. This site uses one dataset for every single-series method, and one for every many-series method. Each notebook then adds a row to the same scoreboard, and the gain of each method is a number, not an impression.
| Dataset | Shape | Used for |
|---|---|---|
| Household power (UCI) | 1 house, 1-minute readings, Dec 2006 to Nov 2010, total plus 3 sub-meters | Every single-series forecast, anomaly and change-point notebook |
| M4 Hourly | 414 series, 700 or 960 hourly training points each, 48-step test | Global models, hierarchies, benchmarks |
The household data matches the Phisaver Project brief: total consumption of one home, with sub-meters that show which part of the house drew the power. It is public (UCI, CC BY 4.0), so every number on this site can be reproduced. Phisaver’s own meter data is private and never appears here.
Files download once into ~/.cache/ml_timeseries (override with ML_TIMESERIES_DATA). The cache sits outside the repo, so no data file is ever committed or published.
Download url to the cache as name unless it is already there.
Download cache: $ML_TIMESERIES_DATA, else ~/.cache/ml_timeseries.
The UCI file is a 133 MB semicolon-separated text file inside a 20 MB zip. Missing readings are written as ?. The first parse takes a few seconds, so the result is cached as Parquet.
| Column | Unit | Meaning |
|---|---|---|
active_kw |
kW | Whole-house active power, averaged over the minute |
reactive_kw |
kW | Reactive power |
voltage |
V | Mains voltage |
current_a |
A | Current |
sub1_wh |
Wh | Kitchen: dishwasher, oven, microwave |
sub2_wh |
Wh | Laundry: washer, dryer, fridge, light |
sub3_wh |
Wh | Water heater and air conditioner |
The sub-meters do not cover the whole house. The rest (lighting, sockets, everything else) is the difference between the total and the three sub-meters.
One-minute household readings indexed by local timestamp (float32, NaN where missing).
2,075,259 minutes from 2006-12-16 17:24:00 to 2010-11-26 21:02:00
missing readings: 1.25%
| ts | 17:24 | 17:25 | 17:26 |
|---|---|---|---|
| active_kw | 4.216 | 5.360 | 5.374 |
| reactive_kw | 0.418 | 0.436 | 0.498 |
| voltage | 234.840 | 233.630 | 233.290 |
| current_a | 18.400 | 23.000 | 23.000 |
| sub1_wh | 0.000 | 0.000 | 0.000 |
| sub2_wh | 1.000 | 1.000 | 2.000 |
| sub3_wh | 17.000 | 16.000 | 17.000 |
Forecasting works on energy per period, not instantaneous power. load_household resamples the minutes into energy in kWh for each period:
A period with fewer than 75% of its minutes present is set to missing rather than averaged from a fragment. An hour with 5 readings would otherwise look like a full hour.
Energy in kWh per freq period; periods with too few minutes present become NaN.
Most models cannot take missing values, so the loader can fill them, and it marks every filled period in a boolean filled column. The backtest uses that flag to score forecasts only against hours the meter actually recorded. The fill rule depends on the gap length:
The Data Preparation notebook compares these choices. Here they are only the default.
Interpolate gaps of at most max_interp steps; fill longer gaps from season steps earlier (or later).
Household energy in kWh per freq period. clean=True fills every gap (see fill_gaps) and adds a filled flag.
hourly_raw = load_household(clean=False)
hourly = load_household()
gaps = hourly_raw["kwh"].isna()
run = (gaps != gaps.shift()).cumsum()
lengths = gaps.groupby(run).sum()
lengths = lengths[lengths > 0]
print(f"{len(hourly):,} hours, {gaps.sum()} missing ({gaps.mean():.2%}) in {len(lengths)} gaps")
print(f"longest gap: {lengths.max()} hours; gaps of 1-3 hours: {(lengths <= 3).sum()}")
hourly.drop(columns="filled").describe().T[["mean", "std", "min", "max"]].round(3)34,589 hours, 438 missing (1.27%) in 16 gaps
longest gap: 121 hours; gaps of 1-3 hours: 9
| mean | std | min | max | |
|---|---|---|---|---|
| kwh | 1.089 | 0.897 | 0.124 | 6.561 |
| sub1_kwh | 0.067 | 0.212 | 0.000 | 2.902 |
| sub2_kwh | 0.078 | 0.251 | 0.000 | 2.786 |
| sub3_kwh | 0.387 | 0.441 | 0.000 | 1.293 |
| other_kwh | 0.557 | 0.495 | 0.083 | 3.890 |
| voltage | 240.846 | 2.995 | 225.835 | 251.902 |
line_chart, bar_chart and show are the chart helpers the site uses. line_chart draws pyecharts lines with the pinned asset host from the repo conventions and a zoom slider, sized to the page width. show renders it with the chart data compacted, which keeps a notebook with several charts under a few hundred KB.
Render a pyecharts chart inline with its JSON compacted (pyecharts pretty-prints one value per line).
pyecharts bar chart of series (name -> values) per label, sized to the page width.
pyecharts line chart of series (name -> values) against x, sized to the page width.
The winter peaks and the August holidays (consumption drops to almost nothing in August 2008) are visible at once. Both matter for every model that follows: the first is annual seasonality, the second is the kind of event no model can predict from the history alone.
At hourly resolution the daily cycle dominates: a low night, a morning peak and a larger evening peak. Weekends look different from weekdays. These are the two seasonal periods (24 and 168 hours) that the classical models must capture.
The M4 competition (2018) is the standard many-series benchmark. Its hourly subset has 414 series of different scales and patterns, with a fixed 48-hour test horizon. The files use the long format that the Nixtla libraries expect: one row per (unique_id, ds, y), where ds is an integer step.
One series as a long-format (unique_id, ds, y) frame.
M4 Hourly as long-format (unique_id, ds, y) frames: (train, test).
414 series; train lengths [700, 960]; test 48 steps each
| unique_id | ds | y | |
|---|---|---|---|
| 0 | H1 | 1 | 605.0 |
| 1 | H1 | 2 | 586.0 |
| 2 | H1 | 3 | 586.0 |