Foundation models without a GPU: Chronos-Bolt tiny and small, Chronos-2 and TiRex on the CPU, with the latency and accuracy a meter gateway or a cheap server would see.
Author
Benedict Thekkel
Why CPU matters
A forecast that runs once an hour per meter does not need a GPU, and most places a forecast is needed (an edge gateway, a small cloud function, a home server) do not have one. The question is how much accuracy survives on a CPU, and how long each forecast takes.
Chronos-Bolt (Amazon, 2024) is built for this: it forecasts all quantiles in one forward pass instead of sampling, and comes in sizes from 9 million parameters (tiny) to 205 million (base). Two other small CPU candidates named in this site’s plan could not be tested: IBM’s TTM (Tiny Time Mixers) would downgrade the shared environment’s PyTorch, and TabPFN-TS does not resolve against it at all.
Everything here runs on the CPU only, with the GPU hidden. The page still carries skip_exec, because it imports the vendor packages that only the root environment has.
import gcimport loggingimport osimport timeimport warningsos.environ["CUDA_VISIBLE_DEVICES"] =""# force CPU for every model on this pageimport numpy as npimport pandas as pdimport tireximport torchfrom chronos import BaseChronosPipeline, Chronos2Pipelinefrom ML_Timeseries.backtest import backtest, record, scorefrom ML_Timeseries.data import bar_chart, load_household, showwarnings.filterwarnings("ignore")logging.disable(logging.WARNING)print(f"CUDA visible: {torch.cuda.is_available()}; CPU threads available: {os.cpu_count()}")
Each model forecasts the 30 backtest windows twice: with four threads (the whole machine) and with one (a fair share on a busy server). Latency is the mean time per 24-hour forecast.
rows = {}for name, (load, forecast) in MODELS.items(): model = load() params =sum(p.numel() for p in (model.model ifhasattr(model, "model") else model).parameters()) row = {"parameters (M)": params /1e6}for threads in (4, 1): torch.set_num_threads(threads) t0 = time.perf_counter()with torch.no_grad(): bt = backtest(y, lambda h, n: forecast(model, h, n)) row[f"ms per forecast, {threads} thread{'s'if threads >1else''}"] =1000* (time.perf_counter() - t0) /30 row |= {k: v for k, v in score(bt, y, exclude=FILLED).items() if k in ("mase", "skill")} rows[name] = rowdel model gc.collect()torch.set_num_threads(4)cpu = pd.DataFrame(rows).T_ = record("Chronos-Bolt tiny, zero-shot (CPU)", "foundation model", {"mase": rows["Chronos-Bolt tiny"]["mase"], "skill": rows["Chronos-Bolt tiny"]["skill"],"seconds": rows["Chronos-Bolt tiny"]["ms per forecast, 1 thread"] *30/1000}, notebook=NB, notes="9 M parameters, one CPU thread")cpu.style.format({"parameters (M)": "{:.0f}", "ms per forecast, 4 threads": "{:.0f}", "ms per forecast, 1 thread": "{:.0f}","mase": "{:.3f}", "skill": "{:+.1%}"})
parameters (M)
ms per forecast, 4 threads
ms per forecast, 1 thread
mase
skill
Chronos-Bolt tiny
9
10
15
0.644
+19.8%
Chronos-Bolt small
48
33
64
0.642
+20.3%
Chronos-2
119
117
272
0.645
+21.7%
TiRex
35
118
115
0.624
+23.6%
show(bar_chart(cpu.index, {"ms per forecast, 1 thread": cpu["ms per forecast, 1 thread"]}, y_name="ms", height="300px"))
Chronos-Bolt tiny is the headline. Nine million parameters, 15 milliseconds per forecast on a single CPU thread, and a skill within two points of Chronos-2, with no training. That is better than every classical and gradient-boosting model on the scoreboard, from a 30 MB file. At 15 milliseconds, one core produces about 240,000 day-ahead forecasts an hour, enough for a large fleet of meters.
The other rows show how the architectures differ off the GPU:
Chronos-2 is the slowest on one thread (about a quarter of a second) and gains the most from four, because its transformer layers are large matrix products that parallelise well.
TiRex is the most accurate and does not speed up with more threads: its xLSTM reads the context one step at a time, and a sequential loop cannot be split across cores. On a GPU it was the slowest of the three for the same reason.
Chronos-Bolt small buys half a point over tiny for four times the latency.
Pick by the deployment, not the leaderboard: tiny where cost and latency dominate, TiRex where the last points of accuracy matter, and check its licence if the use is commercial.