nbdevAuto

Command line automation for the nbdev publish cycle, plus the dataset and image helpers the notebooks in this collection reuse.
Author

Benedict Thekkel

Unlike most sites in this collection, this one documents an installable library rather than a topic. nbdevAuto exists because the nbdev release cycle is a fixed sequence of commands typed in the same order every time: export the notebooks, run the tests, clean them, bump the version, commit, push, build the docs. Each step is one nbdev call, and forgetting one produces a confusing failure two steps later.

The library collapses that sequence into single-word shell commands. It is what just upload and just full_upload invoke under the hood for every submodule in the parent Knowledge repo.

Install

pip install nbdevAuto          # the publish-cycle commands and the helpers
pip install 'nbdevAuto[gh]'    # also installs githubkit, which `ghstatus` needs

Requires Python 3.10 or newer.

Use

Every exported function is also a console script, so the common case is a bare word in the terminal at the root of an nbdev project:

upload -m "commit message"    # export, test, clean, then add/commit/push

Or import the helpers into a notebook:

from nbdevAuto.functions import download_search_images, classify_images
from nbdevAuto.pdf import PDFreader

Contents

Page Covers
Functions Dataset and image helpers: reading a list out of a text file, downloading single images or whole search-driven datasets, verifying and resizing what came back, building the folder layout a classifier expects, running a fastai learner over one image, Kaggle competition and dataset shortcuts, and graph, a graphviz.Digraph subclass preloaded with a rounded, filled style
Automate The command line surface: the prep/commit/push pipeline, the GitHub and PyPI release halves, the help output that lists them all, and the one-line-per-stage console grid every command reports through
PDF Reader PDFreader, a class that converts a PDF through pdf2image and renders page ranges inline when you slice it (pdf[0:5])
GitHub ghstatus: reading .gitmodules, resolving a token, and asking the GitHub API about every submodule at once - CI runs, Pages deploys, Pages builds, remote main shas - plus the --report and --audit views, the exit codes that keep “absent” apart from “cannot ask”, and newest_run, which picks a run by run_number because the API’s page order is not a contract
Fleet fleet: running one command across every submodule of a superproject. A walker that reports only what did work and counts the rest, plus the verbs built on it - upload, push, copy, status, update and sync

The Command Line Surface

Installing the package puts these on your PATH. Most are ordinary functions in Automate, exposed as scripts through [project.scripts]; ghstatus comes from GitHub and fleet from Fleet.

Command Does
prep Bump the version, then export, test and clean the notebooks, refreshing _quarto.yml and the README
gacp git add, commit, and push. Without -m the message is built from the staged paths
status Show the working tree state
upload prep then gacp, the one you actually type
gitrelease Bump the minor version, then tag and create a GitHub release
piprelease Build the sdist and wheel, then upload to PyPI with twine
release gitrelease then piprelease
ghstatus Remote status for every submodule of an nbdev superproject, from the GitHub API. --report for the sync view, --audit to check .gitmodules against the repos the account owns, --json for machine output
fleet Run one command across every submodule: fleet upload|push|copy|status|update|sync. Every verb takes --help, and the destructive ones take --dry-run
h Print the list of available commands

Both prep and upload take -p to choose which version part to increment; it defaults to 2, the patch, so every upload ships a new version.

piprelease is the manual path. Pushing to main publishes to PyPI on its own: publish.yaml runs after CI passes, and skips quietly when the version is already there.

ghstatus needs the gh extra (pip install 'nbdevAuto[gh]'), which brings in githubkit. It is an extra rather than a dependency because githubkit pulls pydantic, httpx and hishel behind it, which is a lot of weight for anyone who only wants upload. Without it the command exits 3 with an install hint rather than a traceback. It takes its token from $GITHUB_TOKEN / $GH_TOKEN, then gh auth token.

The Output Contract

Every command reports through one grid, defined in Automate and reused by Fleet. It exists because these commands run 19 times in a row, so anything that costs three lines per stage costs sixty over a fleet.

1/6  bump version (part 2)  OK       75ms  0.3.67 -> 0.3.68
2/6  nbdev_export           OK      129ms
3/6  nbdev_test             OK       1.5s  6 notebooks, slowest 01_Automate.ipynb 1.0s

Four rules hold it together, and a change to any stage has to keep them:

  • One stage, one line, inside 88 columns. The columns are fixed: n/total (5), label (22), outcome (4), duration (7), detail (the rest). A unit test renders real stages through a console pinned to that width and asserts every line fits.
  • A summariser returns short strings, not a transcript. _test_summary turns a line per notebook into 6 notebooks, slowest X 1.0s; _push_summary turns nine lines of git progress into a sha range and a size. Anything too long for the detail column moves to an indented continuation line, so a verbose summariser degrades the layout rather than breaking it.
  • A stage that shells out must capture at the file-descriptor level. redirect_stdout only rebinds Python’s sys.stdout, so a child process writing to fd 1 walks straight past it - which is how nbdev_readme used to spill sixteen lines of pandoc metadata into the middle of a run. _quiet swaps fds 1 and 2 for a temp file as well and merges both captures.
  • A green OK has to mean something. _run raises on a non-zero exit rather than returning a code nobody reads, and the failure prints the command’s own output. Before that, a failed git push still reported OK.

On a terminal the label is written first without a newline, so a slow stage is visible while it runs, then rewritten in place. Piped output skips that entirely, because a carriage return in a log file is just noise.

Where It Is Used

The parent Knowledge repo’s justfile calls upload once per submodule. just upload runs it only where the working tree is dirty, just full_upload runs it everywhere. That is the main consumer, so a change to automate.py affects the publish path for every site in the collection.

ghstatus is the other half of that loop. upload pushes to 19 repos and then goes blind: whether each repo’s CI passed and whether its Pages deploy actually landed is invisible from the terminal. just ci, just report and just audit in the parent repo are all thin wrappers around this one command, which answers for the whole fleet in about 1.5 seconds instead of 19 browser tabs.

Releases

This package publishes itself. A publish.yaml workflow runs after CI passes on main and uploads to PyPI through Trusted Publishing, so no API token is stored in the repository. It reads __version__ out of nbdevAuto/__init__.py and asks PyPI whether that version already exists: a new version is published, an existing one is skipped without failing the build.

Because upload bumps the patch version on every push, an ordinary commit made with upload ships a release. A plain git commit leaves the version alone and the workflow skips. gitrelease, which bumps the minor version and creates the GitHub release and tag, stays a deliberate manual step.

Not Covered Yet

  • No tests for the helpers. 01_Automate.ipynb, 03_GitHub.ipynb and 04_Fleet.ipynb all carry unit tests for their pure functions - the fleet ones against a throwaway superproject of real git repos - but the dataset and image helpers in 00_Functions.ipynb are exercised only by being used, so a broken one surfaces in a downstream repo rather than here.
  • Undeclared dependencies. pyproject.toml declares graphviz and fastcore, plus githubkit behind the gh extra. The helpers additionally import fastai, fastbook, fastdownload, PIL, matplotlib, tqdm, pdf2image (with poppler behind it), nbdev and kaggle (which also needs its credentials). All of those are imported lazily inside the function that needs them, so installing the package and running the console scripts still works - only the specific helper you call fails. Install them yourself.
  • core.py is an empty nbdev stub (a single foo) left over from 00_core.ipynb, which no longer exists.
  • CHANGELOG.md is still the nbdev stub (Version 1.0.0 - Initial release) while the package is past 0.3.68. Nothing generates it, and upload bumps the patch on every push, so it would need writing by hand or dropping.
  • The docstrings are one-liners, so the rendered API pages are thin. The notebooks carry the real explanation.
Back to top