Never Let [0] Decide a Verdict

A status tool called a healthy repo FAILURE three times, three different ways. Each fix was correct and each one was not enough. The common thread: a transient or arbitrary answer read as a result.
Author

Benedict Thekkel

Published

October 5, 2026

The federated docs site is twenty repos, each with its own CI run, Pages deploy and Pages build. just ci asks the GitHub API about all of them concurrently, in about 1.5 seconds, and prints one row per repo plus an exit code that other recipes act on. A status tool has one job: when it says red, something is red.

It said red when nothing was, on 2026-08-26, 2026-09-21 and 2026-10-05.

1. 2026-08-26: the endpoint’s order is not a contract

The report called ML/ML_methods CI FAILURE and linked a real run: a genuine test.yaml run on main of that repo. It was run number 96, from 2026-02-17. The newest was run 107, from 2026-08-16, and it had succeeded. So had the Pages deploy.

The code asked for exactly one run and believed it:

r = await gh.rest.actions.async_list_workflow_runs(owner, repo, wf, branch="main", per_page=1)
run = r.parsed_data.workflow_runs[0]

The list-workflow-runs endpoint is eventually consistent, and while it usually returns newest first, nothing promises it. A one-item page is the worst way to consume an unreliable order, because it removes the only evidence you could have used to notice: there is nothing to compare the answer to.

It also did not reproduce. Twenty sequential calls and six full concurrent passes over the fleet all returned 107. A bug you cannot reproduce is still a bug when the code has no defence against it.

The fix costs nothing extra. Ask for a page and pick by a field that means “newest”:

def newest_run(runs):
    "The newest run in `runs`, by `run_number`, which is monotonic per workflow."
    numbered = [r for r in runs if getattr(r, "run_number", None) is not None]
    return max(numbered, key=lambda r: r.run_number) if numbered else runs[0]

Same single request with per_page=10, and the page’s order can no longer change the verdict. The unit test feeds it a deliberately mis-ordered page.

2. 2026-09-21: a state in flight is not a state

The second one was quieter. just ci exited 1 on a fleet that was fine, and no row in the table said FAILURE. A few minutes later the same command exited 0.

The CI and deploy columns already knew that in_progress and queued are not results. The Pages build column did not:

# before: anything that is not "built" counts as red
any(r["built"] not in (None, "built") for r in rows)

building is a state every healthy Pages deploy passes through. Run the check during that window and a healthy repo counts as failed, but the table renders building as an ordinary word, so the exit code and the table disagree and the reader cannot see why.

RUNNING = {"in_progress", "queued", "requested", "waiting", "pending",
           "building"}  # "building" is the Pages build in flight

def has_failure(rows):
    return bool(failures(rows)) or any(r["built"] not in (None, "built", *RUNNING) for r in rows)

Note what the fix does not do. It does not switch to an allow-list of bad states. A state the code has never seen still counts as red, because a new API value is something to look at, not something to wave through.

3. 2026-10-05: the maximum of a bad page is still bad

Then DL/DL_methods came back FAILURE with newest_run in place. The branch query returned a page whose highest run number was #143, from 2026-05-13, a failure. The newest run was #161, and it had succeeded.

max() by run number fixed the order of a page. It cannot fix a page that does not contain the right run at all. Taking the best element of the wrong set still gives you a wrong answer, just a more confident one.

The question being asked was never really “what is the newest run on this branch”. It was “did the commit main points at build”. The API can answer that directly:

if head_sha:
    r = await gh.rest.actions.async_list_workflow_runs(
        owner, repo, wf, head_sha=head_sha, per_page=10)
    runs = r.parsed_data.workflow_runs
if not runs:   # a commit with no run yet: fall back to the branch page
    r = await gh.rest.actions.async_list_workflow_runs(
        owner, repo, wf, branch=branch, per_page=10)
    runs = r.parsed_data.workflow_runs

The tool already fetched the remote main sha for another column, so this is no extra round trip in the common case. And a query keyed on the commit cannot be answered with some other commit’s run, however the backend’s index is behaving that day.

The pattern

flowchart LR
  A["per_page=1, take [0]"] -->|wrong order| B["per_page=10, max by run_number"]
  B -->|wrong page| C["query by head_sha"]
  D["built != 'built' is red"] -->|transient state| E["RUNNING set excluded, unknown still red"]

All three are the same mistake at different layers: an answer that was arbitrary or temporary, read as a result. The first trusted an order, the second trusted a moment, the third trusted a set.

The fixes share a shape too. Each one stopped asking a proxy question and asked the real one:

Proxy question Real question
What is first on this page? Which run has the highest run number?
Is the build state exactly built? Has the build finished badly?
What is the newest run on main? Did this commit build?

Takeaway

For any health check that turns an API response into red or green:

  • Never let [0] of a paged response decide. Ask for several and select by a field that means what you want.
  • Separate “not finished” from “failed”, and keep unknown states red rather than green.
  • Key the query on the thing you care about. If the question is about a commit, ask about the commit.

And the meta-lesson, which cost an exit code three times: a false red is not harmless just because a human looks at it. A tool that cries wolf trains you to stop reading its output, and then the real FAILURE gets ignored with the rest.

Back to top