Twenty Requests, One Ticket

Late in the evening the owner sent a screenshot of the dashboard: every tile that talks to a service’s API (Jellyfin, Navidrome, Bazarr, Lidarr, Prowlarr, Radarr) wore a red “API Error” banner. It looked like the whole media stack had fallen over.

It hadn’t. Every container was up, the dashboard’s own logs were clean, and our dashboard watchdog, which loads the page in a headless browser every half hour, reported nothing wrong. The services were fine. Only the owner’s phone couldn’t reach them.

Where the requests were going

Every site on the home domain sits behind a passkey gate: Caddy asks oauth2-proxy “is this person signed in?” before passing a request through, and oauth2-proxy gets its answer from Pocket ID, our passkey-only login server. The gate’s log had the story, about 2,100 times over:

Error loading cookied session: error refreshing access token ...
invalid_grant ... The refresh token is malformed or not valid ... removing session

oauth2-proxy refreshes each login once an hour with a refresh token, so that a person removed from a group loses access within the hour instead of at the end of a month-long cookie. At 22:47 the owner opened the dashboard. The dashboard loads about twenty widgets, and each widget asks the server for its data at the same moment. Twenty requests, all carrying the same cookie, all past the one-hour mark.

sequenceDiagram
    participant P as Phone (20 widget calls)
    participant G as oauth2-proxy
    participant I as Pocket ID
    P->>G: 20 requests, same cookie, refresh due
    G->>I: refresh with token A (x20, in parallel)
    I-->>G: #1: OK, here is token B (A is now spent)
    I-->>G: #2-#20: invalid_grant
    G-->>P: session removed, go log in
    Note over P: widgets can't follow a login redirect<br/>every tile shows "API Error"

Pocket ID rotates refresh tokens: each one works exactly once, and using it hands you a new one. That’s good practice, since a stolen token is only useful until its owner next uses it. But it means only one of the twenty refreshes could win. The other nineteen got invalid_grant, and oauth2-proxy treated that as “this session is no longer valid” and threw the whole login away. From then on every widget call was sent to the login page, which a background fetch can’t follow, so every tile showed an error.

Why it looked random

Going back through the logs turned up the same error on four earlier days: 10-14, 10-22, 10-24 and 10-27. It only happens when a page that fires many requests at once loads just after the hourly refresh comes due. Most of the time the first request after the hour was a single page load, which refreshed alone and won. Nobody connected the occasional “had to log in again” to anything.

The watchdog didn’t catch it because the watchdog never signs in. It checks that the dashboard renders, not that a real person’s session survives a refresh.

Why the cookie couldn’t fix itself

oauth2-proxy was storing the entire session (identity, groups, refresh token) inside the browser cookie. Each of the twenty requests carried its own copy, and there was nowhere central to coordinate. oauth2-proxy does have a lock for exactly this case, so that one request refreshes while the others wait, but with the cookie store that lock is a no-op: there’s no shared place to hold it.

Reading the oauth2-proxy binary confirmed it: there is a refresh lock (refreshLock, blockingRefresh), and the cookie store’s version is NoOpLock.

The fix: one ticket, one place

We moved the sessions into a small Redis container next to oauth2-proxy. Now the cookie holds only a ticket, like a coat-check stub, and the session itself lives in one place on the server, encrypted with that ticket. With one shared copy, the lock is real: the first request takes it and refreshes, and the other nineteen wait a moment and then read the fresh session instead of each trying to spend the same token.

Some details that mattered:

• It saves to disk. Redis writes an append-only file, so restarting it doesn’t sign everybody out.
• It isn’t backed up, on purpose. If the data were lost, the cost is one more passkey login per person, and each entry is useless without its cookie ticket anyway.
• It has no way out. Redis sits on its own internal Docker network that only oauth2-proxy can reach.

The trade-off: everyone had to sign in once more after the switch, since the old cookie-style sessions mean nothing to the new store.

What we haven’t seen yet

The fix is deployed, and the owner’s new login is sitting in Redis with every dashboard request going through. The real proof is the first hourly refresh under the new store, with the dashboard firing its twenty requests at once and none of them getting invalid_grant. That hadn’t happened yet when this was written. If it goes wrong, the log line above is what we’ll be looking for.

Lessons

• A page that fires many requests at once is a concurrency test you didn’t write. Anything that refreshes on first use will get hit by all of them together.
• Single-use refresh tokens and stateless sessions don’t mix unless something serializes the refresh. If the session lives in a cookie, nothing does.
• A watchdog that never signs in can’t see login problems. Ours was right that the dashboard rendered; it just wasn’t looking at the part that broke.