Add robots.txt generation and sitemap chunking

- bunext sitemap now writes public/robots.txt (configurable via the
  robots config key; robots: false or --no-robots to skip). Hand-written
  robots.txt files are never overwritten without a robots config
- Large sitemaps are split into public/sitemap-chunks/sitemap-N.xml
  (sitemap.chunk_size, default 10,000, max 50,000; 50MB per file) with
  sitemap.xml as a sitemap index; stale chunks are cleaned up
- Export bunextGenerateRobotsTxt
- Run tests in publish.sh

Co-Authored-By: Claude Opus 5.5 <[email protected]>

This commit is contained in:

tben

2026-10-14 18:02:52 +01:00

co-authored by

Claude Opus 5.5

parent 1169729b06

commit be4bebaa22

20 changed files with 1406 additions and 89 deletions

									
										publish.sh
									
		+3
		-2
	
												View File
												
				@@ -2,14 +2,15 @@

				set -e

				tsc --noEmit

				bun run test

				if [ -z "$1" ]; then

				    msg="Updates"

				else

				    msg="$1"

				fi

				tsc --noEmit

				rm -rf dist

				tsc