Add robots.txt generation and sitemap chunking

- bunext sitemap now writes public/robots.txt (configurable via the
  robots config key; robots: false or --no-robots to skip). Hand-written
  robots.txt files are never overwritten without a robots config
- Large sitemaps are split into public/sitemap-chunks/sitemap-N.xml
  (sitemap.chunk_size, default 10,000, max 50,000; 50MB per file) with
  sitemap.xml as a sitemap index; stale chunks are cleaned up
- Export bunextGenerateRobotsTxt
- Run tests in publish.sh

Co-Authored-By: Claude Opus 5.5 <[email protected]>

dist/utils/generate-robots-txt.d.ts

@@ -0,0 +1,11 @@
+import type { BunextGenerateRobotsTxtParams } from "../types";
+/**
+ * First line of every generated robots.txt. `bunext sitemap` only
+ * overwrites an existing robots.txt that starts with it, unless robots
+ * is configured explicitly.
+ */
+export declare const RobotsTxtGeneratedMarker = "# Generated by bunext";
+/**
+ * # Generate a robots.txt string
+ */
+export default function generateRobotsTxt(params?: BunextGenerateRobotsTxtParams): string;