Add robots.txt generation and sitemap chunking
- bunext sitemap now writes public/robots.txt (configurable via the robots config key; robots: false or --no-robots to skip). Hand-written robots.txt files are never overwritten without a robots config - Large sitemaps are split into public/sitemap-chunks/sitemap-N.xml (sitemap.chunk_size, default 10,000, max 50,000; 50MB per file) with sitemap.xml as a sitemap index; stale chunks are cleaned up - Export bunextGenerateRobotsTxt - Run tests in publish.sh Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Claude Opus 5.5
@@ -39,7 +39,9 @@ The goal is a framework that is:
|
||||
- [Sitemap](#sitemap)
|
||||
- [Page `sitemap` Export](#page-sitemap-export)
|
||||
- [`bunext sitemap` Command](#bunext-sitemap-command)
|
||||
- [Large Sitemaps](#large-sitemaps)
|
||||
- [Generating at Runtime](#generating-at-runtime)
|
||||
- [robots.txt](#robotstxt)
|
||||
- [Static Files](#static-files)
|
||||
- [Caching](#caching)
|
||||
- [Enabling Cache Per Page](#enabling-cache-per-page)
|
||||
@@ -159,7 +161,7 @@ bun run dev
|
||||
| `bunext dev` | Start the development server with HMR and file watching. |
|
||||
| `bunext build` | Bundle all pages for production. Outputs artifacts to `.bunext/public/pages/` using paths relative to the project root. |
|
||||
| `bunext start` | Start the production server using pre-built artifacts. |
|
||||
| `bunext sitemap` | Generate `public/sitemap.xml` from your pages, using the `sitemap` options in `bunext.config.ts`. See [Sitemap](#sitemap). |
|
||||
| `bunext sitemap` | Generate `public/sitemap.xml` and `public/robots.txt` from your pages, using the `sitemap` and `robots` options in `bunext.config.ts`. See [Sitemap](#sitemap). |
|
||||
|
||||
### Running the CLI
|
||||
|
||||
@@ -694,16 +696,37 @@ export default config;
|
||||
```
|
||||
|
||||
```bash
|
||||
bunext sitemap # writes public/sitemap.xml
|
||||
bunext sitemap -o public/sitemaps/main.xml # custom output path
|
||||
bunext sitemap # writes public/sitemap.xml and public/robots.txt
|
||||
bunext sitemap -o public/sitemaps/main.xml # custom sitemap output path
|
||||
bunext sitemap --origin https://staging.example.com
|
||||
bunext sitemap --no-robots # sitemap only
|
||||
```
|
||||
|
||||
`--output` and `--origin` override `sitemap.output` and `sitemap.origin` from the config.
|
||||
#### Large Sitemaps
|
||||
|
||||
A sitemap file may hold at most 50,000 URLs and 50MB. When the sitemap has more than `chunk_size` URLs (default 10,000), `bunext sitemap` splits it into chunk files and makes `sitemap.xml` a sitemap index pointing to them:
|
||||
|
||||
```
|
||||
public/
|
||||
├── sitemap.xml ← <sitemapindex> referencing the chunks
|
||||
└── sitemap-chunks/
|
||||
├── sitemap-1.xml ← up to chunk_size URLs each
|
||||
├── sitemap-2.xml
|
||||
└── sitemap-3.xml
|
||||
```
|
||||
|
||||
Each index entry's `lastmod` is the newest `lastmod` in its chunk. Chunk files left over from a previous, larger run are removed; other files in the chunks directory are left alone. If the sitemap shrinks back under `chunk_size`, `sitemap.xml` returns to a plain `<urlset>`.
|
||||
|
||||
| Config option | Type | Description |
|
||||
| ------------- | -------- | --------------------------------------------------------------------------------------------------- |
|
||||
| `chunk_size` | `number` | Maximum URLs per file. Default `10000`, capped at `50000`. Chunks are also split at 50MB. |
|
||||
| `chunks_dir` | `string` | Chunk directory relative to the project root. Must be inside `public/`. Default `public/sitemap-chunks`. |
|
||||
|
||||
`--output` and `--origin` override `sitemap.output` and `sitemap.origin` from the config. `--origin` also applies to robots.txt. The command also writes a `robots.txt` — see [robots.txt](#robotstxt).
|
||||
|
||||
### Generating at Runtime
|
||||
|
||||
`bunextGenerateSitemap()` resolves to the same XML as a string, so you can serve it from a route instead of a file:
|
||||
`bunextGenerateSitemap()` resolves to the same XML as a single `<urlset>` (it does not chunk, and logs a warning above 50,000 URLs — use `bunext sitemap` for large sites), so you can serve it from a route instead of a file:
|
||||
|
||||
```ts
|
||||
// src/pages/api/sitemap.ts
|
||||
@@ -738,7 +761,51 @@ export default async function handler(req: Request) {
|
||||
| `default_changefreq` | `BunextSitemapChangeFreq` | `changefreq` for entries that do not set one. |
|
||||
| `default_priority` | `number` | `priority` (0.0 to 1.0) for entries that do not set one. |
|
||||
|
||||
The `sitemap` config key takes the same options, plus `output`: the file path relative to the project root (default `public/sitemap.xml`).
|
||||
The `sitemap` config key takes the same options, plus `output` (the file path relative to the project root, default `public/sitemap.xml`) and the [chunking options](#large-sitemaps).
|
||||
|
||||
### robots.txt
|
||||
|
||||
`bunext sitemap` also writes `public/robots.txt`, served at `/robots.txt`. With no `robots` config it allows every crawler and points to the generated sitemap:
|
||||
|
||||
```
|
||||
# Generated by bunext
|
||||
|
||||
User-agent: *
|
||||
Allow: /
|
||||
|
||||
Sitemap: https://example.com/sitemap.xml
|
||||
```
|
||||
|
||||
Configure it with the `robots` key:
|
||||
|
||||
```ts
|
||||
// bunext.config.ts
|
||||
const config: BunextConfig = {
|
||||
origin: "https://example.com",
|
||||
robots: {
|
||||
rules: [
|
||||
{ user_agent: "*", disallow: ["/admin", "/api/"] },
|
||||
{ user_agent: ["GPTBot", "CCBot"], disallow: ["/"] },
|
||||
],
|
||||
// Block everything on staging
|
||||
disallow_all: process.env.STAGING === "true",
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
| Option | Type | Description |
|
||||
| -------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `rules` | `BunextRobotsRule[]` | Groups of `user_agent` (string or array), `allow`, `disallow` and `crawl_delay`. Default: allow everything for `*`. |
|
||||
| `sitemaps` | `string[]` | Extra `Sitemap:` URLs or paths. The generated sitemap is added automatically when it is inside `public/`. |
|
||||
| `disallow_all` | `boolean` | Block every crawler from the whole site. `rules` and `sitemaps` are ignored. |
|
||||
| `origin` | `string` | Base URL for relative `sitemaps` paths. Defaults to `--origin`, then `sitemap.origin`, then `config.origin`. |
|
||||
| `output` | `string` | Output file relative to the project root. Default `public/robots.txt`. |
|
||||
|
||||
- **Existing files are protected.** Generated files start with `# Generated by bunext`. Without a `robots` config, a `robots.txt` that lacks this line (one you wrote yourself) is left alone and a warning is shown. Setting `robots` in the config overwrites it.
|
||||
- **Opting out.** Set `robots: false`, or pass `--no-robots`, to skip robots.txt entirely.
|
||||
- **Sitemap `exclude` does not add `Disallow` lines.** Leaving a page out of the sitemap and blocking crawlers from it are different things; add `disallow` rules explicitly. Remember robots.txt is public, so `Disallow` paths are visible to anyone.
|
||||
|
||||
To serve it from a route instead, `bunext.bunextGenerateRobotsTxt()` takes the same options except `output` and returns the text.
|
||||
|
||||
---
|
||||
|
||||
@@ -883,6 +950,7 @@ export default config;
|
||||
| `websocket` | `WebSocketHandler<any>` | — | Bun WebSocket handler — see [WebSocket](#websocket) |
|
||||
| `serverOptions` | `ServeOptions` | — | Extra options passed to `Bun.serve()` (excluding `fetch`) — see [Server Options](#server-options) |
|
||||
| `sitemap` | `BunextSitemapConfig` | — | Options for `bunext sitemap` — see [Sitemap](#sitemap) |
|
||||
| `robots` | `BunextRobotsConfig \| false` | — | robots.txt options for `bunext sitemap`, or `false` to skip it — see [robots.txt](#robotstxt) |
|
||||
|
||||
### Middleware
|
||||
|
||||
|
||||
Reference in New Issue
Block a user