How to Auto-Generate robots.txt and Sitemap in Astro

What I Wanted to Do

I wanted to add robots.txt and sitemap.xml to my Astro site for SEO.

Environment

• Astro 5
• Cloudflare Pages

Auto-Generating a Sitemap

1. Install the Plugin

npm install @astrojs/sitemap

2. Edit astro.config.mjs

import { defineConfig } from 'astro/config';
import sitemap from '@astrojs/sitemap';

export default defineConfig({
  site: 'https://your-domain.com',
  integrations: [sitemap()],
});

3. Verify

https://your-domain.com/sitemap-index.xml

Setting Up robots.txt

Save the following as public/robots.txt.

User-agent: *
Allow: /

Sitemap: https://your-domain.com/sitemap-index.xml

Gotchas

• If site is not configured, the sitemap will not be generated
• Place robots.txt inside the public/ directory
• Cloudflare may overwrite robots.txt, but this is generally not a problem

Once the sitemap is in place, also set up SEO meta tags in Astro to round out your SEO configuration.

Related Articles

• How to Deploy Astro to Cloudflare Pages
• How to Add a New Page in Astro
• Google Search Console HTML File Verification with Astro and Cloudflare Pages
• How to Read Cloudflare Pages Build Logs and Fix Errors

Recommended Services

• Fiverr - Find freelance developers and tech experts