Sitemap Generation

Get Help and Help Others
1

Generating a sitemap and making it available to search engines seems to be an important part to SEO. Has anyone thought about how to handle this in Redwood? Building one manually is trivial for small sites, but obviously gets unmanageable quickly if you generate pages dynamically from database records.

If I had to guess, I’d assume this wouldn’t be too tricky to build into the Router (if there isn’t already a magical solution,) but that’s awfully easy to say as someone who hasn’t looked at that part of the codebase :sweat_smile:

2

Ah, that’s important to add indeed! This might be something that could possibly make a great stand alone npm package. A very basic POC could parse the Route.js, skipping and not taking into account dynamic pages. An improved version would run at build time (e.g. Netlify deploy) and go fetch all the things. Even more improved would allow for that fancy weighting/prioritizing by routes. Hmm… :thinking:

Any ideas anyone?

1 Like
3

One could also render a sitemap on-demand via graphql query.

For example, findAll blog posts and render the sitemap via a function or maybe even a page w/ a non-html layout where the cells output the XML (???).

I have done that for RSS feeds (query an API vs DB), but similar in concept.

I could see a sitemap being a composite of approaches:

  • static pages
  • router pages that are not dynamic (ie, not different based on params)
  • graphql-derived pages
2 Likes
4

Thank you both for your guidance here.

It appears that I got this working mostly as David T. described, but thought I’d share the exact steps I had to make in case others stumble upon this thread.

The steps I’d take if I were to do this over again today:

  1. Create a custom function that does the job of building up one long xml string with all the pages of your site in a format accepted by google. This thread was helpful and could be a good resource if you’d like more detail.

In your custom function you’ll want to return an object with the following shape:

return {
    statusCode: 200,
    headers: {
      'Content-Type': 'text/xml',
    },
    body: xml,
  }
  1. Edit netlify.toml to create a new redirect above the existing redirect, which redirects /sitemap.xml to your newly created custom function. This new redirect must be placed above your existing redirect or you’ll get an error.

If this is the only new redirect you’ve added to netlify.toml, the redirects section of your file should look like this:

[[redirects]]
  from = "/sitemap.xml"
  to = "/.netlify/functions/[NAME_OF_YOUR_CUSTOM_FUNCTION]"
  status = 200
  force = true

[[redirects]]
  from = "/*"
  to = "/index.html"
  status = 200

A few things I tried, but didn’t work at the time of this writing:

  • Creating a Sitemap route in Routes.js, which had a path to /sitemap.xml and served an XML string. I’m not sure if this didn’t work because the path had an .xml extension and we were serving HTML, or if the router doesn’t like the XML extension at all.
  • Using the netlify-sitemap build plugin. Though it wasn’t immediately apparent, this plugin doesn’t appear to work for SPAs at the time of this writing.

Anyways, I hope this ends up being helpful down the line.

@betocmn I noticed that you have a sitemap up for Duoflag – if you don’t mind sharing, how did you end up generating it? I’d love to learn of a smoother way than what I describe here.

2 Likes
5

Nice @tctrautman!

Your steps are exactly what I had done 1) custom function to assemble sitemap xml and 2) a redirect.

I don’t know why I didn’t provide a better example, but I used:

To construct the XML by iterating over the various pages I wanted to include (ie, posts or activities and they canonical url).

I would also recommend to:

The prerendering will help Google out when it requests to index the sitemap.

2 Likes
6

For example, if the data is a collection of activities in a feed:


try {
  const { data } = await fetchFeed();

  const urls = (data['results'] || []).map(activity => ({
    url: getCanonicalUrl(activity, 'sitemap', false),
    changefreq: 'monthly',
    lastmod: activity['time'],
    priority: 0.5,
  }));

  const smi = new Sitemap({
    urls: urls,
    hostname: `https://${env.APP_SUBDOMAIN}.${env.APP_HOST}`,
    cacheTime: 600000,
  });

  res
    .header('Content-Type', 'application/xml')
    .status(200)
    .send(smi.toXML());
} catch (error) {
  console.log(error.toString());
  res.status(400).send(error.error);
}

Note: getCanonicalUrl adds some utm social and campaign codes (like if sent to Twitter) in other use cases (but does not for sitemap)

2 Likes
7

This is super helpful – thank you!

8

Hey @tctrautman, I’ve actually just generated it manually to get started. And I also used react-helmet + Netlify pre-rendering to get SEO going.

But it’s on my list to automate this, so this post is super helpful, it just wasn’t a priority so far. Thanks for the reminder and the step-by-step, super useful!

2 Likes
9

BTW - the same basic approach can be applied to dynamically generate RSS feeds, too.

In other apps, I previously have created Netlify functions for the various syndication flavors (rss, atom, json) using Feed:

And then the function assembles the necessary info getSyndicatedFeed and renders the specific format:

const { data } = await fetchFeed();

const feed = getSyndicatedFeed(data);

res
  .header('Content-Type', 'application/xml')
  .status(200)
  .send(feed.rss2());

or feed.atom etc.

2 Likes