Generating a sitemap and Google indexing
Google search
To instruct google to index your github-pages blog you’ll need to login to google and use the search console at https://search.google.com/search-console. You will be asked to provide some meta data to show that you own the site so you’ll most likely need to put a small section in your index.md file, however in my case I have a CNAME configured with github-pages, this very blog, https://blog.richy.net and it so happens my domain record already has a meta entry (to facilitate gmail) so for me, this whole process was very simple and I was verified at once. However google only indexes the pages you tell it to, so it is in your interests to generate a sitemap. There are many ways of doing this and for a very small site a sitemap.txt with a list of URLs may well be good for you, but it turns out to be very simple to get Jekyll to generate one for you.
Generating a sitemap
I chose to use the generator at https://github.com/jekyll/jekyll-sitemap/blob/master/README.md and it was straight-forward. This blog site is virtually vanilla with only Utterences configured, so the first to do is create a Gemfile
in the root of the project:
source "https://rubygems.org"
gem "minima"
gem "jekyll-sitemap"
You will need to add an entry to _config.yml:
plugins:
- jekyll-sitemap
and that’s it. If you are using VisualCode you can rebuild and deploy a container, and you can inspect the _site content which is now generated. You don’t commit any of the _site files and it should be excluded by your .gitignore
anyway, so just commit the Gemfile
and the _config.yml
files. When accepting your Pull Request on Github, your github-pages project will build and deploy, and hey-presto: https://blog.richy.net/sitemap.xml. Splendid!
Adding the sitemap to Google search
This is just to prove it works - here’s a screengrab of the config:
Addendum 22nd November: Adding the sitemap to Bing search
One you are validated with Google, it is essentially trivial to add the sitemap to Bing via their search tools: https://www.bing.com/webmasters/about. You will need to link your Microsft account to your Google account in which case Bing automatically pulls in the sitemaps from Google, if not you will have to validate your domain with Microsoft - I didn’t go through that as I took the easy option. Once Bing has it indexed it will (eventually) be picked up by DuckDuckGo which has a traditional crawler (that will use Bing indexing) and for me that’s all the seach engines covered that I care about.