Often, depending on which plugin or core feature you are using this plugin isn’t there but instead Dynamically generated. Here are a few tips and insights into how Robots.txt works on Shifter and what you can do to improve the SEO of your static WordPress site. Understanding the purpose and how Robots.txt works with WordPress is key to getting optimal SEO on Shifter.
Robots.txt and Sitemaps on Shifter
We use Robots.txt in a few ways with Shifter. First, it’s a way for the static site generator to learn about and capture specific files including your Sitemap XML related ones.
Our generator will take a look at robots.txt and based on what it finds; it will follow files and paths to capture those necessary files. Whether you’ve edited this file and whether your plugin automatically added these URLs depending on which plugin you are using.
Yoast SEO
Yoast is one of the most popular SEO plugins with over 5+ million active installations according to the official plugin directory. Yoast works excellent on Shifter, we use it often, and the Yoast team does a great job of keeping this plugin updates with newer versions of PHP, so it’s a great fit.
The tricky thing about using Yoast on Shifter is how Yoast handles robots.txt. By default, Yoast does not add a reference to their Sitemap Index XML file in the WordPress robots.txt. Out of the box with a clean install with Yoast, the Shifter generator will not see or capture the necessary data unless we change this.
Yoast’s reasoning for this is understandable. Their plugin works dynamically to create a sitemap on the fly as users view pages. If your site has tens of thousands of posts, it’s possible that this is going to strain your server and site performance overall. While running Yoast and WordPress on Shifter, this isn’t a problem at all. We need to add the sitemap path to robots.txt somehow.
How to add Yoast sitemap to robots.txt
Enter, Surbma – Yoast SEO Sitemap to robots.txt, a plugin developed precisely for this purpose. Created by the Surbma team, this plugin is a simple no settings add-on that works alongside the original Yoast plugin. To use it you must also have the Yoast plugin installed. This WordPress plugin will add the robots.txt file reference needed for search engines to crawl your site, even static WordPress sites!
Considering how Yoast works by creating dynamic sitemaps and Shifter is all about static WordPress is not a problem at all. This feature is supported and works seamlessly on Shifter. Your WordPress site will be faster and happier because of it.
Page Speed Rank
One of the most significant benefits for migrating to Shifter that isn’t so obvious at first is SEO rank. By running WordPress as static on Shifters unique hosting platform, users can experience up to 10x the speed without any code changes. Ultimately, this means faster load time for you, your users, and most importantly Google.
Google has confirmed that Page Speed is a part of the official ranking now. The most recent update to this announcement includes a focus for mobile search results, but they do acknowledge Page Speed has been a factor for some time now.
For example, check out their post on the same topic from 2010 stating: “We launched this change a few weeks back after rigorous testing. If you haven’t seen much change to your site rankings, then this site speed change possibly did not impact your site.”
If you are using Yoast SEO, consider adding the Surbma plugin as well to ensure Google and other search engines are correctly crawling your static WordPress site.
Google XML Sitemaps
The Google XML Sitemaps WordPress plugin works a bit differently than Yoast. Its primary focus is not just SEO but solely Sitemaps. It works in a similar way but the main difference between Yoast and Google XML Sitemaps while using them on Shifter is that the Google XML plugin will add the sitemap reference to robots.txt. by default.
The important thing to note is, by default. This setting is optional! If you are using Google XML Sitemaps and you notice that your sitemap isn’t captured on your static site you need to adjust a few settings.
Log into your WordPress site and head over to those the plugin’s settings. There you’ll find a toggle or checkbox labeled “Include sitemap in robots”, which you can check to enable this feature.
Multi-environment Robots.txt Settings
A recent release to our hosting platform included a change to how Shifter generates your WordPress site’s robots.txt file. Depending on which environment you are on and who is looking at it, Shifter will serve different versions of this file.
For example, the static version of your WordPress site will always display the correct unedited version of robots.txt ensuring you have the best chance at optimal SEO and getting indexed.
If you are running our temporary container environment and you are logged out of WordPress, you will be served a dummy robots.txt file asking Google and other search engines to simply ignore that site.
This is important because your temporary container is exactly that. Temporary. It should not get indexed or crawled. This unique feature is available on every Shifter site by default and there is nothing you need to do to enable it. It’s always working!
Give it a try and let us know what you think. Find us online at @getshfiter or use hashtag #getshifter.