One of the most important aspects of SEO is optimizing the crawlability of your site. Search engines have near-endless resources, so they have the power to crawl everything they find — and they will. But, that is not the way it should be. Almost every CMS outputs URLs that don’t make sense and that crawlers could safely skip. With Yoast SEO Premium 18.6, we’re starting a series of additions to clean up those unnecessary URLs, feeds, and assets so that the more critical stuff stands a better chance of being crawled.
Making your site easier to crawl
Google and other search engines crawl almost everything they can find — as Yoast founder Joost de Valk proves in a post on his site. But it can be hard to get them to crawl what you want them to crawl. Moreover, crawlers can come by many times each day and still not pick up the important stuff. There’s a lot to gain for every party involved — from the crawlers, site owners, and environment — to make this process more sensible. Yoast SEO Premium will help search engines crawl your site more efficiently.
In Yoast SEO Premium 18.6, we’re introducing the first addition to our crawl settings, allowing you to manage better what search engines can skip on your site. In this release, we’re starting with those RSS feeds of post comments in WordPress, but we have a long list of stuff that we want to help you manage.
Head over to our new Crawl settings section in the General settings of Yoast SEO Premium and activate the first addition to preventing search engines from crawling the post comment feeds.
This feature is available to all Yoast SEO Premium subscribers in beta form, and we’ve selected not to activate this for every site. In some cases, there still might be sites that use this in a way we can’t anticipate. We’re rolling out more crawling options — big and small — in the coming releases.
Let’s all start cleaning up the crawling on our sites — it’s better for you, your visitors, search engines, and the environment. All with a little help from Yoast SEO Premium. Let’s go!
Go Premium and get access to all our features!
Premium comes with lots of features and free access to our SEO courses!
Only $99 USD / per year (ex VAT) for 1 site
Keeping Bing updated on your site
Yoast SEO 19.0
This article was written by Edwin Toonen and originally published on SEO blog • Yoast.