Aren't search engines excluded via robots

brokenheard320@

We can help you! Are you looking for a competent, flexible and reliable online marketing agency? We look forward to your non-binding inquiry. Contact us Do you notice a large number of “strange” URLs in the crawl? Common reasons for “strange” URLs include: Caused by JavaScript Print versions Parameter URLs, for example for filters Session IDs and much more! I recommend Artur Kosch's detailed blog post about JavaScript and SEO . Irina Hey from RYTE explains to you how to subject JS websites to an SEO audit . Are all important pages regularly crawled by Google Bot? Does Google Bot crawl important and unimportant pages in a healthy ratio.

Is there a current sitemap? Is the sitemap error-free? No pages blocked in robots.txt? Only pages with status code Special Data no “noindex” pages? Is the sitemap submitted to Google Search Console? Is the sitemap entered in the robots.txt of your website? Is the statement rel=prev and rel=next (correctly) used for pagination pages ? To check which pages Google actually crawls (points 37 and 38) - Google tends to "invent" URLs, for example when using JavaScript - the only thing that will help you is a log file analysis . Crawl error You can also use RYTE or the Screaming Frog to determine whether your website is “healthy”: Are there no links on 4xx sites? Are there no links on 3xx sites? Are there no server errors? Are there no redirect chains? Are there no internal links linked via robots.txt.



Are there no soft 404 errors ? You can find soft 404 errors using Google Search Console, but I also recommend that you open a made-up URL in the browser, crawl it using RYTE or Screaming Frog and check the following: An error page should appear The status code of the invented URL should be 404 The URL should under no circumstances redirect to the homepage find soft 404 You can use Google Search Console to see whether Google recognizes some of your pages as soft 404s. My 11 tips for Screaming Frog will help you deal with an external crawler for the first time and will show you, among other things, how to identify redirect chains or links on 4xx.