Getting a new site indexed on Google for the first time can be difficult, you can see our guide Indexing Your WordPress Site on Google (Step-by-Step) for more information about how to do that. An indexing emergency, on the other hand, is when some or all of your pages disappear from the index. You may have deduced that you have an indexing problem after seeing traffic drop, not finding your site in Google, not finding your site when searching using the “site:” operator or via a report in Google Search Console like this one.
What indexing problems will do to your SEO
Being indexed by Google is fundamental to being listed in search results. Google’s search results are not drawn from live websites, but from an index of pages and links found by Google when crawling the web. This index is stored in Google data centers across the world.
Pages that are not in the index cannot feature in search results. If a page was ranking in Google for a search term and then it drops out of the index, you will lose that visibility in Google (unless another page from your website replaces it). When no pages on your site are indexed by Google then you have no visibility and no traffic. That is an SEO disaster!
Troubleshooting Google indexing problems
Here are some steps to go through to find why your site is experiencing indexing problems and how to fix them.
Is your site really online?
As suggested in the first chapter, step one when you have a ranking problem is to ensure that your site is really online by opening it in your browser. Empty your cache and refresh your page to make sure that you are not seeing a cached version of your homepage.
Log in to your WordPress admin as a final check that the site us working correctly.
Is the site set to noindex?
WordPress has a Search engine visibility option to help hide sites from Google. With the “Discourage search engines from indexing this site” option checked, you are requesting Google not to index your site. This is useful when you are not ready to launch your site to the general public. But if the option is accidentally checked when a site should be live, it will be disastrous (and you would be surprised how often it happens!). This why SEOPress adds a red warning to the WordPress admin header when noindex is set sitewide in WordPress.
SEOPress also gives users control over noindex and other robots directives for search engine via the Titles & Metas > Advanced screen. You can also modify these directives for individual pages and posts in the SEO metabox, Advanced settings tab. If you have found that any of these options have been checked, simply unchecking them may solve your Google indexing emergency!
Do you have a security issue or manual action
Check your emails and log in to Google Search Console to make sure that your indexing problem is not linked to a manual action or security issue.
What does Google Search Console tell you?
Google Search Console is a free tool provided by Google specifically to give website owners a way of checking the indexing status of their websites. It was originally called Google Webmaster Tools.
Beyond checking your site is online and that noindex options have not been selected in WordPress, you will need this tool to troubleshoot Google indexing issues. If you have not created a Google Search Console account yet, follow our quick guide Add your site to Google (Search Console).
The Page indexing report first shows a chart of page indexation status over time. Green for indexed, gray for not indexed. Below this report you will find a list of reasons that pages aren’t indexed. This report shows a reason per line with the number of pages not indexed for this reason and a simple line graph showing the trend over time. This line is very useful for spotting recent changes that may explain recent traffic or ranking problems. You can click on each line to get a detailed report on each error.
We will go through some of the most common reasons given by Google below. The screenshot above shows that many pages from this site have been removed from the index because they have been deleted (Not found (404)) or redirected (Page with redirect) – this is because the example is of a site that was migrated from one domain to another. This is not a SEO emergency, because the new site is being indexed and ranked correctly.
Excluded by ‘noindex’ tag
This is the error that will be shown if you have requested Google not to index your WordPress site or individual pages as discussed above. You can solve this by unchecking noindex options in WordPress or SEOPress settings.
URL blocked by robots.txt
A robots.txt file is a simple text file that can be used to give instructions to search engines about which pages they can crawl and index. If many, or all, of your pages are showing as not indexed for this reason, then you may have blocked Google from indexing your site after editing this file.
Goto the robot.txt report in Google Search Console to get more information. You can find the report by clicking on Settings in the left-hand menu and then on OPEN REPORT next to “robots.txt”.
You can edit the robots.txt via FTP or using the dedicated SEOPress PRO feature. When you have corrected the error, go back to the robots.txt report and click on the ellipse (3 points) button and then Request a recrawl.
Not found (404)
When robots or browsers request a file from a website, they receive a numeric code as a short reply. A 200 code indicates that the file was found and is on its way (all is good). A 404 code means that the file was not found. If a lot of URLs are showing up as Not found in the Page indexing report, then you must have deleted or moved pages on your website by changing permalinks.
If changing permalinks was intentional, you should now set up redirections to ensure that Google can associate the old URLs (that were ranking) with new URLs (that may not be indexed yet). See our guide Enable 301 redirections and 404 monitoring on how to do this with SEOPress PRO.
Google Search Console will also be showing reports for other server errors such as 401, 403, 410 or 500. See the dedicated help section on Server errors in the Google Search Console’s Help documentation for more information on resolving these errors.
Discovered – currently not indexed
This error is not likely to occur when an unchanged site drops out of the index, but it may happen after a site migration or if you have added a lot of new content. This error means that Google found new pages but thought that crawling them would overload your server. This is typically a problem your web hosting and your visitors are probably also experiencing slow load times and error messages when they visit the site. Contact your hosting provider to resolve this problem.
Crawled – currently not indexed
You may also get this error message for new sites or new pages that you have added to the site. Google says that has discovered your pages, but is not interested in indexing them. This is not usual in an indexing emergency, but it is annoying. One of the reasons that Google does not want to index your new pages is that they contain no content or no original content (i.e., they are duplicates of content already indexed elsewhere on the web). Among other reasons for this error is the possibility that you rarely add new pages to your site so Google has set indexing to low priority for your site.
We suggest using the URL inspection tool in Google to request indexing for these pages. Inspect one of the URLs showing in the report and click on the TEST LIVE URL at the top of the page. View crawled results and specifically the SCREENSHOT tab to see how Google views the page (there may be an error here that you can fix) and if everything looks OK, click on REQUEST INDEXING.
Getting back into Google’s index
If the Google Search Console has helped you identify and resolve issues with your website, you should go back to the report and tell Google you have fixed the problem. Doing this should speed up the reindexing of your pages. Go back to the Page indexing report and click on the issue that you have resolved and then click the VALIDATE FIX button.
This will immediately start a validation process and you can click on the new SEE DETAILS button to see how this is getting on. You can expect validation to take one or two days but it is entirely automatic.
Another trick to reduce the time it takes Google to reindex your pages is to submit a sitemap. If you have already submitted a sitemap to Google Search Console, delete it and submit it again. More about sitemaps features in SEOPress and Everything You Need to Know About WP Sitemaps.
For increasing the speed of indexing of new pages for your site, look into our powerful Instant Indexing feature.
