How to Deal with the 404 Error for Search Engine Optimization

“Error 404 – Page Not Found” – This is one of the most common web errors we see in our day-to-day Internet surfing. We know that error pages offer an obviously negative user experience, but how do they impact search engines? Are 404 pages bad for SEO? Do search engines penalize websites with too many 404 status code pages?

In this article, I will look at 404 error pages from the perspective of both users and search engines, and will recommend some of the best practices that one can follow to improve their 404 error pages.

404 Page not found

Most common reasons for 404 error pages:

404 errors can happen for multiple reasons. Some of the most common are as follows:

  • You change your permalink or the link structure of your website. (This mostly happens when changing design, or changing permalinks.)
  • You removed content from your website.
  • Somebody linked your website to a misspelled link or to an otherwise incorrect link.

When a user or search engine bot requests access to a bad page on a server, web servers usually reply with an HTTP status code of 404, to tell the user that the page doesn’t exist on the server. Usually search engine bots are smart, and they don’t care much about 404 error pages.  For the real-life user, however,  aa 404 error page presents a bad user experience, and in response the user usually goes back and follows another link.

Another disadvantage is of the 404 error is that it can cause you to miss out on important links from other domains. This is not something that I worry much about, but when a user lands on a website by way of a referral link and sees a 404 page, it usually makes them go away.

Finding 404 links and fixing them for users as well as for SEO:

Let’s figure out what you can do with 404 error pages, and how to deal with them to your best advantage.

We first need to identify any 404 pages on your blog.  The best place to start is with the Google Webmaster tool. Login to your Webmaster tool dashboard > Crawl errors > Not found >

find 404 error pages

Click on any of the links and you will see “linked from” which will give you an idea of where these pages are linked from. This is also handy for finding potential link juice on your website.

If you are using WordPress, you can use a redirection plugin to monitor and redirect 404 error pages from your dashboard. (This is what I use here at ShoutMeLoud). Once you have a list of 404 pages for your domain, here are several solutions for dealing with them:

  • 301 Redirect the link to the most relevant post/article or category. If there is no relevant article or category, follow the next step.
  • If the error is arising due to a misspelled link from another domain, you can either ask the webmaster to update the link, or you can use the 301 redirection.
  • If no relevant article is on your website related to the 404 link, simply let it be. Google will automatically de-index such pages eventually.
  • You can also manually de-index such pages from the web index using the webmaster removal tool. However, if you have thousands of such pages, it will be quite a task for you. So I would suggest either the previous suggestion or the following one.
  • Use a custom 404 page to spice up your page. You can add a search box, show articles relevant to the user’s search query, or show some of the most popular articles from your portal. The whole idea is to give a good user experience.

Another thing to keep in mind is that if you have a portal with thousands of 404 pages, you do not want the search engine bots to waste their limited crawling resources on such pages.  For this reason it is best to re-direct whenever possible or to block bots from accessing these pages using Robots.txt, and remove them using the Google Webmaster tool. This is ideal particularly when you have removed any directory or category/tag pages from your portal.

404 pages are not SEO enemies, but it’s not a good idea to have a long list of 404 pages on your blog. My main concern with 404 pages is the poor user-experience they create.  With search engine optimization revolving around offering a good user experience, this is an issue we do want to make the effort to address.

Related reading:

How do you handle 404 pages on your blog or website? Do you set a redirection or show a custom 404 error page?  Let us know using the comments section below.

If you find the information in this post useful, please share it with your friends and colleagues on Facebook, Twitter and Google Plus.

Subscribe on Youtube

Article By
Harsh Agrawal is a blog scientist and a professional blogger. He loves to read & write about Blogging, SEO, WordPress, Make money online & technology. His blog, ShoutMeLoud receives more than half a million PV in a month, and is an award winning blog.

COMMENTs ( 12 )

  1. says

    Hi Harsh,

    I was wondering if you already have a 404 error page setup and if those errors still come up in WMT is it still a bad effect on SEO?

  2. says

    Hello Harsh,

    Thanks for such an helpful post sir. I was looking for something like this from a long as most of my pages gets deleted from my blog due to which I got so many 404s which ruined the authority. Now your post will help me some to get back/increase the authority of my blog.

    Thanks

  3. says

    Hi Harsh,
    Your post is very knowledgeable. Almost everyday I face the problem of error 404 on many sites and many times when I surf the internet I get irritated. Your suggestions are very useful and effective to handle such errors. The tool that is suggested above to find broken links is useful for SEO. Thank you for sharing the article.

  4. PolarStar says

    Hi Harsh, thanks this is a good and relevant topic. I´ve been dealing with 404s for a while now. I initially had a lot of them because of a Flash banner that google could not recognize for technical reasons, but we solved that.

    Still we have around 500 not found links, this is for a big website, and many of them actually work when you click on the from Google Webmaster Tools. How do you deal with that? From what I´ve been reading there are also server configuration issues that could temporarily create a 404 message for a page that is actually working well.

    Thanks, PS

  5. Mahesh says

    Nice piece of information. I want to know if resubmitting sitemap will help to de-index them faster? Is there any plugin to manage the 404 errors, so it may automatically de-index instead of redirecting?

  6. Abhinav Jain says

    Hi Harsh,

    i would like to add one more point which i have faced. One of my website i am doing SEO for suddenly started showing up increase in 404 pages. It was like it was showing around 200 pages as 404 error pages in google webmaster and then suddenly it went up to 76ooo , 404 pages in google webmaster. Now first issue was how the hell it can happen when the website is having only 1500 pages and it is showing up 76000 pages as 404 error pages found.

    Well yes the website is actually dynamic website. So chances are some id base pages were getting created from some where. Well i found it as to from where the leakage was happening but there was another issue which was that all those 4o4 pages generated by mistake were showing response status as 200, which means all those pages were working and existing in the eye of google. BIG BIG mistake when we open any of those pages in browser it use to show 404 error message but actually those pages were not showing server response status as 404.

    Lesson learned even if your error page is showing 404 message you also need to check it page response status as 404 if it is not showing you need to set it.

  7. says

    I had number of 404 errors and I redirected all of them to related pages before applying for google adsense and after few days I got mail from adsense team with rejection of my application and the one of the reason was somewhat like “You have number of redirects”.
    After that I searched lot about impact of 404 errors pages on seo but I found that they have no such impact but then why google adsense rejected my aplication?Do we need to completely remove all such pages before applying to adsense?

  8. stargaterich says

    The redirect plug-in is the easiest and fastest method to resolve the 404 error pages. I think one way to monetize a blog is to redirect it to an affiliate sales page. Thanks for the great content in this post.

  9. Heru Prasetyono says

    I think thie error message (404-error page not found) has closely related to the worst service of the web host. It often happened to my blog on my initial start blogging.
    I experienced this for almost one month and this drove me mad and frunstrated.
    Until I finally decided to migrate to a new web host service which I can rely on and it serves me well up to now. My first web host service was really disappoited me.
    And I wrote this bad experience in one of my blog posts.

ADD COMMENT