How to get rid of crawl errors created due to language translation plugins (or 404 errors)

Posted on August 21, 2009 by admin

Last day, a blogger was seeking help with a problem he encountered on Google Webmasters. He was using a language translation plugin that automatically translated pages in his website to various different languages. Don’t want to name the plugin here, but I know that it used the Google translator / Babelfish translator service. Now the problem appeared when a lot of these translated URLS started showing up as “page not found” on the Google webmasters console.

I’m assuming that lot of these URLs weren’t approachable because the translation service wasn’t reliable and it would block any automations (which the plugin was using). The plugin clearly is creating some problems here, automatically generating URLs that aren’t really reachable.

Now, there is nothing to panic here as Google webmaster tools normally pick up every feeble 404 signal and just so that the webmaster is aware, shows them on the Webmaster console. These URL’s can be later spidered by Google, and works fine when the translation service is active. But normally, when you have lots of pages to translate, there might be a problem.

So how to fix this problem of large number of 404 errors?

Solution 1 – If you have limited number of URLs

You could manually delete them off your site / turn off the plugin.

Solution 2 – 301 Redirects

You could also issue a 301 redirects to the original URL (without the translation parameters)

Solution 3 – If there are large number of URLS

i) Find out the pattern in the URLs – For instance, for arabic translation the plugin would be using something like – http://www.timnhanh.us/ar/originalurl (ar for Arabic)

With this, issue a command on the Robots.txt file to block scanning all those URLs in the same format.

Example : This command will block all crawling towards the directory “ar” and urls – timnhanh.us/ar~

User-agent: *

Disallow: /ar/

Sometimes, if your translated pages URL structure is complicated, it could run to problems, but generally this is a good solution.

Solution 4 – Google URL removal tool

If none of these methods successfully works, you have Google to your rescue.

Go to the Google Removal Tool and suggest the URLs you’d like to get removed from the index.

404-url-removal

In this case, you can’t suggest mass removal, so you might want to do this in parts by downloading the entire 404 error list from the Google Webmasters console, and pasting them to the removal tool suggestions. I’d really suggest this tool for fine tunings and not mass removals though.

Hope it helped.