robots.txt tips for WordPress to prevent Duplicate Content

When you use the Wordpress.org blog script there is a lot of duplicate content created from the various pages and category pages that the WordPress script creates. If you’ve been reading about SEO, you probably know Google doesn’t like duplicate content and therefore your blog will likely be penalized for it.

Thanks to reading a post by Jeremy at ShoeMoney.com and reading about different User-agents, I was able to create a robots.txt file that will help eliminate duplicate content from Google indexing.

Add the following in your robot.txt file to prevent Google from indexing duplicate content in your WordPress blog:

User-agent: Googlebot

Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /feed/
Disallow: /archives/
Disallow: /index.php
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: */feed/
Disallow: /feed
Disallow: */trackback/
Disallow: /page/
Disallow: /pages/
Disallow: /tag/
Disallow: /category/
Disallow: /*?

User-agent: Googlebot-Image
Disallow: /wp-includes/

User-agent: Mediapartners-Google*
Disallow:

User-agent: ia_archiver
Disallow: /

User-agent: duggmirror
Disallow: /

The above should help with duplicate content on your WordPress blog website.