Blacklisted by Google - Why?
Just over a week ago, news of a TRON sequel (TR2N) generated quite a buzz among the nerd and geek communities of the internet. As the news spread, every self-respecting TRON nerd wrote about it in their blogs, personal websites, and in many commercial sites. Being a big TRON fan myself, I felt the need to geek-out about the news on my own site as well.
My website doesn't get the high volumes of traffic that the better known sites do, but I enjoy a steady, though modest stream of visitors daily. There was a big increase in my traffic as a result of my TR2N news article. Fellow TRON fans and people of similar interest were finding their way to my website to read my write-up about the news. According to my site stats, many of them took the time to browse my other content afterword. Traffic to my TR2N article, most of it coming from Google searches, continued to increase until about 4 days later when it completely disappeared from Google's search results. No combination of search words, with or without quotes, resulted in my TR2N article being displayed on Google. It was entirely removed from Google's search index.
As a result of Google unindexing my page, traffic to it took a big hit. The number of visitors to my website in general dropped back down to before. TRON fans were no longer finding their way to my website - at least not through Google. Google has the most widely used search engine currently in existence. It is where most people turn for search results, including me. If it ain't found on Google, it almost may as well not exist.
Why was my TR2N write-up unceremoniously erased from the internet search engine giant? The rest of my site's content remained in Google's search index, but my latest, and most popular entry was removed. I didn't break any rules or violate any copyright laws in my TR2N write-up. There were (and still are) a zillion other websites reporting the news of a TRON sequel - all of which, as far as I could tell, remained in Google's search results without issue. What made my own personal take on things any different? Why did my most popular page get the cold shoulder from Google?
My initial thoughts focused on a seemingly harmless comment someone left on my now stealthy TR2N page. It included, two links to a bootleg copy of the TR2N video shown at Comic-Con. Literally dozens of other websites had embedded copies and/or links to the video without penalization from Google. Why would this be a problem on my site? One of the links was to a YouTube copy of the video. Disney issued copyright take-down notices to YouTube about the video, so as quickly as people were posting them there, they were in fact disappearing. Google owns YouTube. Could it be that Google blacklisted my TR2N page because of a simple link a that a commenter left on that page? Is Google unindexing pages for simply containing links to off-site content? (see update below)
I removed both video links in said comment the very next morning, just after realizing my page was now off the Google radar screen. For the record, the videos had already been removed by their video host sites prior to my removing the links to them in the comment.
One of Google's many free services is Webmaster Tools. In a nutshell, it offers webmasters some basic diagnostic tools to make sites more Google-friendly, and improve their visibility. I've been using it for almost two months. Reviewing the current status of my site in Google's Webmaster Tools revealed no problems. Everything checked out just fine. I regenerated and resubmitted my sitemap in an effort to get my page recognized by Google once again.
The following day, my TR2N story was reindexed and showing up in Google's search results, but not as well. Previously, it was showing up in the first few pages of results (depending on search criteria). As of writing this, a search on "tron sequel" (without the quotes) returns my home page (not direct URL to the story) on page 12 of results. The same search previously placed the exact URL (permalink) to the story on page 4 of results. So although my once decently-ranked page has returned to Google's search results, it's not as highly-ranked and not nearly as likely to get discovered as before.
Although my current page-templates might benefit from a little tweaking to minimize the potential for redundant words appearing (in side-bar comment links), they are otherwise search-engine friendly. None of my site's content, including my TR2N article, is in contrast with what is laid out in Google's Webmaster Guidelines.
Some may wonder why this is a big deal. After all, I'm just barking about my personal take on an upcoming film being dropped from Google's listings, right? Aww - poor baby me. I don't make a living from my website and my content isn't exactly hottest-topic-of the-day stuff. If that is how you are looking at my concern over Google's removal of my page from their index, than you you've missed the point.
The bigger picture is that if I did rely on my website as a means of primary income, as a growing number of small business owners do, this "little" issue could have been catastrophic. Regardless of the purpose of my website, my voice was effectively silenced. My content was out there but invisible on Google - the search engine that the majority of internet users rely on to find what they're looking for.
Even though it may have been just an entertainment related article, this experience was rather surreal to me. I put a lot of time and effort in to the content on my website: creative videos, product reviews, random photographs, personal thoughts - everything I publish. There are people interested in what I have to present... when they can find it. Having my content do well one day and be all but invisible the next was an ugly experience. Although I'm not the first person this has happened to and won't be the last, the experience was a real eye-opener for me. As far as I can tell, I did everything right, and yet my content was still removed from Google for a time. When it returned, it had a much lower rank and noticeably weaker reach.
Google doesn't owe me anything. As far as I know, it is completely within their right to index or not index whatever they see fit based on whatever they feel relevant. Because they are so widely relied upon - I see a danger in this. I'm not sure if the danger is in Google's ability to decide what people can and cannot find through their service with little explanation, or the fact that most people simply don't feel the need to look anywhere else.
In an effort to minimize the potential for this kind of thing happening again, I've removed the ability for people to create links in their comments. I don't get comment-spam frequently, so I had hoped I wouldn't have to remove that feature just yet. When people take the time to offer their input on my site, I'd prefer to allow them as much freedom in their comments as possible.
Right now, what bothers me the most is that there is no way to know if or when content will end up getting blacklisted. Although I speculate that Google knocked my page out of their index because of links a visitor posted in a comment, I have no way of verifying or dispelling it. Unfortunately, I haven't really learned anything from this experience in terms of maintaining a search-friendly website.
How about a content-checker, Google? With your vast resources, this should be quite an easy feature to make. I'm using your Webmaster Tools. Where's that warning message that tells me why my most popular page just got yanked from your index and what I need to do to resolve the problem? Why all the secrecy? Shouldn't webmasters be warned when we're in danger of having our content unindexed? A little automated alert to us small guys would really help out a lot. I'm ready and willing to play by your rules... what are they?
In light of this rather frustrating experience, I'm still a believer in Google. They're still my home page at home and at work. I do a lot of research at my day-job, and Google is usually my most effective tool. I just wish the G-machine was a little friendlier to my website. I'd be lying right now to say that I am not concerned about this page ending up missing from Google searches, or at least very low ranked because of its subject matter. I think Google is bigger than doing something like that, but I guess time will tell.
If you feel that my concerns are relevant and should be addressed by Google, please post a link to this page on your website or blog (use the permalink at the top) and be sure to include your own thoughts on the matter. Perhaps if enough webmasters raise their voice about this, we'll see some positive changes take place. In the meantime, I'll continue researching to figure out exactly what went wrong.
Update: August 8
I've been able to put to rest the theory of Google temporarily removing my page from their index because of off-site video links. Another one of my recent entires was showing up in Google searches only to completely disappear two days after publishing it. Because the newly unindexed page has no video links and because several other websites contained links to the same videos without Google penalization, it is unlikely the video links had anything to do with it.
I have a new theory as to why the page mentioned in this article and a more recent page ended up blacklisted by Google (although it remained available in Yahoo). You can follow my latest adventures in Google-blacklisting here: Blacklisted by Google... Again