google eu letter magnify

To get my news, I often scan Google News, which aggregates articles from around the world based on what the "robot" has determined is trending. It's all about what's trending. It generally misses breaking news and totally ignores important commentary, like my column. Instead, it seems to prefer going to dead links—by which I mean the Wall Street Journal, which requires a subscription to read.

Today is a case in point. There was a breaking news story out of CES about the Dish Network making a bid to buy Clearwire. But instead of linking to any of the related stories by free and readable websites, Google decides that the top story should come from the Wall Street Journal.

In the past, Google had some sort of deal that allowed users to at least read the lead paragraph, or even the whole article, before being blocked by the paywall. Unless you know a bypass trick (discussed below), this no longer applies. Now you run into the paywall and you are done. I suppose actual subscribers get past this but I wonder how the Google bots get by it to find the story in the first place. Is Google paying? I doubt it. So there must be some sort of backdoor for the Google bots, right?

This is bogus and does a disservice to the Google users. Google goes through all this trouble to defeat people trying to game the system but it lets the Wall Street Journal pull this stunt? This is essentially a paid advertisement for Wall Street Journal subscriptions. Is that what Google wants? Is Google getting a cut?

It can't be that hard for Google to simply fix this annoyance. It would take one minute—five tops. Is that too much work?

If a company has a paywall built into its service, Google should not be searching the site at all. The bots should know there is a paywall and simply avoid the search. And, yes, I would like an explanation as to how exactly the bots can search a site that is blocked by a paywall. It's a mystery.

A paywall should be exactly the same as a "robots.txt" that tells the crawler bot to go away. But something is fishy about the WSJ-Google relationship. The easiest way to kill all bots is a meta tag in the robots.txt file, thus: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">.

The WSJ instead specifically blocks certain sub-directories, but apparently not all. And weirder still, if you do a Google News search for the exact same article, rather than simply click on the link in the rundown, you do get the article through some other mechanism.

I challenge the more technical readers out there to figure out what is going on with these two by examining the robots.txt file employed by the WSJ. A public document is available here.

Whatever the case, sites have paywalls for financial reasons. If they want to play games with their content that's one thing, but it should be banned from search results in the same way Google tries to thwart people trying to game the system. Google will prohibit a hapless blogger at the drop of a hat for some sketchy behavior. How about dealing with bigger sites the same way?

You can Follow John C. Dvorak on Twitter @therealdvorak.

More John C. Dvorak:

Go off-topic with John C. Dvorak.