Sherlock Holmes & the SEO Inquiry-Part I

Sherlock Holmes walked to the window and peered outside. “Ah, Watson! So, the website is not doing too well.”

Watson looked up from the latest edition of the Boston Globe that he was reading and replied. “Holmes, how the devil did you know that?”

“Well, my dear chap. Upon completing my rather energetic walk on the Commons, I observed that you had just gotten the ‘SEO for Dummies” book in the mail, which by the way, is on the dining room table. I further observed that the contented air that you carry about you when you publish a case has been missing for some time. That coupled with the fact that you ate very lightly this morning and that you had printed articles by someone called Matt Cutts led me to believe that the website that you had announced with such fanfare, a month ago, is not proceeding too well.”

“Confound it, Holmes! That’s so elementary.“

“Yes, it is. Isn’t it?”

“So, what’s wrong with the website, Watson?”

“Well, Holmes, for some reason, it doesn’t seem to be getting indexed by the search engines. I’ve added all the necessary meta content and even inserted keywords within the content that are relevant to your field like “master detective”, “cases solved quickly”, “competent discreet detective” etc. And yet, we haven’t had much success.”

“Hmm… And of course, I take it that you’ve used a credible analytics tool to gauge the extent of your failure”

“Yes, Holmes” said Watson, with a resigned air.

“Well, Watson, I suspect that the SEO you had implemented on the website was incomplete. However, this is your lucky day. You may be aware that I had worked with Oxford on a monograph on how to utilize and exploit the Internet resourcefully. Let me give you some tips on how to optimize your website to the fullest extent possible so as to get the search engines to notice and take note of you.”

Search engines & spiders:

In many ways, the search engines are like a complicated criminal – they’re clever, tricky, incomprehensible and very sneaky. But, once you understand how they work, you can turn them to your advantage. That’s where SEO or search engine optimization comes in.

Search engines send little programs called spiders or crawlers to visit your website, index its contents and then display them when someone enters a keyword that triggers your content to be shown. With SEO, you follow some techniques to make your content as relevant as possible to your sales and marketing goals so that the search engines display your website content for search queries relevant to your business. The display results are called search engine result pages (SERPs). You would ideally like the content on your website to be accessible to the spiders so that they index it.

Let’s take a look at a basic check list that you can follow when you implement SEO across the website:

User Input Requirements

I noticed that when someone visits the website, they have to input a zip-code. Did you know that something like this is a roadblock to the spiders? A spider needs an open door to a website so that it can access all the website content and index it. A spider cannot “interact” with a website and therefore it will not be able to access the content of a page that requires user input. As a result, these pages may not get indexed at all.

The solution:

• Remove the user input requirement.
• Move the content in front of the zip-code requirement thus allowing spiders to crawl the site.
• Have user-agent detection to identify a spider and not serve the user input requirement.

Site Security

I noted that you’ve hosted the website on a secure connection using ‘https’. These secure pages can be a challenge for Bing and Yahoo for indexing pages. Also, secure pages can dissuade spiders because they are usually slower to download. Only certain pages of a website need to be secure like the eCommerce page when readers buy a copy of our latest adventure.

The solution:

• Move all web pages that do not need to be secure to an unsecured server.
• Implement user-agent detection. If the user-agent is a spider, the website will serve non-secure content.

Cookie Implementation

Cookies are not good for a search engines health. Let me explain; I think the website requires mandatory cookies. That’s a problem as the spiders are not configured to accept cookies, and therefore cannot crawl a website that requires them which in turn seriously compromises their search engine visibility potential.

The solution:

• Eliminate the acceptance of cookies unless absolutely necessary, such as during a check out process or to access a ‘Members Only’ area.
• Implement user-agent detection. If the user-agent is a spider, the website will not require the acceptance of cookies.

Robots.txt File

When spiders visit the website, they check a file called “robots.txt.” which allows website owners to allow or disallow access to certain parts of the site. If by mistake, you have a nofollow piece of code in that file, it prevents spiders from crawling content-rich areas and as a result all SEO efforts are compromised.

The solution:

Ensure that robots.txt file allows all spiders to crawl all content on your website. Period.

Frames

Frames are so Web 1.0. Here HTML tags used to divide a browser window into different panes of information, each of which you can view and change independently. I sincerely hope that you don’t have this nonsense on the website as the spiders cannot navigate past the frame source page to read the frame content and will not index the majority of the site.

The solution:

• Remove frames and iFrames from the website. In fact, if your web developer suggests this, fire him right away.
• Implement user-agent detection. If the user-agent is a spider, the website will serve the important content without the frames.

To be continued in the next blog entry: Sherlock Holmes & the SEO Inquiry-Part II

• Previous Entry: How to grab Market Share from an existing leader
• Next Entry: Sherlock Holmes & the SEO Inquiry-Part II